LLM Speed Benchmarking: How to Test Local Model Performance Before You Buy Hardware

A new tool called llmspeed.dev just hit Hacker News, and it’s addressing one of the most painful decisions in AI development: how do you know if that expensive GPU you’re eyeing will actually run your chosen LLM at an acceptable speed? The project lets you “feel” how fast a local LLM runs before committing thousands of dollars to hardware. It’s a clever solution to a very real problem—but more importantly, it opens the door to understanding the critical skill of LLM inference benchmarking, something every AI engineer should master.

Whether you’re building a chatbot, a code assistant, or any application requiring real-time language model responses, inference speed isn’t just a nice-to-have metric. It’s the difference between a usable product and an expensive science experiment. Let’s dive into the technical foundations of LLM speed testing and learn how to evaluate hardware requirements systematically.

Why LLM Inference Speed Actually Matters

According to early reports, llmspeed.dev tackles the problem of hardware uncertainty head-on by simulating the experience of different inference speeds. This isn’t just about impatience—it’s about economics and user experience. Consider that a GPT-3.5-scale model generating at 10 tokens per second feels responsive, while the same model at 2 tokens per second creates an agonizing wait that kills engagement.

The math gets serious fast. If you’re serving 100 concurrent users, each needing an average 500-token response, the difference between 20 and 40 tokens/second means either needing two GPU instances or four. At current H100 prices, that’s a $60,000 annual difference in cloud costs—or the decision between a $15,000 and $30,000 on-premise setup.

Understanding these trade-offs requires more than intuition. You need systematic benchmarking skills, which many data science programs gloss over. Platforms like DataCamp offer courses on ML model deployment that touch on performance monitoring, but hands-on LLM benchmarking requires a specific methodology you won’t find in most curriculums.

Understanding Tokens Per Second: The Core Metric

Tokens per second (t/s) is your north star metric for LLM performance, but it’s more nuanced than it appears. There are actually two distinct phases: prefill and decode.

Prefill is the initial processing of your prompt—the model ingests all input tokens at once. This phase is highly parallelizable and typically faster. Decode, however, generates one token at a time, each depending on all previous tokens. This autoregressive nature creates a sequential bottleneck that dominates user-perceived latency.

The Real Formula: Time to First Token + Generation Speed

When evaluating an LLM’s speed, you need:

• TTFT (Time to First Token): Prefill latency, critical for perceived responsiveness
• Inter-token latency: Time between each subsequent token during decode
• Total throughput: Tokens per second averaged across the entire generation

A model that takes 2 seconds to produce the first token but then generates 50 t/s feels different from one with 0.3s TTFT and 25 t/s generation. The former creates an awkward pause; the latter starts immediately but delivers more slowly. Both average to different actual user experiences that simple t/s doesn’t capture.

Practical Benchmarking Methods for Local LLMs

Let’s get concrete. Here’s how to actually benchmark an LLM’s inference speed on hardware you’re considering—or on your current setup to establish baselines.

Method 1: Direct Timing with llama.cpp

For most local LLM work, llama.cpp is the de facto standard. It supports a massive range of models and provides detailed timing information. Here’s a practical benchmark script:

#!/bin/bash
# Benchmark script for llama.cpp inference speed
# Tests generation with varying context lengths

MODEL_PATH="./models/llama-2-7b.Q4_K_M.gguf"
PROMPT="Explain the theory of relativity in detail, covering both special and general relativity."

# Run inference with timing
./main -m $MODEL_PATH \
  -p "$PROMPT" \
  -n 512 \
  --threads 8 \
  2>&1 | tee benchmark_output.txt

# Extract key metrics
grep "timings" benchmark_output.txt
# Look for: eval time = X ms / Y tokens (Z tokens/s)

This command runs the model with your prompt and generates 512 tokens, reporting detailed timing. The critical output line shows “eval time” which is your decode speed—the actual tokens per second users will experience.

Method 2: Python-based Benchmarking with Transformers

If you’re working with Hugging Face models, here’s a more sophisticated approach that measures both phases separately:

import torch
import time
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and measure inference timing
model_name = "meta-llama/Llama-2-7b-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.float16
)

prompt = "Write a detailed analysis of machine learning deployment challenges"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Measure TTFT (prefill)
start_prefill = time.perf_counter()
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=1,
        do_sample=False
    )
ttft = time.perf_counter() - start_prefill

# Measure full generation
start_full = time.perf_counter()
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=200,
        do_sample=False
    )
total_time = time.perf_counter() - start_full

tokens_generated = outputs.shape[1] - inputs['input_ids'].shape[1]
tokens_per_second = tokens_generated / total_time

print(f"Time to First Token: {ttft:.3f}s")
print(f"Total generation: {tokens_generated} tokens in {total_time:.2f}s")
print(f"Speed: {tokens_per_second:.2f} tokens/second")

Run this on the actual hardware you’re evaluating. The results tell you precisely what to expect in production, accounting for model size, quantization level, and GPU memory bandwidth—all critical factors that paper specs don’t reveal.

Estimating Hardware Requirements From Benchmarks

Once you have benchmark data, you can work backwards to hardware requirements. The key variables are:

Memory bandwidth is typically your bottleneck for decode speed. During autoregressive generation, the model loads weights from memory for each token. An RTX 4090 has ~1TB/s bandwidth; an A100 has ~2TB/s. This roughly doubles your decode speed for memory-bound operations, which most large LLM inference is.

VRAM capacity determines what fits. A 7B parameter model in FP16 needs ~14GB just for weights. Add KV cache for long contexts (can be several GB for 4k+ contexts), and you need headroom. Quantization to 4-bit drops this to ~4GB for weights, making a 7B model comfortable on consumer cards.

Compute (TFLOPS) matters more for prefill than decode. Batching multiple requests lets you amortize prefill costs, but for single-user applications, prefill speed determines TTFT.

Many practitioners deepen their understanding of these hardware-software trade-offs through structured learning. Advanced courses on Coursera covering model optimization and deployment architectures help connect theoretical knowledge to these real-world infrastructure decisions.

Beyond Hardware: Optimization Strategies

Before you buy that expensive GPU, consider what software optimizations might close the gap. Modern LLM serving has several layers of speed improvements:

Quantization

Dropping from FP16 to 4-bit quantization (via GPTQ or GGUF formats) typically delivers 3-4x speedup with minimal quality loss on most tasks. You’re trading some precision for massive memory savings and bandwidth improvements. For a 13B model, this often means the difference between needing a $15,000 A6000 and running fine on a $1,500 RTX 4090.

KV Cache Optimization

The key-value cache stores attention states for previously generated tokens. PagedAttention (used by vLLM) reduces memory waste here by 40-60%, letting you fit longer contexts or batch more requests on the same hardware.

Speculative Decoding

Use a small draft model to predict multiple tokens, then verify with your large model in parallel. When it works (often), you get 2-3x speedup by generating several tokens per forward pass instead of one. The catch is it requires careful tuning and doesn’t help all workloads equally.

The tooling landscape evolves rapidly. Projects like llmspeed.dev emerge because the gap between theoretical specs and real performance remains massive. Building your own benchmarking methodology—tailored to your specific models, hardware options, and use cases—is now a core skill for any AI engineer doing production work.

Start by benchmarking a model you know well on hardware you have access to. Establish your baseline. Then test variations: different quantization levels, context lengths, batch sizes. Build intuition for what the numbers mean in actual user experience. Only then can you make informed decisions about where to invest—whether that’s upgrading hardware, optimizing software, or choosing different model architectures altogether.