{
  "video_id": "reddit_1unrse9",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "I benchmarked 13 models at 65K-128K context to find out what actually matters for agentic workloads",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1unrse9/i_benchmarked_13_models_at_65k128k_context_to/",
  "external_url": null,
  "upload_date": "20260705",
  "published_at": "2026-07-05T03:37:23+00:00",
  "transcript": "# I benchmarked 13 models at 65K-128K context to find out what actually matters for agentic workloads — prefill dominates everything, and KV head count beats parameter count\n\nI've been running local LLMs for agentic workflows (tool use, coding agents, RAG) and kept seeing people obsess over tg128 (token generation speed) as the headline performance metric. So I ran a structured long-context benchmark to figure out what *actually* matters when your context window is full. The answer surprised me.\n\n---\n\n**Setup**\n\n- **GPU**: RX 7900 XT 20GB (Vulkan backend, RADV/Mesa)\n- **Backend**: llama.cpp / llama-bench (build 9860)\n- **Flags**: `-ngl 99` (GTT spill), `-fa on`, `-ub 2048 -b 16384`, ASPM=performance, bare TTY to free VRAM\n- **13 models**: 5 dense, 6 MoE, 1 Mamba2 hybrid, 1 MLA MoE — ranging from 5GB to 18GB\n- **3 KV cache tiers**: Q8_0 K / Q4_0 V (aggressive), Q8_0 K / Q8_0 V (symmetric), F16 (baseline)\n- **Context sizes**: 512, 4K, 16K, 65K, 131K — both pure prefill (pp) and prompt+gen (pg)\n- Full run took ~21 hours across two sessions\n\n---\n\n**Full prefill speed results (Q8_0 K / Q8_0 V KV cache, tokens/sec)**\n\nIf you just want the raw numbers, here's every model tested. pp = pure prompt processing (prefill), tg128 = token generation (decode). Sorted by pp131K.\n\n|Model|Size|Type|pp512|pp4K|pp16K|pp65K|pp131K|tg128|\n|:-|-:|:-|-:|-:|-:|-:|-:|-:|\n|Trinity-Mini|16G|MoE 3B/26B|2639|2924|2370|**1419**|**923**|**150**|\n|Granite-4.0-H-Small|17G|Mamba2+MoE|1115|1271|1220|1043|875|71|\n|Ornith-9B / Qwen3.5-9B|6G|Dense|2103|2220|1943|1274|873|92|\n|Qwen3.6-35B-A3B|18G|MoE 3B/35B|2184|2736|2227|1268|802|110|\n|Gemma-4-26B-A4B|14G|MoE 4B/26B|2523|2798|2076|1024|600|119|\n|North-Mini-Code|15G|MoE 3B/30B|2155|2187|1568|900|579|134|\n|Gemma-4-12B|7G|Dense|1492|1498|1145|595|350|66|\n|Qwen3.6-27B|16G|Dense|693|681|602|406|285|32|\n|Granite-4.1-8B|5G|Dense|1965|1807|1124|442|244|93|\n|Ministral-3-14B|8G|Dense|1419|1325|916|404|232|67|\n|Apriel-1.6-15B|9G|Dense|1332|1208|812|347|197|66|\n|Devstral-24B|15G|Dense|829|796|628|313|---|42|\n|GLM-4.7-Flash|16G|MoE (MLA)|1822|1054|358|---|---|---|\n\nA few things to note: Devstral-24B couldn't complete the 131K test (8 KV heads × 128 dim = 160 KB/token — KV cache alone is ~21GB at 131K). GLM-4.7-Flash crashed above 16K (MLA issue, see Finding 5). Ornith-9B is architecturally identical to Qwen3.5-9B.\n\n---\n\n**Finding 1: At 65K+ context, prefill is 94–99% of wall-clock time. tg128 is nearly irrelevant for short agentic outputs.**\n\nHere's the wall-clock breakdown for a real agentic query — 65K context in, 300 tokens out (typical tool-use response). Sorted by total time:\n\n|Model|Type|Prefill|Decode|Total|Prefill %|\n|:-|:-|-:|-:|-:|-:|\n|Trinity-Mini (MoE 3B/26B)|MoE|46.2s|2.0s|48.2s|96%|\n|Qwen3.6-35B-A3B (MoE)|MoE|51.7s|2.7s|54.4s|95%|\n|Ornith-9B / Qwen3.5-9B|Dense|51.4s|3.3s|54.7s|94%|\n|Gemma-4-26B-A4B (MoE)|MoE|64.0s|2.5s|66.5s|96%|\n|Granite-4.0-H-Small (Mamba2)|Mamba2|62.8s|4.2s|67.1s|94%|\n|North-Mini-Code (MoE)|MoE|72.8s|2.2s|75.0s|97%|\n|Gemma-4-12B|Dense|110.2s|4.5s|114.7s|96%|\n|Granite-4.1-8B|Dense|148.4s|3.2s|151.6s|98%|\n|Qwen3.6-27B|Dense|161.4s|9.3s|170.7s|95%|\n|Ministral-3-14B|Dense|162.0s|4.5s|166.5s|97%|\n|Apriel-1.6-15B|Dense|188.9s|4.6s|193.5s|98%|\n|Devstral-24B|Dense|209.5s|7.2s|216.6s|97%|\n\nDecode is 1–5% of the time you actually wait. If your agent makes a short tool call or writes a brief response, **the only thing that matters is how fast you can process the context window.**\n\nThis means benchmark reports that lead with tg128 are misleading for agentic use cases. **pp65K / pp131K is the metric that matters.** The `pg(prompt, gen)` blended metric is better but still obscures the split — a model with fast prefill + catastrophically slow decode can look mediocre on pg despite being excellent for short outputs.\n\n---\n\n**Finding 2: KV head count is the dominant architectural factor for long-context prefill — not parameter count, not MoE vs dense**\n\nPrefill speed retention (% of pp4K speed) at increasing context, all models:\n\n|Model|Size|KV Heads|pp4K|16K|65K|131K|Type|\n|:-|-:|:-|-:|-:|-:|-:|:-|\n|Granite-4.0-H-Small|17G|Mamba2\\*|1271|**96%**|**82%**|**69%**|Mamba2+MoE|\n|Qwen3.6-27B|16G|4×256|681|88%|60%|42%|Dense|\n|Ornith-9B / Qwen3.5-9B|6G|4×128|2220|87%|57%|39%|Dense|\n|Trinity-Mini|16G|4×128|2924|81%|49%|32%|MoE|\n|Qwen3.6-35B-A3B|18G|4×128|2736|81%|46%|29%|MoE|\n|Gemma-4-12B|7G|8×128|1498|76%|40%|23%|Dense|\n|Gemma-4-26B-A4B|14G|4×256|2798|74%|37%|21%|MoE|\n|North-Mini-Code|15G|4×128|2187|72%|41%|26%|MoE|\n|Apriel-1.6-15B|9G|8×128|1208|67%|29%|16%|Dense|\n|Ministral-3-14B|8G|8×128|1325|69%|31%|18%|Dense|\n|Granite-4.1-8B|5G|8×128|1807|62%|24%|**14%**|Dense|\n|Devstral-24B|15G|8×128|796|79%|39%|---|Dense|\n|GLM-4.7-Flash|16G|MLA (1×576)|1054|**34%**|---|---|MoE (MLA)|\n\n*\\*Granite-H-Small has 4 attention layers + 36 Mamba2 layers (recurrent state, no KV cache)*\n\nOrnith-9B / Qwen3.5-9B (9B dense, 4 KV heads × 128 dim = 64 KB/token KV) is **4.4× faster at 128K context** than Apriel-15B (15B dense, 8 KV heads × 128 dim = 160 KB/token) — despite being the same dense class and half the size. The difference is purely KV cache architecture. Every attention pass has to scan the full KV cache, and 8 KV heads means 2.5× more data to scan per token.\n\n**Practical rule**: When evaluating a model for long context, check `n_kv_heads` and `head_dim` in the config *before* looking at parameter count. Two models from the same family can differ by 3–4× at 128K if one has 4 KV heads and the other has 8.\n\n---\n\n**Finding 3: Mamba2 hybrid models have near-flat prefill scaling. The architecture actually delivers.**\n\nI was skeptical of the Mamba2 hype, but the data is clear. Granite-4.0-H-Small (IBM, 4 attention layers + 36 Mamba2 layers) retains **69% of its pp4K speed at 131K context** — every transformer model in the test dropped below 42%.\n\n|Model|pp4K|pp131K|Slowdown|\n|:-|-:|-:|:-|\n|Granite-H-Small (Mamba2)|1271|**875**|1.45×|\n|Trinity-Mini (MoE)|2924|923|3.2×|\n|Ornith-9B / Qwen3.5-9B (Dense GQA)|2220|873|2.5×|\n|Granite-8B (Dense)|1807|244|7.4×|\n\nAt 131K context, Granite-H-Small (17GB) ties Ornith-9B / Qwen3.5-9B (6GB) at ~875 t/s despite being 3× the file size. The Mamba2 layers use fixed recurrent state instead of growing KV cache, so only 4 attention layers contribute to KV growth.\n\nThe catch: its decode is slow (71 t/s) and its reasoning quality is low. But for the specific workload pattern of \"huge context, short output\" — which is exactly what agentic tool use looks like — the prefill scaling advantage is real and measurable. If someone trains a good model on this architecture, it could be a serious agentic contender.\n\n---\n\n**Finding 4: F16 KV cache can be FASTER than Q8/Q4 quantized KV — the dequantization paradox**\n\nConventional wisdom says quantize your KV cache (Q8_0 K / Q4_0 V) for speed — smaller cache, less bandwidth. I tested this head-to-head at 65K context, comparing F16 baseline against Q8_0 K / Q8_0 V (the results were identical to Q8K/Q4V within ±1% — V cache quantization choice turned out to be irrelevant):\n\n|Model|Type|Q8K/Q8V|F16|F16 advantage|\n|:-|:-|-:|-:|:-|\n|Gemma-4-26B-A4B|MoE|1015|1554|**+53%**|\n|Gemma-4-12B|Dense (7GB)|593|857|**+44%**|\n|Qwen3.6-35B-A3B|MoE|1273|1573|**+24%**|\n|Ornith-9B / Qwen3.5-9B|Dense (6GB)|1276|1544|**+21%**|\n|Trinity-Mini|MoE|1429|1583|+11%|\n|Granite-H-Small|Mamba2|1040|984|-5%|\n|Ministral-3-14B|Dense (8KV)|409|335|**-18%**|\n|Granite-4.1-8B|Dense (8KV)|447|358|**-20%**|\n|Apriel-1.6-15B|Dense (8KV)|351|120|**-66%**|\n\nF16 wins for **MoE models** and **small dense models**. It loses badly for **dense models with many KV heads**. Here's why:\n\nThe Q8/Q4 KV cache dequantization is a *compute* operation that scales with context length. At 65K context, the attention kernel has to dequantize 65K × n_kv_heads × head_dim Q8/Q4 elements per token. This compute cost exceeds the bandwidth saved by halving the cache size.\n\nMeanwhile, F16 KV doubles the cache but requires zero dequantization. For MoE models, the larger F16 cache causes GTT spill — but only the active parameters (~3B of 35B) cross PCIe, so the spill penalty is small. For dense models with 8 KV heads, the F16 cache is both larger *and* all weights spill — double penalty.\n\n**Updated rule:**\n\n- **F16 wins**: MoE models (tiny active spill footprint), small dense models (<10GB), efficient-GQA dense (4 KV heads)\n- **F16 loses**: Dense + 8+ KV heads + >10GB (full weight spill + large KV = catastrophic)\n- **Q8K/Q4V vs Q8K/Q8V**: Complete wash (±1%) across every model. V cache quantization choice is irrelevant — pick whichever.\n\nTest F16 on your hardware at your actual working context. The conventional wisdom isn't always right.\n\n---\n\n**Finding 5: MLA (Multi-head Latent Attention) degrades hard on Vulkan as context grows**\n\nGLM-4.7-Flash (1 KV head, 576 dim — MLA architecture) showed a steep prefill degradation:\n\n|Context|pp (t/s)|vs pp512|\n|:-|-:|:-|\n|pp512|1822|100%|\n|pp4K|1054|58%|\n|pp16K|358|20%|\n|pp65K|crashed|---|\n\nThat's an 80% drop from 512 to 16K context. The 65K benchmark test crashed (Vulkan DeviceLost). However — I want to be clear about what this data does and doesn't show. I didn't test any context sizes between 16K and 65K, so the exact crash boundary is unknown. I'm also running this same model in daily use at 20K+ context without crashes, so it's not a hard wall at 16K — the failure point is somewhere above that.\n\nWhat is clear from the data: MLA's prefill scaling on Vulkan is dramatically worse than standard attention. Whether that becomes a hard crash or just very slow depends on context size and VRAM headroom. The 66% drop from pp4K to pp16K is measured and real — MLA's KV compression/decompression kernel appears to scale poorly with context on the Vulkan backend. **Don't extrapolate short-context MLA benchmarks to long context on Vulkan, and don't assume the degradation is linear — it looks like it accelerates.** This may be backend-specific — CUDA or Metal may handle MLA's attention pattern better.\n\n---\n\n**Finding 6: MoE models win the speed × intelligence composite for agentic work**\n\nCombining the speed data with Artificial Analysis Intelligence Index v4.1 scores into a composite (intelligence weighted by inverse wall-clock time). Score = AA Intel × (50s / wall_clock_at_65K), so 50s wall = 1.0× AA multiplier.\n\n|Rank|Model|AA Intel|Size|Wall (65K)|Type|Composite|\n|:-|-:|-:|-:|-:|:-|-:|\n|1|Qwen3.6-35B-A3B|32|18G|54s|MoE|**29.4**|\n|2|Trinity-Mini|24|16G|48s|MoE|**24.9**|\n|3|Gemma-4-26B-A4B|26|14G|67s|MoE|**19.6**|\n|4|North-Mini-Code|21|15G|75s|MoE|14.0|\n|5|Ornith-9B / Qwen3.5-9B|15|6G|55s|Dense|13.7|\n|6|Qwen3.6-27B|37|16G|171s|Dense|10.8|\n|7|Gemma-4-12B|18|7G|115s|Dense|7.8|\n|8|Granite-4.0-H-Small|7|17G|67s|Mamba2|5.2|\n|9|Apriel-1.6-15B|19|9G|194s|Dense|4.9|\n|10|Ministral-3-14B|15|8G|167s|Dense|4.5|\n|11|Granite-4.1-8B|12|5G|152s|Dense|4.0|\n|12|Devstral-24B|17|15G|217s|Dense|3.9|\n\nGLM-4.7-Flash excluded — no 65K data (MLA crash).\n\nThe smartest model (Qwen3.6-27B, AA=37) finishes 6th because it takes 3× longer than the MoE models. The top MoE (Qwen3.6-35B-A3B, AA=32) delivers 87% of the intelligence at 3× the speed. For agentic workflows where the model makes multiple short-output iterations, that tradeoff wins.\n\nMoE models benefit twice: fast decode (read only active params) *and* cheap GTT spill (only active weights cross PCIe when context forces spillover). The combination makes them disproportionately strong at long context on constrained VRAM.\n\n---\n\n**TL;DR — Practical takeaways for local agentic LLM deployment**\n\n1. **Stop obsessing over tg128.** At 65K+ context, prefill is 94–99% of wall-clock for short outputs. Benchmark pp65K / pp131K instead.\n2. **Check KV head count first.** 4 KV heads × 128 dim (64 KB/token) scales dramatically better than 8 KV heads × 128 dim (160 KB/token) at long context — regardless of model size.\n3. **Test F16 KV cache at your working context.** It can be 20–53% faster than Q8/Q4 for MoE and small dense models. The conventional \"always quantize KV\" wisdom breaks at long context on some architectures.\n4. **MoE models are the sweet spot for agentic long-context on consumer VRAM.** Cheap GTT spill + fast decode + competitive intelligence. A 30B-class MoE delivers 80%+ of top dense model intelligence at 2–3× the speed.\n5. **Mamba2 hybrids are real.** Near-flat prefill scaling to 131K. Currently let down by training quality, not architecture. Worth watching.\n6. **MLA degrades hard on Vulkan as context grows.** GLM-4.7-Flash lost 80% of prefill speed from 512 to 16K context and crashed at 65K. It's usable at moderate context (I run it daily at 20K+), but the scaling curve is steep and non-linear. Don't extrapolate short-context MLA benchmarks.\n\n---\n\nAll data collected on llama.cpp build 9860, Vulkan backend, RX 7900 XT 20GB. Your mileage will vary on other hardware — especially the F16 KV findings, which depend on your GPU's bandwidth vs compute ratio and your VRAM headroom. The architectural patterns (KV heads matter, prefill dominates, MoE scales better) should generalize.\n\nHappy to share the raw JSONL or the benchmark script if anyone wants to reproduce.\n\n\n\n--- Top Comments ---\n\n\n[1 upvotes] Your post is getting popular and we just featured it on our Discord! [Come check it out!](https://discord.gg/PgFhZ8cnWW)\n\nYou've also been given a special flair for your contribution. We appreciate your post!\n\n*I am a bot and this action was performed automatically.*\n\n[37 upvotes] Usually these benchmarks draw no conclusions and you’re left to interpret what’s going on. I appreciate the explanations. Thanks. \n\n[20 upvotes] pp65k or more on the first prompt is usually only reached when resuming old convos. The normal flow from average agentic coding use is filling 20-25k context with relevant info on the task, and then driving the conversation forward upto 180k+, then compacting. Successive prompts should not cause re-pp.\n\nIf your agentic tasks are regularly filling up 65k at the very first prompt, then your context is probably polluted with needless info or unrelated junk.\n\n[15 upvotes] Also, wanted to add a few more thing which I think you got wrong - \n\n1. MoE are sweet spot for small tasks. That 20% difference between MoE and Dense output quality makes a LOT of difference when the task becomes bigger or more complex. MoEs simply start to make mistakes and ignore requirements.  \n2. In agentic coding use 75%+ of the time is TG - there is a cycle of planning, re-planning, writing code via tool use, review and re-write, debugging - ALL of which uses TG heavily. Once the initial existing code has been ingested along with task specs etc, thats the only part which requires PP. EVERY other part will depend on TG.\n\n[13 upvotes] Is quantizing kv cache conventional wisdom??  ",
  "transcript_chars": 14771,
  "ingested_at": "2026-07-05T13:30:10.282816+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 205,
    "upvote_ratio": 0.91,
    "num_comments": 62,
    "author": "linuxid10t",
    "is_self": true
  }
}