{
  "video_id": "reddit_1uu6p9o",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "**Your $80 Tesla P100 has been doing silently noisy math in llama.cpp for years. Three lines fix it, for free.**",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1uu6p9o/your_80_tesla_p100_has_been_doing_silently_noisy/",
  "external_url": null,
  "upload_date": "20260712",
  "published_at": "2026-07-12T05:41:12+00:00",
  "transcript": "\\## TLDR;\n\nShipped — in turboquant v0.3.0, downloadable now. [https://github.com/TheTom/llama-cpp-turboquant/releases/tag/tqp-v0.3.0](https://github.com/TheTom/llama-cpp-turboquant/releases/tag/tqp-v0.3.0)\n\nllama.cpp's CUDA code has a flag that means \"this GPU is fast at fp16, so do the math in fp16.\"  \nThe GTX 10-series and P40's (sm\\_61) were exempted from it long ago. The P100 (sm\\_60) was not,   \nironically, because it's the one Pascal card with fast fp16 hardware. Nvidia put fast FP16  \nsilicon on the P100, so it makes total sense they'd want to tap into that extra performance.   \nWhat they didn't check, apparently, was the price.\n\nPR Status: TheTom (merged) [https://github.com/TheTom/llama-cpp-turboquant/pull/212](https://github.com/TheTom/llama-cpp-turboquant/pull/212)  \nspiritbuun (open) [https://github.com/spiritbuun/buun-llama-cpp/pull/80](https://github.com/spiritbuun/buun-llama-cpp/pull/80)   \nGGML: Strict Policy on AI-assisted code contributions. I'll try and get around  \nto hand-writing an issue for them. Highly suggest one of the forks above as an alternative.\n\nThe patch is 3 lines.\n\n\\## Body\n\nA few days ago I was benchmarking buun's new KV-cache codec on my quad-P100 box and comparing notes with the  numbers buun was getting on his 3090 — the same model kept showing systematically different quality floors between our machines. I thought everything was the same? Normally, I'd probably just think there's way too many variables between all this code to chalk it up to any one thing… I decided it was worth chasing. It was. And it led me to a serious bug that's been sitting in llama.cpp for years.\n\nSo I measured it. Against fp32-reference logits (KL divergence over the full distribution, Qwen3.6-27B, wikitext-2):\n\nHeadline:\n\n\\- \\*\\*Median KLD: 0.0023 → 0.000001\\*\\* (\\~2300× tighter)  \n\\- \\*\\*Top-token agreement: 96.5% → 99.9%\\*\\* — stock, about 1 in 29 of the model's next-token  \n  picks were different from what the math says they should be\n\n\\*\\*What's the extra math cost me in performance?\\*\\* I benchmarked prefill and decode at 8k depth on  \nthree model classes (27B hybrid, 4B dense, 36B MoE). The prefill was identical within noise on all  \nthree, decode actually \\~1.4% \\*faster\\* patched. The \"fast\" path was buying nothing: real  \nworkloads on P100 are bound by GEMM and memory bandwidth, not the fp16 vector path. The  \npatch is 3 lines, extending the exact same exemption sm\\_61 already had.\n\n\\*\\*Before everyone freaks out and assumes their 4090 is broken\\*\\* this is \\*measured on sm\\_60 only\\*.   \nYour GTX 1080/P40 was always fine (already exempt). Volta and newer are untouched by this patch and  \nrun different kernels entirely — whether other arches have their own unmeasured precision  \nstories is a separate research question I'm still digging into. Don't read this as \"all  \nGPUs are broken\"; read it as \"one specific GPU was, and now it isn't.\"\n\nPost Production Edit - TheTom: \"Verified on my side before merging: the three gates are the only 600-vs-610 distinction anywhere in the CUDA tree, so the carved sm\\_60 path is preprocessor-identical to the long-proven sm\\_61 path, and a Blackwell build showed bit-identical PPL with decode unchanged, confirming zero effect on other arches.\"\n\n\\*\\*Why you might care in 2026:\\*\\* P100s are going for \\~$80 shipped right now while the DRAM  \ncrisis sends everything else to the moon. 16GB of HBM2 at 732 GB/s. The market priced the P40 at \\~$300 partly because it \"runs better.\" Some of that gap was this bug.\n\n\\- Full technical writeup with methodology and receipts: [https://gist.github.com/apollo-mg/9218d50a209d70a85f033bf182657818](https://gist.github.com/apollo-mg/9218d50a209d70a85f033bf182657818)\n\nFound and isolated by running Fable 5 through my custom P/ReAct/R agent loop.   \nIt wrote the scripts, the hardware provided the receipts. Same workflow  \nas my KV-checkpoint sidecar patch ([https://www.reddit.com/r/LocalLLaMA/s/VTIwEFpYgc](https://www.reddit.com/r/LocalLLaMA/s/VTIwEFpYgc))  \nthat got merged into turboquant last week.\n\n\n\n--- Top Comments ---\n\n\n[102 upvotes] This is one of those 'little' things that people do that unequivocably make the world just a bit better.  It all adds up.\n\n[50 upvotes] I asked my AI \"wtf is this dude talking about\".  Sorry, no offense.\n\nClaim: llama.cpp's CUDA path picks a fast fp16 math mode for GPUs flagged as having strong fp16 throughput. sm_61 cards (GTX 10-series, P40) were excluded from that fast path already. sm_60 (P100) was left in it, despite the P100 actually having real fp16 hardware — meaning it's doing more of the compute in lower-precision fp16 instead of fp32, which increases numerical error (quantization-like noise) without helping speed, because real workloads are bottlenecked on memory bandwidth/GEMM, not the fp16 vector unit. So the \"fast\" flag is buying nothing on this card and costing accuracy. A 3-line patch removes P100 from that fast-path exemption, and the author claims it cuts output-distribution error (KL divergence vs fp32 reference) by ~2300x with no measurable speed loss, sometimes even a hair faster.\n\nSo yeah - sounds like a real improvement.\n\n[34 upvotes] Damn you're on 🔥\n\n[32 upvotes] Great work.  Applied the patch and here are my results:\n\nbuild             │ mean KLD │ same top-token\n\nstock P100        │ 0.0122   │ 95.09% \n\npatched (control) │ 0.000000 │ 99.997%\n\n",
  "transcript_chars": 5382,
  "ingested_at": "2026-07-12T13:30:06.537812+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 154,
    "upvote_ratio": 0.83,
    "num_comments": 25,
    "author": "apollo_mg",
    "is_self": true
  }
}