{
  "video_id": "reddit_1uq0fpe",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "Qwen3.6-27B - Effect of KV quantization on KLD - Q8, Q6, Q5 (bartowski)",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1uq0fpe/qwen3627b_effect_of_kv_quantization_on_kld_q8_q6/",
  "external_url": null,
  "upload_date": "20260707",
  "published_at": "2026-07-07T16:39:09+00:00",
  "transcript": "[Lower is better - Quantization increases from right to left](https://preview.redd.it/rt8p71gj5ubh1.png?width=1962&format=png&auto=webp&s=1a9feef3f3c5d9c97e2a6c243758e5a646026721)\n\nI recently made a post [here](https://www.reddit.com/r/LocalLLaMA/comments/1unpelb/getting_close_to_100k_context_on_32gb_vram_with/) about how I squeezed more context into a Q8 model of bartowski's Qwen3.6-27B. My reasoning was that in my (anecdotal) experience, a Q8 has been performing a lot better than a Q6 or a Q5.\n\nThere were a lot of comments about quantizing KV of a higher model and some folks suggested just going with a lower quant like Q6 but with full unquantized KV. So I just wanted to test that hypothesis with KLD.\n\n**Base reference is Q8 with no KV quantization. That's because my 5090 only can fit a Q8.**\n\nHere are my findings. Detailed test setup and approach follow below.\n\n* Q8 does perform better than Q6 and Q5 (no surprises there)\n* Much wider gap between Q6 and Q5 than Q8 and Q6.\n* Q8 and Q6 have a steep drop the minute we put v at q4\\_0. Doesn't matter what quant we use for k.\n* If you have to use q4\\_0 for v, you might as well use (q8\\_0, q8\\_0) on Q6 quant (this really surprised me)\n* Q5 is more tolerant of v quantization than Q8 or Q6.\n* With (q4\\_0, q4\\_0), Q8 and Q6 converge.\n\n**Recommendation: Use whatever you can fit in VRAM, and just use (q8\\_0, q8\\_0). It's almost free.**\n\n**-------**\n\n**Test setup:**\n\nI used llama-perplexity to generate this data. My primary use case for this model is only for coding and primarily python. So I wanted to use a python sample file. Downloaded a bunch of open source coding repos (transformers, torch, huggingface etc) and concatenated the python source files to generate a massive 230MB text file.\n\nI wanted to use as high a context as my system could manage. I have a 5090 and 64GB RAM. Through trial and error, I could get up to 50K context and I just kept that for all the tests.\n\nIt seemed like the KLD improves and converges with higher number of chunks. So decided to use a chunk size of 32.\n\nUsed Qwen-3.6-27B (duh!) to put together a script to run all the different combinations. The command I used to generate the base logits was:\n\n    build/bin/llama-perplexity \\\n        -m ~/myp/models/bartowski_Qwen_Qwen3.6-27B-Q8_0.gguf \\\n        --temp 0.6 \\\n        --top_p 0.95 \\\n        --top_k 20 \\\n        --min_p 0.0 \\\n        --repeat-penalty 1.0 \\\n        --presence-penalty 0.0 \\\n        -c 50000 \\\n        -t 16 \\\n        -ngl 99 \\\n        --flash-attn on \\\n        -kvo -b 1024 -ub 256 \\\n        --kl-divergence-base ~/tmp/base_50k_coding.kld \\\n        --chunks 32 \\\n        -f python_corpus.txt\n\nOnce this completed, I added the additional flag `--kl-divergence` for the other runs to use this as the base.\n\nEach run took 17 minutes to complete and there were 23 runs in total, so ... uh ... it took a long time. \n\n**DISCLAIMER**\n\n* Learning as I go. Tell me if this is stupid or if I'm completely off base.\n* As benchmarks go, I think your experience matters more. I think very often we're afraid to trust our own instinct. A benchmark isn't gospel truth.\n* I don't know how important those distances are in the chart. End of the day, Q6 unquantized is 0.01 units away from Q8 unquantized. I don't know but that sounds like an insanely good compromise.\n* I still want to use Q8 model. From my own personal experience, I feel it understands better and writes better code.\n* I used Bartowski for no specific reason other than I have the models on my machine already. I have no opinion about Unsloth models. They may be better or worse for all I know.\n\n\n\n**Raw Data**\n\n|model|Q8\\_0|Q6\\_K\\_L|Q5\\_K\\_L|\n|:-|:-|:-|:-|\n|(no\\_kv,no\\_kv)|0|0.010771|0.0228|\n|(none,q8\\_0)|0.005399|0.01069|0.022322|\n|(q8\\_0,q8\\_0)|0.00541|0.010709|0.022486|\n|(q8\\_0,q5\\_1)|0.00736|0.011715|0.023135|\n|(none,q5\\_1)|0.007397|0.011648|0.023194|\n|(none,q4\\_0)|0.01164|0.014789|0.024295|\n|(q8\\_0,q4\\_0)|0.011824|0.014666|0.024101|\n|(q4\\_0,q4\\_0)|0.020817|0.022166|0.027909|\n\n\n\n--- Top Comments ---\n\n\n[23 upvotes] Good lord look at the KV graph. Almost no difference between f16 and q8\\_0 on both K and V at Q6 base quant, but immediate degradation with a Q8 base quant. Thanks for confirming that my current set up is \"as good as it gets for my VRAM\" too takes one thing off my plate.\n\n[13 upvotes] I really, really am interested in this topic, thanks OP. \n\n...but what I would really like to see, is the agentic performance at long context. Either some tool calling benchmark at \\~100k context, or complex task that goes into few dozens thousand tokens, like coding, with palpable/visual output. This is what people here claim is being hugely affected by the kv quantization itself. \n\n[8 upvotes] This confirms for me that q8\\_0,q5\\_1 is the sweet spot. If you have the time and interest, you could check kvarn6,kvarn6 kvarn6,kvarn5 kvarn5,kvarn5 kvarn5,kvarn4 and kvarn4,kvarn4 from the beellama fork. ",
  "transcript_chars": 4946,
  "ingested_at": "2026-07-08T01:30:27.240352+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 100,
    "upvote_ratio": 0.97,
    "num_comments": 56,
    "author": "BitGreen1270",
    "is_self": true
  }
}