{
  "video_id": "reddit_1tr9vzn",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "Qwen3.6-27B Quantization Benchmark",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1tr9vzn/qwen3627b_quantization_benchmark/",
  "external_url": null,
  "upload_date": "20260529",
  "published_at": "2026-05-29T17:53:19+00:00",
  "transcript": "Hi everyone!\n\nThis is my attempt to benchmark and compare the quality of some of the well known Qwen3.6 27B quantizations on HuggingFace (unsloth, mradermacher, IQ4\\_XS from cHunter789 and Ununnilium), from Q8 all the way down to Q2.\n\n# Measurement method\n\nI'm using llama.cpp's `llama-perplexity` to measure the **mean KLD** and **Same Top P Percentage** between the quantized model and the base (BF16 version).\n\nAll runs were using the same context length of 8192 tokens, KV cache quantized to q8\\_0 so I can make sure the entire model fit in the GPU.\n\n# Understand KLD and Same Top P\n\nTo understand the test result, it would be useful to understand the difference between the two metrics I used.\n\nWhen an LLM predicts the next word of a given prompt, for example **\"Today I will do my\"**, it looks at its entire vocabulary and assigns a confidence score to every single token. Then samples the top tokens and pick the final one, based on the given temperature.\n\n* **KL Divergence (KLD)** measures how much the confidence distribution of the quantized model drifts away from the base. In this example, the base model might assign 90% confidence to \"homework\", 5% to \"bike\" and 1% to \"banana\". But the poorly quantized one might give 50% to \"homework\", 30% to \"bike\" and \"20%\" to \"banana\".\n* **Same Top P** tracks how often the quantized model picks the same token as the base model. In this example, the model might just pick \"homework\" as the next token for the prompt.\n\nSo, while you might get a good token choice with the quantized model (**Same Top P** is high), it's important to look at the **Mean KLD** to see how stable the inner probability of the model is, the lower, the better.\n\n# Benchmark result\n\n# Unsloth's quantization\n\nhttps://preview.redd.it/awcfprb5744h1.png?width=3600&format=png&auto=webp&s=3ac8937eeac49b6b4d3920cd2b4b52e99a25e269\n\nNothing special, higher quants are better than lower quants. Q6 to Q8 are pretty much lossless. You can see Q8\\_0 has a higher **Same Top P**, but underlying, the **Mean KLD** tells us that UD-Q8\\_K\\_XL is better. Anything below Q4 are for the desperate, like the 5060ti 16GB club.\n\nThe 4-bit cluster is a bit more interesting. Different people may have a different take on this, but to me, Q4\\_K\\_XL is a good quality-compromise if you can afford the VRAM. If you're tight, IQ4\\_XS could serve you well, IQ4\\_NL is not much difference. And in that case, there's no need to stretch for Q4\\_K\\_M. You can skip Q4\\_K\\_S.\n\nFrom Q3\\_K\\_XL, the quality degradation is more drastic. The KLD went all above 0.1 and matching token selection dropped to 90-85% can tell a lot about the instability.\n\n# mradermacher's and other quants\n\nI've seen people mention mradermacher's i1 quants here and there, and also IQ4\\_XS quants from cHunter789 and Ununnilium. I have been personally using Ununnilium's IQ4\\_XS for a while now. So I want to put them all on the same table to see how they fit. But a single diagram will not be enough so I will break them into 4 groups: Q8-Q6, Q5, Q4 and Q3-below.\n\n# 8-bit and 6-bit quantization\n\nhttps://preview.redd.it/6om7k1x6744h1.png?width=1600&format=png&auto=webp&s=28c6b79b867976de16a01b39b5dd20d422d77762\n\nmradermacher's Q6\\_K seems to be a clear winner over Unsloth's Q6\\_K here. The mean KLD is near perfect (0.027352), and 97.011% token selection match.\n\n# 5-bit quantization\n\nhttps://preview.redd.it/j7cs0cs7744h1.png?width=1600&format=png&auto=webp&s=8a8ba0e99a2c275034de0d7ebb357c1adfbed7cd\n\nIn this group, Unsloth is a winner. With about 300-500MB difference in size, you can skip Q5\\_K\\_S and go for Q5\\_K\\_M. Unsloth's Q5\\_K\\_M is clearly better in both matching token selection and KLD.\n\n# 4-bit quantization\n\nhttps://preview.redd.it/ywleki49744h1.png?width=3300&format=png&auto=webp&s=5db6b1d3899171afad5093557f849539332ea33d\n\nUnsloth beats all of the 4-bit quants here. But if you are looking for some alternative quants to save VRAM, like ones on 16GB, pay attention to IQ4\\_XS (it will help but of course, you will not be able to get above 65k context window).\n\nmradermacher's IQ4\\_XS is a clear winner among all the other IQ4\\_XS quants, but at 15.1 GB, it would be a bit tight. cHunter's IQ4\\_XS is also very good at 14.7 GB.\n\n# 3-bit and below\n\nhttps://preview.redd.it/fgjixv7a744h1.png?width=3300&format=png&auto=webp&s=45d85e85e57cfb7da11fbff2b5f4172634e20a1e\n\nAgain, mradermacher's quants filled in the gap between Unsloth's quants here, so you get a bit more choice, but tbh, at this range, you better off with Unsloth's Q3\\_K\\_XL or at least Q3\\_K\\_M.\n\nI was very interested to see how some new quants like IQ3\\_S, IQ3\\_M perform, but they turned out a bit disappointed.\n\n# Raw benchmark data\n\nIf you are interested, here's the raw benchmark data table after all the run.\n\n|Quantization|Mean PPL(Q)|Mean KLD|RMS Δp (%)|Same top p (%)|\n|:-|:-|:-|:-|:-|\n|UD-Q8\\_K\\_XL|6.569706|0.015495|2.448|97.407|\n|Q8\\_0|6.567807|0.020497|2.701|97.753|\n|UD-Q6\\_K\\_XL|6.541421|0.023398|2.903|97.436|\n|mradermacher/Q6\\_K|6.541627|0.027352|3.045|97.011|\n|Q6\\_K|6.566514|0.027766|3.014|97.112|\n|UD-Q5\\_K\\_XL|6.625155|0.045526|4.021|96.187|\n|Q5\\_K\\_M|6.658295|0.05277|4.26|95.864|\n|mradermacher/Q5\\_K\\_M|6.630279|0.053246|4.372|95.664|\n|mradermacher/Q5\\_K\\_S|6.613859|0.055034|4.476|95.505|\n|Q5\\_K\\_S|6.652629|0.055888|4.414|95.674|\n|UD-Q4\\_K\\_XL|6.647006|0.06656|5.023|94.621|\n|Q4\\_K\\_M|6.672841|0.070345|5.334|94.228|\n|IQ4\\_NL|6.619131|0.071724|5.497|94.106|\n|IQ4\\_XS|6.61994|0.072223|5.481|94.016|\n|mradermacher/IQ4\\_XS|6.611545|0.073705|5.648|93.852|\n|mradermacher/Q4\\_K\\_M|6.685347|0.074124|5.507|94.08|\n|cHunter/IQ4\\_XS-i1|6.656157|0.075933|5.645|93.77|\n|Q4\\_K\\_S|6.690623|0.078947|5.72|93.833|\n|mradermacher/Q4\\_K\\_S|6.642023|0.080407|5.825|93.657|\n|Ununnilium/IQ4\\_XS-pure|6.765894|0.084115|6.127|92.407|\n|UD-Q3\\_K\\_XL|6.620281|0.105386|7.077|91.837|\n|Q3\\_K\\_M|6.453757|0.129404|7.893|90.437|\n|mradermacher/Q3\\_K\\_L|6.482496|0.136127|8.116|90.213|\n|mradermacher/Q3\\_K\\_M|6.481299|0.140487|8.424|89.934|\n|mradermacher/IQ3\\_XS|6.981601|0.161364|9.182|88.767|\n|UD-IQ3\\_XXS|6.994512|0.176688|9.626|87.953|\n|mradermacher/IQ3\\_S|7.405328|0.176782|9.637|88.689|\n|Q3\\_K\\_S|7.068685|0.178631|9.61|87.681|\n|mradermacher/IQ3\\_M|7.454224|0.180647|9.824|88.603|\n|mradermacher/Q3\\_K\\_S|6.910989|0.181172|9.82|87.422|\n|UD-Q2\\_K\\_XL|7.316461|0.229068|11.399|85.95|\n|UD-IQ2\\_M|7.468708|0.241252|11.91|85.319|\n|UD-IQ2\\_XXS|8.507239|0.40986|16.708|78.483|\n\nThere are many more Qwen3.6 27B quantizations on HuggingFace, like ones from bartowski, huihui,... within my time budget (not money budget, since I'm basically using modal.com's free monthly credit :P), I cannot benchmark them all.\n\nIf you are interested in doing your own benchmark, I also attached the script in my original blog post, so you can run it on your own.\n\nSee it here: [https://www.huy.rocks/everyday/05-29-2026-ai-qwen3-6-27b-quantization-benchmark](https://www.huy.rocks/everyday/05-29-2026-ai-qwen3-6-27b-quantization-benchmark)\n\nWould love to see the result if any of you decided to run on your own.\n\nThanks for reading this far!\n\n\n\n--- Top Comments ---\n\n\n[62 upvotes] I don’t understand anything from that. I asked my qwen3.6-27B Q2_K_S and he also have no idea what’s going on…\n\n[15 upvotes] Thank you for the bench! You should be aware of benchmark limitations:\n\n\n1. It uses small context window – 8192 is usable for chat, but does not represent agentic usecases, working with big documents.\n\n\n2. It uses pretty limited, but fast to compute mean KLD + Top-K metrics. The real question is how this affects model output in a practical sense. \n\n\nUnsloth have used \"flip\" metric, for example – does the quantization flips the model response in the benchmarks?\n\n\nTop-K and KLD do not represent benchmark performance, unfortunately. Maybe they have close correlation, but I have not seen a proof of that.\n\n\n3. Q8 cache quant is understandable, but it also will limit performance in the long run. I do not know how it affects models with different quants, however. Quntization can regularize outputs, make them more predictable. \n\n\nIn turboquant discussion Georgi Gerganov (llama.cpp maintainer) he has shared AIME25 benchmarks results which show performance degradation of the context quant. Q8 with rotation (default in llama.cpp now) looks solid, though – https://github.com/ggml-org/llama.cpp/pull/21038#issuecomment-4150413357\n\n[5 upvotes] Here is the link to the script if you want to run it on your own:\n\n[https://gist.github.com/huytd/ac6457b4581598a198c027e4051380de](https://gist.github.com/huytd/ac6457b4581598a198c027e4051380de)",
  "transcript_chars": 8585,
  "ingested_at": "2026-05-30T01:30:07.379761+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 128,
    "upvote_ratio": 0.96,
    "num_comments": 47,
    "author": "bobaburger",
    "is_self": true
  }
}