{
  "video_id": "reddit_1tkmgwj",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "Qwen-27B-IQ4_KS for ik_llama.cpp, especially for NVIDIA with 16GB VRAM",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1tkmgwj/qwen27biq4_ks_for_ik_llamacpp_especially_for/",
  "external_url": null,
  "upload_date": "20260522",
  "published_at": "2026-05-22T15:32:55+00:00",
  "transcript": "Hi everyone,\n\nI'm presenting a new quantization of the Qwen-27B model, created specifically with 16GB VRAM NVIDIA GPUs in mind. I used quants that, unfortunately, are not yet available in the main upstream `llama.cpp`. I'm talking about the KS and KSS quants developed by ikawrakow. After many trials, I managed to create a 14.1GB model which, in my testing, delivers results highly comparable to my previous 14.7GB IQ4_XS quantization.\n\n**Model Link:** [cHunter789/Qwen3.6-27B-i1-IQ4_KS-GGUF](https://huggingface.co/cHunter789/Qwen3.6-27B-i1-IQ4_KS-GGUF)\n\n**ik_llama.cpp Project:** [ikawrakow/ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp)\n\nUnfortunately, the `ik_llama.cpp` project required to run this model is **NVIDIA CUDA and CPU only**. There is currently no way to run this on AMD or Apple Silicon (Metal) :/\n\nUsing this model with `ik_llama.cpp` and a `Q4_0` Hadamard KV cache allows for a **105k context window**.\n\n### Benchmark Results & Real-World Impressions\nThe model was heavily tested in daily production workflows for several days. It runs much faster (1.5x-1.75x) and more reliably than the previous iteration—completely eliminating the issue of \"blank outputs\", while the search-replace functionality works flawlessly.\n\n* **Qwen Benchmark:** Successfully passed the performance evaluations on [qwen3-6-27b-benchmark.vercel.app](https://qwen3-6-27b-benchmark.vercel.app).\n* **Needle In A Haystack:** Successfully evaluated with satisfying results across the full 100k context window.\n* **Comparison:** In direct testing, this model performs slightly better than my previous variant: `Qwen3.6-27B-i1-IQ4_XS-GGUF`.\n\n### Perplexity (PPL) Testing\nPerplexity evaluations were conducted focusing exclusively on the KV Cache quantization setup (`q4_0`), as this is the primary target use case:\n\n```bash\nwget [https://www.gutenberg.org/files/2600/2600-0.txt](https://www.gutenberg.org/files/2600/2600-0.txt) -O pg19.txt\n\n./llama-perplexity -m Qwen3.6-27B.i1-IQ4_KS-attn_qkv-IQ4_KSS.gguf -f pg19.txt -c 65536 --chunks 32 -ngl 99 -khad -vhad -ctk q4_0 -ctv q4_0 -fa 1 -b 512 -ub 512\n```\n\n**Test Log Output:**\n```text\nperplexity: calculating perplexity over 12 chunks, n_ctx=65536, batch_size=512, n_seq=1\nperplexity: 71.10 seconds per pass - ETA 14.22 minutes\n[1]6.6897,[2]7.0032,[3]7.1989,[4]7.3327,[5]7.4816,[6]7.3770,[7]7.4325,[8]7.4378,[9]7.4754,[10]7.5192,[11]7.5669,[12]7.4040,\n\nFinal estimate: PPL over 12 chunks for n_ctx=65536 = 7.4040 +/- 0.02773\n```\n\n*Note: I currently do not have the capability to run KLD (Kullback–Leibler divergence) tests.*\n\n### Example Server Configuration\nFor reference, here is the server configuration I used during my tests:\n\n```bash\nllama-server \\\n        -m \"$MODEL_PATH\" \\\n        -a Qwen3.6-27B \\\n        --ctx-size 105000 \\\n        --chat-template-file chat_template.jinja \\\n        --n-gpu-layers 99 \\\n        --cache-type-k q4_0 \\\n        --cache-type-v q4_0 \\\n        --batch-size 512 \\\n        --ubatch-size 256 \\\n        --flash-attn on \\\n        --no-mmap \\\n        --host 0.0.0.0 \\\n        --port 8081 \\\n        --reasoning on \\\n        --reasoning-format deepseek \\\n        -t 8 \\\n        --parallel 1 \\\n        -khad \\\n        -vhad \\\n        --chat-template-kwargs '{\"preserve_thinking\": true}' \\\n        --defrag-thold 0.3 \\\n        --jinja \\\n        --cont-batching \\\n        --temp 0.15 \\\n        --top-k 1 \\\n        --min-p 0.1 \\\n        --repeat-last-n 512 \\\n        --repeat-penalty 1.05\n```\n\n```\n\n\n\n--- Top Comments ---\n\n\n[13 upvotes] Important to note that PPL actually barely moves with kv cache quants. KLD would show the degradation much faster. As much as I'd like to use 27B on my 16GB 5080, it's quite low quality no matter what you do. I'm preferring 35B at Q8 in terms of quality\n\n[5 upvotes] You’re the fking man\n\n[3 upvotes] Awesome work! Always wanted to create my own quants but lacking hardware. Also check ubergarm on hf. Im using his quants with MTP on ik_llama and i highly recommend it.\n\nhttps://huggingface.co/ubergarm/Qwen3.6-27B-GGUF\n\n[3 upvotes] Tks? Did you investigate mtp/dflash/n-gram? Also nvfp4?",
  "transcript_chars": 4101,
  "ingested_at": "2026-05-23T01:30:30.812436+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 57,
    "upvote_ratio": 0.91,
    "num_comments": 27,
    "author": "Pablo_the_brave",
    "is_self": true
  }
}