{
  "video_id": "reddit_1w1lq7u",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "Qwen 3.8 27B at 50 tok/s with 100k Context on a 16GB GPU! (beellama.cpp)",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1w1lq7u/qwen_38_27b_at_50_toks_with_100k_context_on_a/",
  "external_url": null,
  "upload_date": "20260829",
  "published_at": "2026-08-29T12:50:05+00:00",
  "transcript": "I wanted to share my successful setup for running a **Qwen 3.8 27B** model with a massive context window on a consumer 16GB GPU (RTX 4070 Ti SUPER). The goal was to fit everything into VRAM without sacrificing quality or speed.\n\n# 🧠 Key Components\n\n* **Model:** `Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller` from [jrell on Hugging Face](https://huggingface.co/jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller). It's a custom hybrid quantization specifically designed to fit Multi-Token Prediction (MTP) and long contexts into a 16GB VRAM budget.\n* **Chat Template:** I used the Jinja template from [peculiar-ragdoll's Qwen-Sharp-Chat-Templates](https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates). It helps use fewer thinking tokens without noticeably affecting quality, which is great for speed.\n* **Inference Engine:** This is crucial. I used **beellama.cpp** ([GitHub link](https://github.com/Anbeeld/beellama.cpp)) because it supports the `kvarn` KV cache types, which are key to this optimization.\n\n# 🖥️ Optimized llama-server Command (Windows)\n\nHere's the polished command I'm running. The magic is in the `kvarn` cache settings and the tail precision.\n\n    %LLAMA_DIR%/llama-server.exe ^\n    -m %MODEL_PATH% ^\n    -a %MODEL_NAME% ^\n    --port 11434 ^\n    --temp 1.0 ^\n    --top-p 0.95 ^\n    --top-k 20 ^\n    --min-p 0.0 ^\n    --presence-penalty 0.0 ^\n    --repeat-penalty 1.0 ^\n    --parallel 1 ^\n    --n-gpu-layers 99 ^\n    --batch-size 1024 ^\n    --ubatch-size 256 ^\n    --flash-attn on ^\n    --spec-type draft-mtp ^\n    --spec-draft-n-max 2 ^\n    --cache-type-k kvarn5 ^      <-- Key: Higher precision for K cache\n    --cache-type-v kvarn4 ^      <-- Key: Balanced precision for V cache\n    --kv-tail-tokens 1024 ^      <-- Keeps recent tokens at full precision\n    --ctx-size 100000 ^\n    --fit-ctx 100000 ^\n    --jinja ^\n    --chat-template-kwargs \"{\\\"preserve_thinking\\\": true, \\\"reasoning_effort\\\":\\\"medium\\\"}\" ^\n    --chat-template-file %MODEL_JINJA% ^\n    --no-mmproj-offload ^\n    --threads 7 ^\n    --threads-batch 8 ^\n    --metrics ^\n    --verbosity 3 ^\n    --perf\n\n# 📊 Results & Optimization Notes\n\n|Metric|Result|Note|\n|:-|:-|:-|\n|**Generation Speed**|**47-50 tokens/second**|Excellent for a 27B dense model.|\n|**Context Window**|**100,000 tokens**|Successfully pushed from 88k by optimizing the cache.|\n|**VRAM Usage**|\\~15.93 GB (70 MB free)|Perfectly tuned to the limit for maximum context.|\n|**KV Cache Type**|`kvarn5` (K) / `kvarn4` (V)|Uses the `kvarn` types from beellama. The asymmetric mix balances memory and quality.|\n|**Precision Tail**|`--kv-tail-tokens 1024`|This is key. It keeps recent tokens at higher precision, preserving output quality.|\n\n**What I Optimized:**\n\n1. **KV Cache Quantization:** Moving from `kvarn5/kvarn5` to `kvarn5/kvarn4` saved \\~6% VRAM, allowing the context size to increase from 88k to 100k with minimal quality loss.\n2. **Batch Sizes:** Set `--batch-size 1024` and `--ubatch-size 256` to balance prompt processing speed and VRAM usage.\n3. **Threading:** Adjusted `--threads 7` and `--threads-batch 8` for my Ryzen 7 CPU.\n4. **Speculative Decoding:** Using `--spec-type draft-mtp` with 2 draft tokens (the model supports this) gave a huge speed boost.\n\nThe near-lossless `kvarn` quantization for the KV cache is the real star here. It delivers q5-class fidelity at q4-class memory usage, which is incredible.\n\nHopefully, this helps anyone trying to squeeze maximum performance out of a 16GB card! Happy to answer any questions.\n\n\n\n--- Top Comments ---\n\n\n[7 upvotes] RTX 5080 16GB + 9800X3D, Qwen3.8-27B IQ4\\_XS-Smaller, BeeLlama, MTP off,\n\n130K ctx, kvarn4  \nI also set -ngl 67 (I started with 99 and gradually decreased it until I found the sweet spot at 67, which frees up more VRAM for context)  \n50 t/s\n\n    llama-server ^\n    -m \"F:\\.lmstudio\\models\\Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller.gguf\" ^\n    -c 130000 ^\n    -ngl 67 ^\n    -sm none ^\n    -fa on ^\n    -t 2 ^\n    -tb 2 ^\n    -b 512 ^\n    -ub 512 ^\n    --fit off ^\n    --parallel 1 ^\n    --temp 1.0 ^\n    --top-p 0.95 ^\n    --top-k 20 ^\n    --min-p 0.0 ^\n    --presence-penalty 0.0 ^\n    --repeat-penalty 1.0 ^\n    -ctv kvarn4 ^\n    -ctk kvarn4 ^\n    --chat-template-file \"F:\\.lmstudio\\models\\chat_template.jinja\" ^\n    --jinja ^\n    --reasoning-preserve ^\n    --no-mmproj-offload ^\n    --reasoning-format deepseek ^\n    --chat-template-kwargs \"{\\\"reasoning_effort\\\":\\\"xhigh\\\"}\"\n\n[2 upvotes] achieved 50 tps on a v100 16gb with unsloth iq4_xs mainline llama.cpp, limited context and no mtp/multimodal, q8 cache\n\n[2 upvotes] how does TG(tps) and PP change as the input context incrases from something small, up to say 64k?\n\n\n\n[1 upvotes] Nice! I’m gonna try this",
  "transcript_chars": 4676,
  "ingested_at": "2026-08-29T13:30:02.956416+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 55,
    "upvote_ratio": 0.95,
    "num_comments": 16,
    "author": "qaf23",
    "is_self": true
  }
}