{
  "video_id": "reddit_1tvff62",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "Another shout out to llama.cpp build b9455 2x3090",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1tvff62/another_shout_out_to_llamacpp_build_b9455_2x3090/",
  "external_url": null,
  "upload_date": "20260603",
  "published_at": "2026-06-03T05:05:43+00:00",
  "transcript": "https://preview.redd.it/xyvtkzwr005h1.png?width=645&format=png&auto=webp&s=aebd5b5ef79255247c9bc91fb69d8423a0c61f86\n\nAs you guys know, the next highest quant is Unsloth's /Qwen3.6-27B-UD-Q8\\_K\\_XL.gguf. With llama.cpp before, i was getting  30-50 tk/s.  vllm was kicking llama's ass with its tensor splits speeding up the 2x3090s at 70+ tk/s for months.  But I can't seem to find good quants for vllm and settle for some unknown qwen3.6-mtp-8.0...it was also making minor coding mistakes here and there... now being able to run unsloth's UDQ8KXL at 70+t/s, its code output are so clean, its like a different beast altogether.\n\nFinally got around to test out the llama ver b9455b with tensor-split, and holy f.  Results below:\n\n     llama.cpp server for Qwen3.6-27B-MTP UD-Q8_K_XL (MTP speculative decoding).\n    export LD_LIBRARY_PATH=/home/llama.cpp-b9455/build/bin:${LD_LIBRARY_PATH:-}\n    exec /home/llama.cpp-b9455/build/bin/llama-server \\\n      --host 0.0.0.0 --port 8000 \\\n      --model /home/projects/Qwen3.6-27B-MTP/Qwen3.6-27B-UD-Q8_K_XL.gguf \\\n      --n-gpu-layers 99 \\\n      --ctx-size 262144 \\\n      --parallel 1 --kv-unified \\\n      --batch-size 4096 \\\n      --ubatch-size 512 \\\n      --tensor-split 50,50 -sm tensor \\\n      --flash-attn on \\\n      --cache-type-k q8_0 --cache-type-v q8_0 \\\n      --spec-type draft-mtp \\\n      --spec-draft-n-max 3 \\\n      --jinja \\\n      --no-mmap \\\n      --temp 0.6 \\\n      --top-p 0.95 \\\n      --top-k 20 \\\n      --min-p 0.0 \\\n      --presence-penalty 0.0 \\\n      --metrics\n\n\\-------------------------------\n\nNo more watching paint dry:\n\n* `ctx` = true context (incl. cached) send\n* `pp` = prefilled tokens / prefill time / prefill t/s\n* `out` = decode tokens / decode time / decode t/s\n\nExample coding run below:\n\nctx 27K · pp 27K/18.8s 1417t/s · out 248/3.0s 81t/s · cold\n\nctx 31K · pp 3.8K/3.2s 1171t/s · out 353/4.7s 74t/s · 27K cached\n\nctx 37K · pp 6.7K/5.7s 1184t/s · out 335/4.5s 74t/s · 31K cached\n\nctx 43K · pp 5.5K/4.9s 1121t/s · out 357/5.0s 71t/s · 37K cached\n\nctx 44K · pp 1.3K/1.5s 861t/s · out 377/5.2s 72t/s · 43K cached\n\nctx 2.7K · pp 2.0K/1.5s 1294t/s · out 691/9.7s 71t/s\n\nctx 13K · pp 7.2K/5.0s 1421t/s · out 964/13.0s 73t/s · 5.5K cached\n\nctx 46K · pp 27K/19.8s 1370t/s · out 694/10.2s 67t/s · 19K cached\n\nctx 52K · pp 2.4K/2.6s 919t/s · out 464/6.9s 66t/s · 50K cached\n\nctx 58K · pp 6.5K/6.3s 1036t/s · out 101/1.5s 69t/s · 52K cached\n\nctx 60K · pp 2.1K/2.3s 889t/s · out 163/2.2s 74t/s · 58K cached\n\nctx 2.1K · pp 2.1K/2.3s 880t/s · out 1.9K/32.7s 57t/s\n\nctx 63K · pp 6.0K/4.8s 1266t/s · out 856/12.3s 69t/s · 57K cached · queue 1\n\nctx 7.3K · pp cached · out 4.5K/82.5s 54t/s · 7.3K cached\n\nctx 64K · pp 7.8K/5.6s 1402t/s · out 453/5.8s 78t/s · 57K cached\n\nctx 65K · pp 2.3K/2.8s 823t/s · out 99/1.4s 71t/s · 63K cached\n\nctx 65K · pp 120/0.4s · out 93/1.3s 70t/s · 65K cached\n\nctx 68K · pp 68K/54.2s 1247t/s · out 2.0K/28.8s 68t/s · cold\n\nctx 27K take 18.8s to fill cold.  ctx100K will take \\~60+s. Imagine every turn, waiting a minute.. or 5 minutes for pp to fill..\n\n\n\n--- Top Comments ---\n\n\n[8 upvotes] >Finally got around to test out the llama ver **b9455b** ....\n\nUse latest llama.cpp version for even better t/s. [PR 23861](https://github.com/ggml-org/llama.cpp/pull/23861) merged after above version. [PR 23988](https://github.com/ggml-org/llama.cpp/pull/23988) too.\n\n[7 upvotes] Amazing find. Will be benchmarking that build today on my nodes.\n\nedit: tried it, too bad I noticed that llamacpp still doesn't fully support 4x3090's out of the box so the performance was very bad. gemini suggests something called \"nccl\" will try that. Maybe it also explains why I was getting very low tps with mistral medium 128b dense on that rig.\n\nedit 2: I got 28tps on bf16 gguf model kv cache q8 while nccl=on... build: b9487, driver 595, cuda 13.2\n\n[2 upvotes] What pcie connection do you have between the cards? Also, did you use nvidia drivers that enable P2P?\n\n[2 upvotes] OMGoodness! I am trying b9445 right now with Qwen3.6-27B-MTP Q8\\_0, 256k context, q8\\_0 KV cache and they fixed the memory issues! Recovered about 2.5 GB of vram or 40k context. WOW! Thank you llama.cpp dev team!\n\nOn 2x RTX 3090 24G\n\n[2 upvotes] Are they connected via NVLink?\n\nI've been reading here that tensor-split will be slow without NVLink.",
  "transcript_chars": 4291,
  "ingested_at": "2026-06-03T13:30:05.444420+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 54,
    "upvote_ratio": 0.95,
    "num_comments": 28,
    "author": "Fabulous_Fact_606",
    "is_self": true
  }
}