{
  "video_id": "reddit_1w5isz3",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "Confirmed bolting Q8 NGram into IQ4 Qwen no speed degradation",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1w5isz3/confirmed_bolting_q8_ngram_into_iq4_qwen_no_speed/",
  "external_url": null,
  "upload_date": "20260902",
  "published_at": "2026-09-02T18:32:00+00:00",
  "transcript": "This came from another thread or comment. I forgot exactly where, but the basic idea was to replace the 51B N-gram layer in Qwen 3.8 Next with a much higher precision version.\n\nSomeone running a 5090 replaced the N-gram portion of their Qwen 3.8 UD Q4 model with BF16.\n\nSince I'm already running IQ4\\_XS, I wanted to try something similar. I don't have enough storage for the BF16 N-gram weights right now, so I replaced the lower-precision N-gram portion with Q8 instead.\n\nI'm still testing whether it improves the actual model output, but in terms of inference speed, the difference appears to be very small.\n\nBefore bolting on the Q8 N-gram:\n\n    n_gen = 2588, tg = 8.82 t/s, tg_3s = 8.81 t/s\n    n_gen = 2616, tg = 8.82 t/s, tg_3s = 9.19 t/s\n    n_gen = 2645, tg = 8.83 t/s, tg_3s = 9.37 t/s\n    n_gen = 2675, tg = 8.84 t/s, tg_3s = 9.78 t/s\n    n_gen = 2704, tg = 8.84 t/s, tg_3s = 9.50 t/s\n    n_gen = 2732, tg = 8.85 t/s, tg_3s = 9.06 t/s\n    n_gen = 2761, tg = 8.85 t/s, tg_3s = 9.32 t/s\n    n_gen = 2790, tg = 8.86 t/s, tg_3s = 9.50 t/s\n    n_gen = 2819, tg = 8.86 t/s, tg_3s = 9.39 t/s\n    n_gen = 2847, tg = 8.87 t/s, tg_3s = 9.22 t/s\n    n_gen = 2875, tg = 8.87 t/s, tg_3s = 9.15 t/s\n\nWith the Q8 N-gram bolted onto IQ4\\_XS:\n\n    n_gen = 263, tg = 10.76 t/s, tg_3s = 11.04 t/s\n    n_gen = 294, tg = 10.70 t/s, tg_3s = 10.18 t/s\n    n_gen = 329, tg = 10.76 t/s, tg_3s = 11.32 t/s\n    n_gen = 361, tg = 10.73 t/s, tg_3s = 10.46 t/s\n    n_gen = 395, tg = 10.76 t/s, tg_3s = 11.11 t/s\n    n_gen = 430, tg = 10.81 t/s, tg_3s = 11.37 t/s\n    n_gen = 460, tg = 10.75 t/s, tg_3s = 9.91 t/s\n    n_gen = 494, tg = 10.77 t/s, tg_3s = 11.16 t/s\n    n_gen = 531, tg = 10.86 t/s, tg_3s = 12.15 t/s\n\nSo far, there doesn't seem to be any meaningful speed penalty from replacing the lower-precision N-gram layer with Q8. The actual output quality is still being tested. It looks like Q8 Ngram is faster, but at the end both are **steady state at 10.1\\~ tok/s -ish.**\n\nAnd without MTP since i am using earlier merge from unsloth,  \nRAM 96GB DDR4 (32x3 Three channel active 2400MHZ)  \nXeon e5 2690v4  \nRTX 3090 capped at 250 W\n\nEdit: the state\\_dict grew from 90\\~ GB to 115GB  \nEdit 2: Fck me i forgot to put the code, for stitching it, wait 8 hours from now i'll upload it since it is already very very late at night\n\n\n\n--- Top Comments ---\n\n\n[17 upvotes] Yes I was wondering about quantization of that part, probably it makes sense to not quantize it (or use more bits)\n\n[5 upvotes] maybe not related but why your ram 96gb 3 channel? will it have more speedup if you go 4 channel?",
  "transcript_chars": 2576,
  "ingested_at": "2026-09-03T01:30:03.291478+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 52,
    "upvote_ratio": 0.96,
    "num_comments": 15,
    "author": "Altruistic_Heat_9531",
    "is_self": true
  }
}