{
  "video_id": "reddit_1wgiefk",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "Running Qwen3.8-Flash-Next locally on a 12GB VRAM card",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1wgiefk/running_qwen38flashnext_locally_on_a_12gb_vram/",
  "external_url": "https://carteakey.dev/blog/running-qwen3-8-flash-next-locally/",
  "upload_date": "20260914",
  "published_at": "2026-09-14T22:34:29+00:00",
  "transcript": "Now that the dust has settled a bit - here's a write-up on running Qwen3.8-Flash-Next (125B-A6B MoE + 51B n-gram table) on  relatively middle-tier hardware (RTX 4070 12GB + 64GB DDR5-5600 + Gen4 NVMe on Linux).\n\nI started out with bare 6 tok/s and through latest patches and optimizations getting close to 20 tok/s generation. You just need enough RAM.   \n  \nFor me this is the most intelligence possible on this machine right now. The 27B dense is not a choice because of low VRAM but may make more sense for other configs like 24GB VRAM owners. It actually surpasses the 27B model on most tasks as well so its great for Low VRAM, High/fast RAM configs.\n\nPP is still a bit low at 300-350 tok/s. \n\n**What helped**  \n\\- Using AtomicChat's 4.27 bpw quant [https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/AtomicChat/Qwen3.8-Flash-Next-GGUF)   \n\\- Ngram SSD offloading (lazy-mode)  \n\\- --fit on --fit-target 512 helps automatically select the right params.\n\n\\- **Master branch (19.35 t/s):** Latest commit with MoE improvements.\n\n**- MTP Variant - PR #28243 + Compact MTP (20.65 t/s)**: MTP support is not yet merged so need to apply this PR enables Daniel Han's 1.78 GB \\`shared-Q4\\_K\\_M\\` compact head. Combined with \\`-ncmoe 45\\`, it yields 77–96% acceptance and breaks through the 20 t/s barrier on every tested task (coding, summarization, creative). \n\nWith such low VRAM, MTP is not a huge jump because you have to give up a few layers to store the MTP head in VRAM. Only the shared + Q4\\_K\\_M in MTP gets a beneficial uptick.\n\nUsing commercial models to research, optimize and benchmark inference for local models helps a ton (GLM 5.3 flash with opencode go, so did Astra, Gemini 3.8 etc.) \n\nLot more details in the post (AI-assisted).\n\n\n\n--- Top Comments ---\n\n\n[13 upvotes] Impressive. I'll give this a shot.\n\n20t/s is just barely feasible for overnight runs. \n\n[3 upvotes] Nice, I'm trying to configure it in a RTX 3060 12GB with 96GB RAM.\n\nWhat post?\n\nHow is your context? Is that quant better than Unsloth's? Why?\n\n[3 upvotes] Yea ddr5 is making alot of diference here, I have 3090 + 64gb ddr4 and I can't reliably get 20t/s even though my gpu should supposedly be better than the 4070 in this use case, the bigger throughput for the ddr5 makes the model feasible as long as a gpu has enough vram to get the active experts onto its vram since the speed gets bottlenecked by the RAM speed.",
  "transcript_chars": 2429,
  "ingested_at": "2026-09-15T01:30:04.141864+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 55,
    "upvote_ratio": 0.94,
    "num_comments": 10,
    "author": "carteakey",
    "is_self": false
  }
}