{
  "video_id": "reddit_1w6lk6z",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "Increasing active parameters per token in MOE (Qwen 35B A4B+) reduce reasoning token by 8.5% - and you don't need to train or finetune!",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1w6lk6z/increasing_active_parameters_per_token_in_moe/",
  "external_url": null,
  "upload_date": "20260903",
  "published_at": "2026-09-03T21:58:56+00:00",
  "transcript": "I want to share a short paper just published exploring a simple but surprisingly effective optimization for sparse MoE reasoning models.\n\n**The idea:** Instead of retraining anything, we just tweak the router at *runtime*. Specifically, we expand the expert selection budget (N≥K*N*≥*K*) **only in the late transformer layers**, with a linear decay factor applied to the extra experts. Early layers stay untouched. So Qwen 3.6 35B A3B becomes **Qwen 3.6 35B A4B+** !\n\n**What we found — \"Succinct Convergence\":**  \nWhen you give the model more expert capacity at the decision-critical final layers, it stops rambling. It reaches the ***same*** *correct answer* via significantly **shorter reasoning** trajectories.\n\n**Results on full MMLU-Pro (714 questions, Qwen3.6-35B-A3B):**\n\n* 📉 **8.5% reduction** in mean reasoning tokens\n* ⚡ **10.9% drop in latency** (p=6.5×10−6)\n* 🎯 **Accuracy unchanged** (*84.5% vs 84.0% native, p=0.77 — statistically indistinguishable*)\n* 🆓 **Zero training cost** — pure inference-time routing modification\n\n**Links:**\n\n* 📄 Paper: [https://zenodo.org/records/22255483](https://zenodo.org/records/22255483)\n\nthere you can also take a look to my github repo (with beta version code) and the detailed json results of MMLU-Pro benchmark.\n\nIn the future i hope i can make same experimentation with a larger model like DeepSeek V4 Flash Q2.0 \n\nI'm a Non-native english speaker, part of this post was generated , for translation reason with the help of AI.\n\n\n\n--- Top Comments ---\n\n\n[15 upvotes] Really nice. We are doing something that looks something like this, just a tad more elaborate.  \n  \nWe have found out somewhat the same thing you have found, ie. the model (and many like it) are not fine tuned at all. Lots of room for improvement.  \n  \nHopefully we will release something late september. This is though a very straightforward way to reduce compute.\n\nGreat work!\n\n[9 upvotes] I'd use Gemma 26BA12B\n\n[9 upvotes] Interesting ! At which point does adding experts stop providing improvements ?",
  "transcript_chars": 2022,
  "ingested_at": "2026-09-04T01:30:03.610834+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 71,
    "upvote_ratio": 0.97,
    "num_comments": 15,
    "author": "Specific-Tax-6700",
    "is_self": true
  }
}