{
  "video_id": "reddit_1ucqs1k",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "Top-N-Sigma: Remove unconditional softmax+sort by TimNN · Pull Request #22645 · ggml-org/llama.cpp",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1ucqs1k/topnsigma_remove_unconditional_softmaxsort_by/",
  "external_url": "https://github.com/ggml-org/llama.cpp/pull/22645",
  "upload_date": "20260622",
  "published_at": "2026-06-22T17:18:14+00:00",
  "transcript": ">\n\n>**Overview**  \nCurrently, the Top-N-Sigma sampler does an unconditional softmax+sort at the end.  \nIn the (common, I believe) case of Top-N-Sigma being followed by Dist, this expensive work is completely wasted.\n\n>**Additional information**  \nOn my M3 Max MacBook Pro, this PR increases the t/s for `google_gemma-4-E4B-it-Q8_0` **by 50%, from \\~30t/s to \\~45t/s, reducing the time per token by 10ms**.  \n(I'm not sure about the exact API contract between chained samplers and don't know if this might adversely affect other sampler chains that might rely on the current behavior).\n\nThat's a good % & t/s. Wish this had more t/s stats with few more models.\n\nSomebody please give us Tiny ELI5 version for this if possible. But let us know whether  this is applicable for all backends & all models? Thanks\n\n\n\n--- Top Comments ---\n\n\n[16 upvotes] basically it skips the softmax+sort when the next sampler (like dist) would just recompute it anyway. frees up gpu cycles for the actual generation. should work on any backend since it's just a sampling change, not an arch thing\n\n[11 upvotes] I'm amazed we still have such low hanging fruit in regards to optimizations. Really cool PR.\n\n[2 upvotes] Does it affect other backend",
  "transcript_chars": 1223,
  "ingested_at": "2026-06-23T01:30:04.692117+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 51,
    "upvote_ratio": 0.97,
    "num_comments": 10,
    "author": "pmttyji",
    "is_self": false
  }
}