{
  "video_id": "reddit_1tany5t",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "Drastically improve prompt processing speed for --n-cpu-moe partially offloaded models",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1tany5t/drastically_improve_prompt_processing_speed_for/",
  "external_url": null,
  "upload_date": "20260512",
  "published_at": "2026-05-12T02:12:28+00:00",
  "transcript": "# Bigger ubatch made gpt-oss-120b prompt processing much faster on my RTX 3090\n\nI was tuning `gpt-oss-120b-F16.gguf` with llama.cpp on a 24 GB RTX 3090 and found that increasing the physical micro-batch size (`-ub`) can massively improve prompt processing throughput, as long as you also raise `--n-cpu-moe` enough to keep the run inside VRAM.\n\nThe llama.cpp defaults are `-b 2048` and `-ub 512`; I included that default run as its own point in the chart.\n\nHere are the informal `llama-bench` results I charted:\n\n|ubatch|n-cpu-moe|prefill|generation|\n|:-|:-|:-|:-|\n|256|25|240.03 tok/s|33.14 tok/s|\n|512 (default)|26|380.27 tok/s|32.29 tok/s|\n|2048|25|1112.54 tok/s|32.96 tok/s|\n|4096|26|1682.47 tok/s|32.38 tok/s|\n|8192|28|2090.68 tok/s|30.05 tok/s|\n\nCompared with the llama.cpp default `-ub 512`, prompt processing went from about 380 tok/s to about 2091 tok/s, roughly a 5.5x gain. Compared with the smaller `-ub 256` run, it was about an 8.7x gain. Token generation dropped from about 32.3 tok/s at default settings to 30.1 tok/s at `-ub 8192`, about a 7% reduction.\n\nThe catch is that the larger ubatch needs more GPU compute workspace. On my machine, `-ub 4096` needed `--n-cpu-moe 26`, and `-ub 8192` needed `--n-cpu-moe 28`. So this is a throughput trade: move a few more MoE layers to CPU to make enough room for the bigger batch, and prompt-heavy workloads get dramatically faster while generation gets a little slower.\n\nhttps://preview.redd.it/s750judj7m0h1.png?width=2250&format=png&auto=webp&s=c696d26db310933120b9b99c310b2662e2d4f390\n\nNote: the first four prefill points are `pp4096`; the 8192 ubatch point is from a `pp8192` run, so treat this as an informal tuning result rather than a perfectly controlled benchmark.\n\n\\-----\n\nOne of the reasons I bought a DGX Spark was to have better prompt processing speeds. If I had known about this trick, I might not have done that in retrospect, even though it is a very nice machine, and still gets slightly better prompt processing performance and like double the token generation speed for gpt-oss-120b. Higher ubatch *drastically* closes the gap.\n\n\n\n--- Top Comments ---\n\n\n[12 upvotes] fwiw the reason -ub helps so much here is that with --n-cpu-moe your attention and router still run on the 3090 and those are the launch-overhead bound kernels during prefill. bigger ubatch means fewer kernel launches per chunk so the GPU stays saturated. generation doesn't move because that's one token at a time, you're memory-bandwidth bound on the CPU expert weights and that part doesn't care about -ub at all. nice writeup, this trick is buried in the llama.cpp issues.\n\n[5 upvotes] You are a legend, sir!  \nThis was the only thing that helped me.   \nEverything else just said turn flash attention on.   \nI was only using the -b flag, but as soon as i increased -ub, it became ideal.  \nThank you for your service!!!\n\n[4 upvotes] I mean the default -ub is set at 512 because it's a safe number to keep cards with lower amounts of VRAM from having memory spikes. If you have the VRAM you can adjust until you hit the saturation or VRAM limits. Once you're saturated the benefits stop and if you hit VRAM the dreaded OOM.\n\nThe baseline is set so there isn't  a million reddit posts saying \"Llama is GARBAGE all I get is OOM!\" LMAO. \n\nThere can also be thermal throttling with larger batch sizes, though this is mainly a unified memory issue. \n\nI only have an 8gb card and I ride the line so I always run 2048/512 on my models that take up 6gb+ and 2048/2048 on small models if it makes sense\n\nNice work though, I like to see posts with real test data.",
  "transcript_chars": 3602,
  "ingested_at": "2026-05-12T11:04:31.363301+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 69,
    "upvote_ratio": 0.94,
    "num_comments": 23,
    "author": "coder543",
    "is_self": true
  }
}