{
  "video_id": "reddit_1ulymml",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "llamacpp patch - DeepSeek V4 Flash running with full 1M token context locally on RTX 5090",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1ulymml/llamacpp_patch_deepseek_v4_flash_running_with/",
  "external_url": null,
  "upload_date": "20260702",
  "published_at": "2026-07-02T23:54:02+00:00",
  "transcript": "Wanted to try running DeepSeek V4 Flash locally but found it asking for absurd amounts of VRAM at higher context lengths (\\~256GB at 1M). Turned out the DSA lightning indexer lacks proper llamacpp support. Did a bit of digging and there's an upstream PR to address the issue (shoutout [u/fairydreaming](https://www.reddit.com/user/fairydreaming/), PR [\\#24231](https://github.com/ggml-org/llama.cpp/pull/24231)), but even there it's not wired into the model graph and has no CUDA path yet. So I wired it in and patched a CUDA kernel this morning and figured I'd share in case it's useful to anyone else looking to run something like this.\n\n**Hardware:** RTX 5090, 9950X3D, 96GB DDR5\n\n**Model:** [DeepSeek-V4-Flash, mixed Q8/Q4/Q2 quant by antirez](https://huggingface.co/antirez/deepseek-v4-gguf/blob/main/DeepSeek-V4-Flash-Layers37-42Q4KExperts-OtherExpertLayersIQ2XXSGateUp-Q2KDown-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-fixed.gguf)\n\n**Before / after (256K context):**\n\n|Before|After|\n|:-|:-|\n||||\n|Compute buffer|\\~67 GiB (OOM)|3.2 GiB|\n|Prefill|56 t/s|\\~263 t/s|\n|Decode|\\~14 t/s|\\~14 t/s|\n|1M context|impossible (\\~256GB)|works (3.75 GiB at ubatch 768, \\~9gb at 2048)|\n\n**Validated presets:**\n\n|Context|Prefill|Decode|Peak VRAM|\n|:-|:-|:-|:-|\n|||||\n|256K|\\~263 t/s|14 t/s|\\~29 GiB|\n|512K|256 t/s|13.7 t/s|\\~28 GiB|\n|1M|159 t/s\\*|13.7 t/s|\\~31 GiB|\n\n\\*lower ubatch on 32gb 5090 at 1M - should be \\~full speed if given the full \\~9gb vram\n\nCorrectness: verified briefly with a needle-in-haystack test - planted a random fact at 10%/50%/90% depth in a 100K-token document, model retrieved it correctly every time. Also retrieved correctly at 512K and 1M's harder 50% depth.\n\nSource + build instructions + full writeup: [https://github.com/spencer-zaid/llama.cpp/blob/deepseek-lid-cuda/docs/deepseek-v4-lid-cuda.md](https://github.com/spencer-zaid/llama.cpp/blob/deepseek-lid-cuda/docs/deepseek-v4-lid-cuda.md)  \nBranch: [https://github.com/spencer-zaid/llama.cpp/tree/deepseek-lid-cuda](https://github.com/spencer-zaid/llama.cpp/tree/deepseek-lid-cuda)\n\nNo prebuilt binary (single GPU tested RTX 5090). Build instructions in the doc in case you need them\n\n\n\n--- Top Comments ---\n\n\n[8 upvotes] This looks amazing!\n\n[5 upvotes] commendable work, curious what TTFT and/or the tg-end2end looks like?\n\n[5 upvotes] dude ds4 flash is incredible model to run a single 5090\n\n[2 upvotes] I've been following that lightning indexer fix in llama.cpp. You are a champion, sir. Now to see about porting it to Metal...",
  "transcript_chars": 2503,
  "ingested_at": "2026-07-03T01:30:02.052482+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 80,
    "upvote_ratio": 0.97,
    "num_comments": 15,
    "author": "da_dragon321",
    "is_self": true
  }
}