{
  "video_id": "reddit_1usacge",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "Deepseek V4 Flash on a single RTX 6000 Pro - vLLM-Moet",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1usacge/deepseek_v4_flash_on_a_single_rtx_6000_pro/",
  "external_url": null,
  "upload_date": "20260710",
  "published_at": "2026-07-10T02:06:03+00:00",
  "transcript": "Wow...\n\n[https://github.com/kacper-daftcode/vLLM-Moet](https://github.com/kacper-daftcode/vLLM-Moet)\n\nUsing this customized vllm provided as a docker, I'm able to run DS V4 Flash on a single RTX 6000 Pro (apparently it also works on a single 5090 - check his readme, but I haven't tried). Apparently this also works with GLM 5.2 (though you need at least two 6000 pros, which is still amazing).\n\nSetting 130K context, I needed around 150 GB of RAM to get past the safetensor sharding, but once it is fully loaded in VRAM I am able to fit it all in the GPU (If you have less than this much RAM, create/extend your swap so you can get past the loading stage, but expect to wait 20 mins for the initial load).\n\nI'm running some benchmarks (see below) - and as I haven't run DS V4 before, not sure which parameters I should be using with this customized engine.\n\nBut just wanted to share - this is a very interesting feat by this developer as the magic sauce to get it this small to fit in smaller VRAM is compression of routed experts to 2 bit while keeping fp4 experts (kudos to him, a genius no doubt), but more testing required to see how usable it is (will look to do some coding sessions with it).\n\nInitial run - Single RTX 6000 Pro\n\n    | model             |            test |               t/s |       peak t/s |          ttfr (ms) |       est_ppt (ms) |      e2e_ttft (ms) |\n    |:------------------|----------------:|------------------:|---------------:|-------------------:|-------------------:|-------------------:|\n    | deepseek-v4-flash |            tg32 |    128.58 ± 11.43 | 132.73 ± 11.80 |                    |                    |                    |\n    | deepseek-v4-flash |  ctx_tg @ d4096 |     103.21 ± 9.94 | 106.54 ± 10.26 |                    |                    |                    |\n    | deepseek-v4-flash |    tg32 @ d4096 |    111.93 ± 18.57 | 115.54 ± 19.17 |                    |                    |                    |\n    | deepseek-v4-flash |  ctx_pp @ d8192 | 7907.32 ± 9331.94 |                |  5257.95 ± 2446.23 |  3813.06 ± 2446.23 |  5257.95 ± 2446.23 |\n    | deepseek-v4-flash |  ctx_tg @ d8192 |    110.42 ± 11.02 | 113.98 ± 11.37 |                    |                    |                    |\n    | deepseek-v4-flash |    tg32 @ d8192 |    129.20 ± 30.24 | 133.37 ± 31.21 |                    |                    |                    |\n    | deepseek-v4-flash | ctx_pp @ d16384 | 5057.32 ± 2173.55 |                |  5396.66 ± 2464.32 |  3951.77 ± 2464.32 |  5396.93 ± 2464.13 |\n    | deepseek-v4-flash | ctx_tg @ d16384 |    111.62 ± 16.28 | 115.22 ± 16.80 |                    |                    |                    |\n    | deepseek-v4-flash |   tg32 @ d16384 |     107.76 ± 4.27 |  111.24 ± 4.41 |                    |                    |                    |\n    | deepseek-v4-flash | ctx_pp @ d32768 |    2613.02 ± 9.10 |                |   12685.53 ± 25.40 |   11240.64 ± 25.40 |   12686.86 ± 25.37 |\n    | deepseek-v4-flash | ctx_tg @ d32768 |    103.65 ± 17.46 | 106.99 ± 18.03 |                    |                    |                    |\n    | deepseek-v4-flash |   tg32 @ d32768 |     100.84 ± 3.82 |  106.61 ± 7.36 |                    |                    |                    |\n    | deepseek-v4-flash | ctx_pp @ d65535 |  3856.66 ± 580.03 |                | 17068.97 ± 2595.20 | 15624.08 ± 2595.20 | 17070.57 ± 2595.74 |\n    | deepseek-v4-flash | ctx_tg @ d65535 |    118.46 ± 18.93 | 126.93 ± 23.61 |                    |                    |                    |\n    | deepseek-v4-flash |   tg32 @ d65535 |    119.35 ± 18.77 | 134.06 ± 25.60 |                    |                    |                    |\n\n\n\n--- Top Comments ---\n\n\n[12 upvotes] this guy also has a qwen3.6-27b engine that does full 256K context on a 32GB blackwell at FP6. \n\nhow come no one mentioned him before.\n\n[14 upvotes] Wow....\n\nThis completely unrelated genius developer is amazing! \n\nCan you share their Linkedin? \n\nI want to ask them if they could write a tutorial on how to properly paste markdown into a reddit post. \n\n\n\n[4 upvotes] > I'm able to run DS V4 Flash on a single RTX 6000 Pro (apparently it also works on a single 5090 - check his readme, but I haven't tried).\n\nIt works on DDR4+Core i7 too - I have tried.\n\n[3 upvotes] Tried running with 128Gb ram and an RTX pro 6000 but ran out of RAM :(",
  "transcript_chars": 4359,
  "ingested_at": "2026-07-10T13:30:19.146415+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 53,
    "upvote_ratio": 0.87,
    "num_comments": 20,
    "author": "live4evrr",
    "is_self": true
  }
}