{
  "video_id": "reddit_1vq6fdj",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "Qwen3.8-27b on RTX 3090 - 82 tps single request, up to 672 tps peak",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1vq6fdj/qwen3827b_on_rtx_3090_82_tps_single_request_up_to/",
  "external_url": null,
  "upload_date": "20260816",
  "published_at": "2026-08-16T19:38:07+00:00",
  "transcript": "Hi,\n\nAfter a long night of optimizations, I believe I have made the fastest inference engine for Qwen3.6-28B on a 3090.\n\nQuick metrics:\n\n\\- 250w power capped\n\n\\- Up to 195k context (ships with 150k for safety though)\n\n\\- 82 tps single request, 417 tps sustained with 64 concurrent\n\n\\- Between 17% to 149% faster than ninfer depending on the amount of concurrent requests.\n\nQuick how:\n\n\\- W4A16 quantization -> 16.8gb in vram - cache 66k\n\n\\- + fp8 KV cache -> 16.8 gb in vram - cache 155k\n\n\\- + lm\\_head int8 -> 15.4 gb in VRAM - cache 192k\n\n\\- + embed\\_tokens int8 -> 14.2 gb in VRAM - cache 200k\n\nQuantization loss of 0.6% in the lm head and quant embed compared to bf16.\n\nIt runs via vLLM and needs a few patches to work perfectly, but should be easier to setup than ninfer.\n\nAlso only tested on linux, but should work on windows too.\n\n[https://github.com/syv-ai/qwen38-27b-rtx3090](https://github.com/syv-ai/qwen38-27b-rtx3090)\n\n\n\n--- Top Comments ---\n\n\n[86 upvotes] \"I believe I have made the fastest inference engine for Qwen3.6-28B on a 3090.\"\n\nThat's when I knew an human wrote this post\n\n[14 upvotes] RTX 3090 owners continuing to win every single day\n\n[12 upvotes] \\> Quality loss is about 0.6% for a round trip.\n\nHow did you measure it?\n\n\\> I have made the fastest inference engine  \n  \nThat's a nice work of course, but you modified vllm actually?..",
  "transcript_chars": 1360,
  "ingested_at": "2026-08-17T01:30:03.989908+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 66,
    "upvote_ratio": 0.88,
    "num_comments": 38,
    "author": "iamMess",
    "is_self": true
  }
}