{
  "video_id": "reddit_1utwqf8",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "Ultra budget 20GB vram with 448GB/s for $100 bucks.",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1utwqf8/ultra_budget_20gb_vram_with_448gbs_for_100_bucks/",
  "external_url": null,
  "upload_date": "20260711",
  "published_at": "2026-07-11T21:49:49+00:00",
  "transcript": "Here is the upper limit of what can be done with $100 bucks worth of video cards. \n\nYou can have 3 concurrent users with plenty of context,  better speeds or close enough speeds than a bunch of cards that provide less VRAM and cost 4+ times.\n\n    0.00.008.388 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)\n    0.00.008.391 I device_info:\n    0.00.089.439 I   - CUDA0   : NVIDIA P102-100 (10144 MiB, 10013 MiB free)\n    0.00.197.645 I   - CUDA1   : NVIDIA P102-100 (10144 MiB, 10013 MiB free)\n    0.00.197.656 I   - CPU     : Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz (128396 MiB, 128396 MiB free)\n    0.00.197.728 I system_info: n_threads = 6 (n_threads_batch = 6) / 12 | CUDA : ARCHS = 600,610,750,860,890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | \n    0.00.197.764 I srv          init: running without SSL\n    0.00.197.849 I srv          init: using 11 threads for HTTP server\n    0.00.198.515 I srv         start: binding port with default address family\n    0.00.199.823 I srv  llama_server: loading model\n    0.00.199.902 I srv    load_model: loading model '/models/Qwen3.6-35B-A3B-UD-IQ4_XS.gguf'\n    0.00.199.906 I common_init_result: fitting params to device memory ...\n    0.00.199.907 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)\n    0.00.987.288 W common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 99, abort\n    0.23.223.625 W llama_context: n_ctx_seq (32768) < n_ctx_train (262144) -- the full capacity of the model will not be utilized\n    0.23.481.073 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)\n    0.23.570.914 I srv    load_model: initializing slots, n_slots = 3\n    0.23.598.842 W srv    load_model: speculative decoding will use checkpoints\n    0.23.598.851 W common_speculative_init: no implementations specified for speculative decoding\n    0.23.598.852 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 32768\n    0.23.598.854 I slot   load_model: id  1 | task -1 | new slot, n_ctx = 32768\n    0.23.598.854 I slot   load_model: id  2 | task -1 | new slot, n_ctx = 32768\n    0.23.598.961 I srv    load_model: prompt cache is enabled, size limit: 8192 MiB\n    0.23.598.963 I srv    load_model: use `--cache-ram 0` to disable the prompt cache\n    0.23.598.964 I srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391\n    0.23.598.965 I srv    load_model: context checkpoints enabled, max = 32, min spacing = 8192\n    0.23.598.985 I srv          init: idle slots will be saved to prompt cache upon starting a new task\n    0.23.628.848 I init: chat template, example_format: '<|im_start|>system\n    You are a helpful assistant<|im_end|>\n    <|im_start|>user\n    Hello<|im_end|>\n    <|im_start|>assistant\n    Hi there<|im_end|>\n    <|im_start|>user\n    How are you?<|im_end|>\n    <|im_start|>assistant\n    <think>\n    \n    </think>\n    \n    '\n    0.23.666.546 I srv          init: init: chat template, thinking = 0\n    0.23.666.572 I srv  llama_server: model loaded\n    0.23.666.575 I srv  llama_server: server is listening on http://127.0.0.1:5802\n    0.23.666.579 I srv  update_slots: all slots are idle\n    0.48.181.695 I srv    operator(): Chat format: peg-native\n    0.48.182.094 I slot get_availabl: id  2 | task -1 | selected slot by LRU, t_last = -1\n    0.48.182.101 I srv  get_availabl: updating prompt cache\n    0.48.182.111 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000\n    0.48.182.123 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 98304 tokens, 8589934592 est)\n    0.48.182.128 I srv  get_availabl: prompt cache update took 0.02 ms\n    0.48.182.302 I slot launch_slot_: id  2 | task 0 | processing task, is_child = 0\n    0.48.182.309 I slot process_sing: id  0 | task -1 | saving idle slot to prompt cache\n    0.48.182.311 I slot process_sing: id  1 | task -1 | saving idle slot to prompt cache\n    0.48.186.009 I srv    operator(): Chat format: peg-native\n    0.48.189.081 I srv    operator(): Chat format: peg-native\n    0.49.483.103 I slot get_availabl: id  1 | task -1 | selected slot by LRU, t_last = -1\n    0.49.483.111 I srv  get_availabl: updating prompt cache\n    0.49.483.116 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000\n    0.49.483.119 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 98304 tokens, 8589934592 est)\n    0.49.483.120 I srv  get_availabl: prompt cache update took 0.01 ms\n    0.49.483.178 I slot launch_slot_: id  1 | task 2 | processing task, is_child = 0\n    0.49.483.179 I slot process_sing: id  0 | task -1 | saving idle slot to prompt cache\n    0.49.483.181 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1\n    0.49.483.181 I srv  get_availabl: updating prompt cache\n    0.49.483.182 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000\n    0.49.483.183 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 98304 tokens, 8589934592 est)\n    0.49.483.183 I srv  get_availabl: prompt cache update took 0.00 ms\n    0.49.483.215 I slot launch_slot_: id  0 | task 3 | processing task, is_child = 0\n    0.51.242.275 I slot create_check: id  0 | task 3 | created context checkpoint 1 of 32 (pos_min = 1376, pos_max = 1376, n_tokens = 1377, size = 62.813 MiB)\n    0.51.367.765 I slot create_check: id  1 | task 2 | created context checkpoint 1 of 32 (pos_min = 670, pos_max = 670, n_tokens = 671, size = 62.813 MiB)\n    0.51.367.773 I slot print_timing: id  2 | task 0 | prompt processing, n_tokens =   1377, progress = 1.00, t =   3.19 s / 432.28 tokens per second\n    0.51.480.037 I slot create_check: id  2 | task 0 | created context checkpoint 1 of 32 (pos_min = 1376, pos_max = 1376, n_tokens = 1377, size = 62.813 MiB)\n    0.56.647.801 I slot print_timing: id  0 | task 3 | n_decoded =    100, tg =  23.30 t/s, tg_3s =  23.30 t/s\n    0.56.653.219 I slot print_timing: id  2 | task 0 | n_decoded =    100, tg =  23.30 t/s, tg_3s =  23.30 t/s\n    0.56.692.218 I slot print_timing: id  1 | task 2 | n_decoded =    100, tg =  23.54 t/s, tg_3s =  23.54 t/s\n    0.59.655.679 I slot print_timing: id  0 | task 3 | n_decoded =    171, tg =  23.43 t/s, tg_3s =  23.60 t/s\n    0.59.661.606 I slot print_timing: id  2 | task 0 | n_decoded =    171, tg =  23.42 t/s, tg_3s =  23.60 t/s\n    0.59.702.608 I slot print_timing: id  1 | task 2 | n_decoded =    171, tg =  23.56 t/s, tg_3s =  23.58 t/s\n    1.02.659.591 I slot print_timing: id  0 | task 3 | n_decoded =    242, tg =  23.49 t/s, tg_3s =  23.64 t/s\n    1.02.665.066 I slot print_timing: id  2 | task 0 | n_decoded =    242, tg =  23.49 t/s, tg_3s =  23.64 t/s\n    1.02.705.486 I slot print_timing: id  1 | task 2 | n_decoded =    242, tg =  23.58 t/s, tg_3s =  23.64 t/s\n    1.03.253.784 I slot print_timing: id  0 | task 3 | prompt eval time =    2873.48 ms /  1381 tokens (    2.08 ms per token,   480.60 tokens per second)\n    1.03.253.789 I slot print_timing: id  0 | task 3 |        eval time =   10897.06 ms /   256 tokens (   42.57 ms per token,    23.49 tokens per second)\n    1.03.253.791 I slot print_timing: id  0 | task 3 |       total time =   13770.54 ms /  1637 tokens\n    1.03.253.792 I slot print_timing: id  0 | task 3 |    graphs reused =        253\n    1.03.253.924 I slot      release: id  0 | task 3 | stop processing: n_tokens = 1636, truncated = 0\n    1.03.259.600 I slot print_timing: id  2 | task 0 | prompt eval time =    4178.32 ms /  1381 tokens (    3.03 ms per token,   330.52 tokens per second)\n    1.03.259.605 I slot print_timing: id  2 | task 0 |        eval time =   10898.93 ms /   256 tokens (   42.57 ms per token,    23.49 tokens per second)\n    1.03.259.606 I slot print_timing: id  2 | task 0 |       total time =   15077.26 ms /  1637 tokens\n    1.03.259.607 I slot print_timing: id  2 | task 0 |    graphs reused =        253\n    1.03.259.741 I slot      release: id  2 | task 0 | stop processing: n_tokens = 1636, truncated = 0\n    1.03.288.482 I slot print_timing: id  1 | task 2 | prompt eval time =    2960.66 ms /  1381 tokens (    2.14 ms per token,   466.45 tokens per second)\n    1.03.288.486 I slot print_timing: id  1 | task 2 |        eval time =   10844.49 ms /   256 tokens (   42.36 ms per token,    23.61 tokens per second)\n    1.03.288.487 I slot print_timing: id  1 | task 2 |       total time =   13805.15 ms /  1637 tokens\n    1.03.288.488 I slot print_timing: id  1 | task 2 |    graphs reused =        253\n    1.03.288.614 I slot      release: id  1 | task 2 | stop processing: n_tokens = 1636, truncated = 0\n    1.03.288.625 I srv  update_slots: all slots are idle\n\n  \nEnd result for Qwen 3.6 35B at Q4 is 70tk/s and 1,257 PP  at 96K context providing 32K of context for each user.\n\nNow, I know that Pascal is no longer supported on 13x and while that statement is correct, there are no new features for these GPU's so there is no point in upgrading. Llama.cpp will run on the long term on 12.8 as it supports  6.1, 7.0, 7.5, 8.0, 8.6, 8.9, 9.0, 10.0, 12.0.  and there is 12.9 which supports the same. 12.8.2 update was released on April 2026 and 12.9.2 update was released on May 2026. that was 2 months ago. These Cards probably still have lifetime and llama.cpp probably stay on 12.x  as 12.x provides the broadest support as it does not want to alienate a very large portion of their user base. \n\nTo llama.cpp devs, \n\nThank you guys.. You really make a difference looking after the little guys.\n\n\n\n--- Top Comments ---\n\n\n[18 upvotes] P102-100s are basically 1080 Ti. They were bitcoin mining cards and have PCI-E 1x 1.0 slots (I believe), so it takes awhile for things to load onto them but...not a bad price at all. They however are power hungry little beasties, needing dual 8 pin connectors, but you can probably limit their wattage significantly and not impact them much.\n\n[5 upvotes] I see single stream is 430/23  \nSo 3 concurrencies are \\~70 but working from the same prompt. I would argue the PP won’t scale as well.\n\n[5 upvotes] shame everything else is expensive.\n\nmakes me buying a 200$ codex sub look beyond ridiculous ",
  "transcript_chars": 10430,
  "ingested_at": "2026-07-12T01:30:01.812658+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 60,
    "upvote_ratio": 0.93,
    "num_comments": 32,
    "author": "Boricua-vet",
    "is_self": true
  }
}