{
  "video_id": "reddit_1u8i79d",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "llama.cpp - how to free up even more space on your GPU",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1u8i79d/llamacpp_how_to_free_up_even_more_space_on_your/",
  "external_url": null,
  "upload_date": "20260617",
  "published_at": "2026-06-17T18:23:10+00:00",
  "transcript": "For the past week or two, llama.cpp has been working much better from the RAM usage prespective. I no longer see any memory leaks, and everything fits nicely on the GPU - my defaults are **--n-gpu-layers 99 --no-mmap --mlock** to avoid using the regular RAM, since I use my 3090 with an eGPU setup: Qwen3.6-27B-UD-Q5\\_K\\_XL-mtp, q4\\_0, 150k context\n\nI wanted to create this thread to see if there are any additional tricks for freeing up even more memory so that I can further increase my context size.\n\n\n\nMy list of VRAM-related parameters for a given model (which is, of course, the biggest factor in memory footprint):\n\n* **--no-mmproj-offload**: this is the biggest win: if you have a model with vision, you can offload the mmproj to CPU. It is a little drop in terms of performance, but you'll end up with 1GB additional free space on your card.\n* **--cache-type-k, --cache-type-v**: KV cache (obviously) - reduce memory allocation by 50%, 75%, etc. but of course, quality will drop in return. my observation is that since attention rotation has been introduced, I can even use q4 without much noticable drop of quality, since I can use a bigger base model - which helps me more vs drop of quality because of KV cache.\n* **--cache-type-k-draft, --cache-type-v-draft**: same applies to the mtp model's KV cache\n* **--spec-draft-n-max**: guess up to x future tokens ahead in a single forward pass. With coding, I'm usually fine with \"2\" as the value. \"1\" consumes slightly less memory, but TPS drops about 5%. \"3\" doesn't make sense for my use case - consumes more memory, but same TPS as with \"1\"\n* **--flash-attn on**: this is the default value by now, as far as I know. Memory allocation would grow if you'd turn it off, but you cannot turn it off anyway if you use a quantized v cache\n\n\n\nParameters I thought would help, until I realized they actually don't:\n\n* **--ctx-checkpoints**: I've heard that decreasing this value would also decrease memory allocation, but it's not the case for me. Default is 64, and no change for me when I decrease it a small value\n* **--parallel**: number of active user request at a time. Since 1 is the default value, you cannot do anything with it in a single user setup. However, if you increase it, your KV cache for your main session will be reduced accordingly (50%, 66%, etc.)\n* **--fit-target**: sets a strict safety buffer margin (in Megabytes - default 1024) that the engine must leave completely empty on your GPU (for example, reserved for video I/O). Since my monitor is plugged into a different card, I reduced it to 64, but it didn't help at all. As far as I know, llama cpp now runs an internal calculation loop at startup to automatically adjust some variables to prevent itself from an OOM crash.\n\n\n\nI've shared my tips, what's one of yours? Is there anything else at all? Is your experience different to mine? thanks!\n\n\n\n--- Top Comments ---\n\n\n[17 upvotes] The batch size also affect VRAM usage but it also ties to your prefill processing speed. In desperate cases where I absolutely need every MB of VRAM, I set as low as 512 :( \n\n\nbtw if anyone find this sentence weird to read (why would I use \"no offload\" when I want to offload mmproj into system RAM)\n> --no-mmproj-offload: this is the biggest win: if you have a model with vision, you can offload the mmproj to CPU. \n\nIt's because from llama.cpp's perspective, offloading means using GPU rather than CPU. Like `--kv-offload` means putting kv cache into vram (which is the default setting). \n\nAnother btw, --fit is automatically disabled when you manually specify the number of GPU layers, that's why --fit-target didn't change anything for you.\n\n[11 upvotes] >--no-mmproj-offload: this is the biggest win: if you have a model with vision, you can offload the mmproj to CPU\n\nYes.. I learned this little trick and not having to shrink your context or shuffle layers around. It goes from like 1s to 15s but you don't send that many images most of the time.\n\n[8 upvotes] `--flash-attn` default is auto, but this usually means on.\n\nAlso, butchering your KV cache isn't worth it. Anything below Q8 ends up having visible effects on generation quality; this is compounded with using thinking, this is also compounded with using highly quantized models, and this is *also* compounded with having small models (below 20b). Check Anbeeld (the BeeLlama author)'s blog, anything below Q6 ain't worth it, and upstream llama.cpp doesn't have Q6 yet.\n\nhttps://anbeeld.com/articles/kv-cache-quantization-benchmarks-for-long-context\n\nQuantizing the model once is like turning an uncompressed image into a high quality JPEG (or not-so-high quality if it isn't imatrixed), thinking with butchered KV cache is like repeatedly JPEGing a JPEG.\n\n`--spec-draft-n-max` is both use case sensitive and hardware sensitive. The community has settled on 2 for a Nvidia recommendation, but 3 seems to be the better choice for AMD.\n\n`--no-mmproj` is also a useful flag if you never intend on using multimodal functionality at all.\n\n`--ctx-checkpoints` on Qwen 3.6 reserves about 256mb per checkpoint, but Gemma 4 uses something like 4 times that. It is safe to set this to zero and pay the context r\n\n[4 upvotes] Following this as a single RTX 3090 user",
  "transcript_chars": 5235,
  "ingested_at": "2026-06-18T01:30:25.589432+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 65,
    "upvote_ratio": 0.97,
    "num_comments": 40,
    "author": "imgroot9",
    "is_self": true
  }
}