{
  "video_id": "reddit_1typjmc",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "120 tok/s on 12GB VRAM with Gemma 4 12B QAT MTP",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1typjmc/120_toks_on_12gb_vram_with_gemma_4_12b_qat_mtp/",
  "external_url": null,
  "upload_date": "20260606",
  "published_at": "2026-06-06T18:53:19+00:00",
  "transcript": "Google just released the QAT (Quantization-Aware Training) variant of their Gemma 4 models, including 12B, so it was only natural for me to benchmark it on my 12GB GPU since it fits entirely in VRAM. I was pleasantly surprised with the result!\n\nBy using llama.cpp patched with the Gemma 4 MTP PR, and loading Unsloth's [gemma-4-12B-it-qat-GGUF](https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF) quant and Google's [gemma-4-12B-it-qat-q4\\_0-unquantized-assistant](https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-unquantized-assistant) QAT assistant / draft model, which I converted to GGUF and uploaded to HuggingFace as [gemma-4-12B-it-qat-assistant-MTP-Q8\\_0-GGUF](https://huggingface.co/Janvitos/gemma-4-12B-it-qat-assistant-MTP-Q8_0-GGUF) using llama.cpp's convert\\_hf\\_to\\_gguf.py, I was able to achieve **120 tok/s** with [mtp-bench.py](https://gist.github.com/am17an/228edfb84ed082aa88e3865d6fa27090/)!\n\nBefore we start, here's my PC specs:\n\n    OS: CachyOS\n    GPU: RTX 4070 Super 12GB (iGPU as main GPU)\n    CPU: AMD Ryzen 7 9700X\n    RAM: 32GB DDR5-6000\n\nHere's my llama.cpp command:\n\n    llama-server \\\n      -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \\\n      --model-draft gemma-4-12B-it-qat-assistant-MTP-Q8_0.gguf \\\n      --spec-type draft-mtp \\\n      --spec-draft-n-max 4 \\\n      --ctx-size 131072 \\\n      --temp 1.0 \\\n      --top-p 0.95 \\\n      --top-k 64\n\nFor comparison, here's my [mtp-bench.py](http://mtp-bench.py) benchmark results **without** MTP:\n\n    ❯ ./mtp-bench.py\n     code_python        pred= 192 draft=   0 acc=   0 rate=n/a tok/s=59.9\n     code_cpp           pred= 192 draft=   0 acc=   0 rate=n/a tok/s=60.0\n     explain_concept    pred= 192 draft=   0 acc=   0 rate=n/a tok/s=59.9\n     summarize          pred= 192 draft=   0 acc=   0 rate=n/a tok/s=59.9\n     qa_factual         pred= 192 draft=   0 acc=   0 rate=n/a tok/s=59.9\n     translation        pred= 192 draft=   0 acc=   0 rate=n/a tok/s=60.0\n     creative_short     pred= 192 draft=   0 acc=   0 rate=n/a tok/s=60.0\n     stepwise_math      pred= 192 draft=   0 acc=   0 rate=n/a tok/s=59.8\n     long_code_review   pred= 192 draft=   0 acc=   0 rate=n/a tok/s=57.6\n    \n    Aggregate: {\n     \"n_requests\": 9,\n     \"total_predicted\": 1728,\n     \"total_draft\": 0,\n     \"total_draft_accepted\": 0,\n     \"aggregate_accept_rate\": null,\n     \"wall_s_total\": 30.2\n    }\n\nHere's my [mtp-bench.py](http://mtp-bench.py) benchmark results **with** MTP:\n\n    ❯ ./mtp-bench.py\n     code_python        pred= 192 draft= 172 acc= 133 rate=0.773 tok/s=130.5\n     code_cpp           pred= 192 draft= 187 acc= 128 rate=0.684 tok/s=120.4\n     explain_concept    pred= 192 draft= 213 acc= 119 rate=0.559 tok/s=105.7\n     summarize          pred= 192 draft= 168 acc= 134 rate=0.798 tok/s=133.5\n     qa_factual         pred= 192 draft= 210 acc= 120 rate=0.571 tok/s=107.2\n     translation        pred= 192 draft= 175 acc= 132 rate=0.754 tok/s=128.6\n     creative_short     pred= 192 draft= 240 acc= 110 rate=0.458 tok/s=94.0\n     stepwise_math      pred= 192 draft= 165 acc= 135 rate=0.818 tok/s=135.7\n     long_code_review   pred= 192 draft= 197 acc= 125 rate=0.634 tok/s=111.7\n    \n    Aggregate: {\n     \"n_requests\": 9,\n     \"total_predicted\": 1728,\n     \"total_draft\": 1727,\n     \"total_draft_accepted\": 1136,\n     \"aggregate_accept_rate\": 0.6578,\n     \"wall_s_total\": 15.66\n    }\n\nTo achieve this, all you need is a 12GB NVIDIA GPU and enough free VRAM to fit Gemma 4 12GB + assistant entirely in GPU memory. With CachyOS and my dGPU set as a secondary GPU, this gives me pretty much 100% free VRAM. On Windows, or if using your dGPU as your main GPU, you will probably loose 500MB+ of VRAM to the OS and driver, so you might need to lower the context size, or it might simply not work. You'll probably need to do some testing 😄\n\nHere's step-by-step instructions to get this working:\n\n    1. Clone llama.cpp\n    git clone https://github.com/ggml-org/llama.cpp.git\n    cd llama.cpp\n    \n    2. Fetch and switch to the Gemma 4 MTP PR branch\n    git fetch origin pull/23398/head:gemma4-mtp\n    git checkout gemma4-mtp\n    \n    3. Build with CUDA support for NVIDIA GPUs\n    cmake -B build -DGGML_CUDA=ON -DBUILD_SHARED_LIBS=OFF\n    cmake --build build --config Release -j$(nproc)\n    \n    4. Download Unsloth's Gemma 4 12B QAT here: https://huggingface.co/unsloth/gemma-4-12B-it-qat-GGUF\n    \n    5. Download Google's Gemma 4 assistant / draft here https://huggingface.co/Janvitos/gemma-4-12B-it-qat-assistant-MTP-Q8_0-GGUF\n    \n    6. Load the models with llama-server\n    llama-server \\\n      -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf \\\n      --model-draft gemma-4-12B-it-qat-assistant-MTP-Q8_0.gguf \\\n      --spec-type draft-mtp \\\n      --spec-draft-n-max 4 \\\n      --ctx-size 131072 \\\n      --temp 1.0 \\\n      --top-p 0.95 \\\n      --top-k 64\n\nCheers 😄\n\n\n\n--- Top Comments ---\n\n\n[15 upvotes] This is awesome, thanks for sharing! I hope the Gemma 4 MTP PR gets merged soon.\n\n\nWhat's your VRAM usage with this setup? Did you need all 12GB or could it fit in less than that? Asking because I have 16GB but I'd like to run two models on it at the same time (the other one is smaller).\n\n[3 upvotes] wow I tried setting this up today too but I failed because no matter which branch or pull requests image i compiled I always got `llama_model_load: error loading model: unknown model architecture: 'gemma4-assistant'` when loading the assistant draft model. Even with [am17an's fork](https://github.com/am17an/llama.cpp/tree/gemma4-mtp)\n\nNot sure what I did wrong though\n\n[3 upvotes] Thanks for this.\n\nDo you think it will work with an amd 6900xt? \n\nI am running this on my secondary machine for text processing and default unsloth studio setup gives me under 50t/s",
  "transcript_chars": 5734,
  "ingested_at": "2026-06-07T01:30:29.638487+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 136,
    "upvote_ratio": 0.98,
    "num_comments": 35,
    "author": "janvitos",
    "is_self": true
  }
}