{
  "video_id": "reddit_1tjh7az",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "110 tok/s with 12GB VRAM on Qwen3.6 35B A3B and ik_llama.cpp",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1tjh7az/110_toks_with_12gb_vram_on_qwen36_35b_a3b_and_ik/",
  "external_url": null,
  "upload_date": "20260521",
  "published_at": "2026-05-21T11:09:47+00:00",
  "transcript": "Had been getting [great MTP performance](https://www.reddit.com/r/LocalLLaMA/comments/1t82zxv/80_toksec_and_128k_context_on_12gb_vram_with/) with [llama.cpp](https://github.com/ggml-org/llama.cpp) on my RTX 4070 Super 12GB, until they actually merged the MTP PR. Then, performance tanked and was barely above non-MTP. So, I decided to try out [ik\\_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp) since it also supports MTP and is apparently better optimized for CPU offloading. I did not expect such a huge speed boost!\n\n# Before moving on with the benchmark results, here's my PC specs:\n\n    OS: CachyOS (HIGHLY recommended)\n    GPU: RTX 4070 Super 12GB\n    CPU: AMD Ryzen 7 9700X\n    RAM: 48GB DDR5-6000 EXPO I\n\n# UPDATED: For comparison, here's the regular llama.cpp [mtp-bench.py](https://gist.github.com/am17an/228edfb84ed082aa88e3865d6fa27090/) results with byteshape's recently released [Qwen3.6-35B-A3B-IQ4\\_XS-4.19bpw](https://huggingface.co/byteshape/Qwen3.6-35B-A3B-MTP-GGUF) quant, which has [similar accuracy](https://www.reddit.com/r/LocalLLaMA/comments/1tipihx/qwen_36_35b_gguf_ntp_vs_mtp_quantization_results/) to Unsloth's Q4_K_XL, but is 4GB smaller:\n\n    ❯ ./mtp-bench.py\n     code_python        pred= 192 draft= 122 acc= 118 rate=0.967 tok/s=79.8\n     code_cpp           pred= 192 draft= 117 acc= 110 rate=0.940 tok/s=89.1\n     explain_concept    pred= 192 draft= 124 acc= 113 rate=0.911 tok/s=88.0\n     summarize          pred= 192 draft= 139 acc= 127 rate=0.914 tok/s=95.0\n     qa_factual         pred= 192 draft= 133 acc= 128 rate=0.962 tok/s=97.0\n     translation        pred= 192 draft= 125 acc= 117 rate=0.936 tok/s=91.6\n     creative_short     pred= 192 draft= 109 acc=  99 rate=0.908 tok/s=82.1\n     stepwise_math      pred= 192 draft= 130 acc= 125 rate=0.962 tok/s=97.0\n     long_code_review   pred= 192 draft= 121 acc= 115 rate=0.950 tok/s=88.2\n    \n    Aggregate: {\n     \"n_requests\": 9,\n     \"total_predicted\": 1728,\n     \"total_draft\": 1120,\n     \"total_draft_accepted\": 1052,\n     \"aggregate_accept_rate\": 0.9393,\n     \"wall_s_total\": 21.86\n    }\n\nThis gives a **89.76 tok/s** average.\n\n# Here's my llama.cpp launch command. Temperature is set to 0.0 for the benchmark to prevent diverging results between runs:\n\n    llama-server \\\n      -m Qwen3.6-35B-A3B-IQ4_XS-4.19bpw.gguf \\\n      --fit on \\\n      --fit-target 512 \\\n      --ctx-size 131072 \\\n      --cache-type-k q8_0 \\\n      --cache-type-v q8_0 \\\n      --cache-type-k-draft q8_0 \\\n      --cache-type-v-draft q8_0 \\\n      --spec-type draft-mtp \\\n      --spec-draft-p-min 0.75 \\\n      --spec-draft-n-max 3 \\\n      --no-mmap \\\n      --mlock \\\n      --threads 8 \\\n      --temp 0.0\n\n# Now, here's the benchmark results with the same quant, but running with ik_llama.cpp:\n\n    ❯ ./mtp-bench.py\n     code_python        pred= 192 draft= 135 acc= 122 rate=0.904 tok/s=105.1\n     code_cpp           pred= 192 draft= 136 acc= 120 rate=0.882 tok/s=110.3\n     explain_concept    pred= 192 draft= 133 acc= 116 rate=0.872 tok/s=109.0\n     summarize          pred=  56 draft=  38 acc=  37 rate=0.974 tok/s=122.3\n     qa_factual         pred= 192 draft= 141 acc= 127 rate=0.901 tok/s=116.0\n     translation        pred= 192 draft= 143 acc= 113 rate=0.790 tok/s=104.1\n     creative_short     pred= 192 draft= 133 acc= 118 rate=0.887 tok/s=109.4\n     stepwise_math      pred= 192 draft= 140 acc= 125 rate=0.893 tok/s=114.6\n     long_code_review   pred= 192 draft= 128 acc= 108 rate=0.844 tok/s=101.4\n    \n    Aggregate: {\n     \"n_requests\": 9,\n     \"total_predicted\": 1592,\n     \"total_draft\": 1127,\n     \"total_draft_accepted\": 986,\n     \"aggregate_accept_rate\": 0.8749,\n     \"wall_s_total\": 16.64\n    }\n\nThat's a **110.24 tok/s** average, or **23%** increase!\n\n# If you want to get similar results on a 12GB RTX GPU, make sure you use the following ik_llama.cpp launch parameters, as they can differ from llama.cpp:\n\n    llama-server \\\n      -m Qwen3.6-35B-A3B-IQ4_XS-4.19bpw.gguf \\\n      --fit \\\n      --fit-margin 1664 \\\n      --ctx-size 131072 \\\n      --cache-type-k q8_0 \\\n      --cache-type-v q8_0 \\\n      --cache-type-k-draft q8_0 \\\n      --cache-type-v-draft q8_0 \\\n      --multi-token-prediction \\\n      --draft-p-min 0.75 \\\n      --draft-max 3 \\\n      --no-mmap \\\n      --mlock \\\n      --threads 8 \\\n      --temp 0.0\n\nI also want to mention that I'm on CachyOS running my GPU as a secondary GPU, with the monitor plugged in the iGPU, so I can use 100% of available VRAM.\n\nIf you get an \"out of memory\" (OOM) error while loading the model or working with it, try increasing --fit-margin to 1792 or even 2048.\n\nCheers :)\n\n\n\n--- Top Comments ---\n\n\n[1 upvotes] Your post is getting popular and we just featured it on our Discord! [Come check it out!](https://discord.gg/PgFhZ8cnWW)\n\nYou've also been given a special flair for your contribution. We appreciate your post!\n\n*I am a bot and this action was performed automatically.*\n\n[43 upvotes] little \"trick\" for who uses cachyos (kde wayland) and doesn't have a secondary gpu/igpu: create a custom cpu only sddm session.\n\n```fish\nsudo nano /usr/share/wayland-sessions/plasma-cpu.desktop\n```\n\n```config\n[Desktop Entry]\nName=KDE Plasma (CPU)\nComment=KDE Plasma on Wayland with software rendering\nExec=env LIBGL_ALWAYS_SOFTWARE=1 GALLIUM_DRIVER=llvmpipe KWIN_COMPOSE=Q /usr/bin/startplasma-wayland\nDesktopNames=KDE\nType=Application\n```\n\nyou can select this new session at login: click left bottom corner and switch to \"KDE Plasma (CPU)\"\n\nthis session will compute all kde compositor graphics on cpu. Animations will be disabled and/or slow as hell BUT you'll essentially run a full kde session with few hundreds mb of vram (i was using tty before to achieve the same lol)\n\nin my case:\n- KDE wayland idle > 1024mb vram\n- KDE cpu only idle > 126mb vram\n\n[28 upvotes] Please include your llama.cpp full command as well.\n\nAlso when did you try llama.cpp? Because couple of MTP related PRs got merged on llama.cpp side in last 24 hours.\n\n[21 upvotes] It looks like the speedup might come from the much higher acceptance rate with ik\\_llama.cpp. With ik\\_llama.cpp it's never below 0.790 and with llama.cpp it goes as low as 0.477. I wonder why that would be the case. Which settings were you using with llama.cpp?\n\n[16 upvotes] It is a pity that @ikawrakow is no longer contributing with llama.cpp and we have this fragmentation. Hoping that one day this drama ends and huggingface  him to bring his improvements to llama.cpp.",
  "transcript_chars": 6452,
  "ingested_at": "2026-05-21T19:45:40.504744+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 212,
    "upvote_ratio": 0.96,
    "num_comments": 75,
    "author": "janvitos",
    "is_self": true
  }
}