{
  "video_id": "reddit_1usy9ie",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "Tencent-HY3 is the real deal on 128GB!",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1usy9ie/tencenthy3_is_the_real_deal_on_128gb/",
  "external_url": null,
  "upload_date": "20260710",
  "published_at": "2026-07-10T19:53:09+00:00",
  "transcript": "I'm *really* impressed with HY3. If you haven't heard of this model, it's a new 295B-A21B MoE release from Tencent that [competes directly on the frontier of open weights models, at a significantly smaller size](https://the-decoder.com/tencent-releases-hy3-open-source-model-that-allegedly-matches-models-up-to-five-times-its-active-size/), comparable to DeepSeek v4 Flash but with better benchmarks. I was intrigued by this article, and I'd just recently finished updating my Macbook M5 Max 128GB setup from antirez's DeepSeek V4 Flash quant running on dwarfstar to Unsloth's IQ3\\_XXS on mainline llama.cpp. I figured I had a good baseline for comparisons, if I could get it running, so I set about researching, and this is what I found.\n\nFirst off, I had to pick a quant. There are a few on HF, and after some comparison shopping I settled on [this UD128 \"unsloth dynamic\"-style 107GB quant](https://huggingface.co/YanissAmz/Hy3-295B-A21B-GGUF). It was the only one that had published perplexity numbers at the time I searched, and while that's not KLD, it shows the creator was at least thinking about measuring quality degradation. PPL didn't see horrible for a dynamic 3-bit quant, and it felt like a similar checkpoint to the UD DS4 quant I was using.\n\nNext, I had to get llama.cpp working. As the quant's readme helpfully notes, there's [PR #25395](https://github.com/ggml-org/llama.cpp/pull/25395) which implements support for this model and its built-in speculative decoding module all at once! A quick build got this up and running:\n\n    git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp\n    git fetch origin pull/25395/head:hy3 && git checkout hy3\n    cmake -B build -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON\n    cmake --build build --config Release -j\n\nDon't forget to raise your Mac's GPU memory ceiling from the default 96GB! I put mine at 122GB to ensure 24k context would fit safely for testing (it resets on reboot):\n\n    sudo sysctl iogpu.wired_limit_mb=124928\n\nHowever, when I tried to run the model, it wasn't recognized by llama.cpp, so the server errored out. A quick review of the log and the quant's readme explained why: \"these files carry `general.architecture = hy-v3` (this port's original naming). PR #25395 registers the arch as `hy_v3` (underscore)\". I could have avoided this hiccup if I'd paid a bit more attention, but I put together a quick script to swap those characters (21 instances, all in the first of the 3 GGUF shards). I can share if anyone wants it, but honestly, just ask your agent to do the job -- it takes 2 minutes to fix the GGUF. Running my launch script again, after a nail-biting \\~30 second load time to read 107GB from the SSD, the WebUI popped open and we were up and running!\n\n**Benchmarks (M5 Max, llama-bench, Metal, q8\\_0 KV cache,  MTP off)**\n\n|test|tokens/sec|\n|:-|:-|\n|prefill pp512 @ empty ctx|528|\n|decode tg128 @ empty ctx|32.4|\n|prefill pp512 @ 16K ctx|124|\n|decode tg128 @ 16K ctx|16.3|\n\n***In practical use, token generation speed is \\*DOUBLE\\* what I was getting from DeepSeek, with the same or better quality outputs!*** And I haven't even trialed MTP yet to see if there's any improvement... Here's my llama.cpp launch command:\n\n    ~/llama.cpp-hy3/build/bin/llama-server \\\n      -m ~/AI/models/Tencent-HY3-295B-A21B-YanissAmz/Hy3-UD128-00001-of-00003.gguf \\\n      -a \"Hy3 295B-A21B (UD128)\" \\\n      -ngl 99 \\\n      -c 24576 \\\n      -ctk q8_0 -ctv q8_0 \\\n      -fa on \\\n      -np 1 \\\n      --jinja \\\n      --temp 0.9 --top-p 1.0 \\\n      --host 127.0.0.1 --port 8080\n\nI've only played around with the model for a couple hours, but I am really impressed with performance on normal prompts and basic tool use. I've been using it to do ML research using the HF MCP and CLI tools, and it's **definitely** better than my DS4 setup so far -- on vibes and basic tool calls, at least. I haven't set it loose on longer-horizon agentic tasks or challenging coding yet, so I apologize for not having more in-depth reporting to offer at this time. Here's my launch command:\n\nI made this post because I'm really hyped to have a new large MoE to play with at this size checkpoint, and a very promising one at that. I hope it inspires some other folks to give it a try, and report back on their experience or compare other quants/MLX performance.\n\n**EDIT:** Got MTP working, n=2 is the best-performing variant on my hardware. Now peaking at **38 tok/sec**, so **+19% speedup!** Similar speeds as my Qwen3.6-27B-MTP q8\\_0 daily driver now.\n\n\n\n--- Top Comments ---\n\n\n[41 upvotes] Hard agree!  reviewed it here: https://youtu.be/32pkyoYmNjI  but the TLDR version:\n\n* Significantly better coder than deepseek-v4-flash\n* Keep using deepseek-v4-flash only if speed is the primary concern\n* Decent amount better than MiniMax M3\n* Thinks a lot, which makes it feel slow\n\n\n\nEDIT:  oops I forgot something important.  Deepseek can fit 1million context whereas I can only get like 190k on Hy3.  This could be a deal breaker for some.  I\"m also not hating on Deepseek v4, it's an engineering marvel.\n\n[17 upvotes] Curious to hear from someone with high VRAM (jealous), but how does something like Q3 Hy3 compare with basically full quant Qwen3.6:27B? Is running a larger MoE model, even in Q3, actually better than a full dense model?\n\n[11 upvotes] I just got the cables for DGX Spark gonna run 2x and 3x spark Hy3 Tensor Parallel performances. Looks interesting!\n\n[9 upvotes] Huh, straight up Apache2 licensed? Did they finally get rid of their weird territory restrictions?",
  "transcript_chars": 5538,
  "ingested_at": "2026-07-11T01:30:08.688128+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 139,
    "upvote_ratio": 0.94,
    "num_comments": 79,
    "author": "returnity",
    "is_self": true
  }
}