{
  "video_id": "reddit_1tt6oja",
  "channel_slug": "LocalLLaMA",
  "channel_handle": "r/LocalLLaMA",
  "title": "I ported NVIDIA Parakeet (speech-to-text) to ggml: same output as NeMo, faster, GGUF-quantized, no Python",
  "url": "https://www.reddit.com/r/LocalLLaMA/comments/1tt6oja/i_ported_nvidia_parakeet_speechtotext_to_ggml/",
  "external_url": null,
  "upload_date": "20260531",
  "published_at": "2026-05-31T20:35:51+00:00",
  "transcript": "I ported NVIDIA's Parakeet speech-to-text models to pure C++/ggml (the engine behind llama.cpp and whisper.cpp). It runs the FastConformer TDT / CTC / RNNT / hybrid models with no Python and no PyTorch, on CPU and GPU (CUDA, HIP, Vulkan, Metal).\n\nThe goal was to match NeMo exactly, then make it deployable anywhere. Where it landed:\n\n* Output is byte-for-byte identical to NeMo (WER 0 on the f32/f16 path).\n* Faster than NeMo's own PyTorch runtime: up to \\~5x on the larger TDT/hybrid models on GPU, up to \\~1.86x on CPU when quantized, and about 2x less memory.\n* Around 600x realtime on GPU on a 23s clip (one hour of audio in roughly 6 seconds).\n* Quantized GGUF for every variant: f16, q8\\_0, q6\\_k, q5\\_k, q4\\_k.\n\nhttps://preview.redd.it/t33li6b5aj4h1.png?width=1600&format=png&auto=webp&s=e50eaf8e1e3ba22314ad25586ec40ec613154b23\n\nIt also does cache-aware streaming with real-time end-of-utterance, word-level timestamps with confidence, and exposes a small flat C-API so you can embed it pretty much everywhere. The GGUF is self-contained: the tokenizer/vocab is baked into the model file, no external files needed.\n\nIt ships as a backend in LocalAI too, so you get an OpenAI-compatible /v1/audio/transcriptions endpoint fully local. (Disclosure: I work on LocalAI.)\n\nhttps://reddit.com/link/1tt6oja/video/nxngb7x1aj4h1/player\n\n  \n\n\nLinks:\n\n* Code (MIT): [https://github.com/mudler/parakeet.cpp](https://github.com/mudler/parakeet.cpp)\n* Models (GGUF): [https://huggingface.co/mudler/parakeet-cpp-gguf](https://huggingface.co/mudler/parakeet-cpp-gguf)\n\nAll credit to NVIDIA for the Parakeet models and to ggml for the runtime. Benchmarks, methodology, and per-model plots are in the repo. Happy to answer questions about the port, the decoders, or the numbers.\n\n\n\n--- Top Comments ---\n\n\n[5 upvotes] wow, this is awesome! i just finished a \"shitty voice robot\" project for our little one last weekend, using an ONNX based parakeet inference pipeline. while cross-platform, i'd have preferred something based on GGLM. and here it is. Thanks!\n\n[5 upvotes] Very nice!\n\nAre there plans to do the same for the Nvidia's Canary family of models?\n\n[4 upvotes] This nice, I very recently published a port/recipe to run parakeet in intel NPUs, specifically through the wyoming protocol (the use used by home assistant for voice interaction): [https://github.com/cibernox/wyoming-parakeet-on-intel-npu](https://github.com/cibernox/wyoming-parakeet-on-intel-npu)\n\nIf we were able to run gguf on the NPU, which in theory is possible, we could have a greater convenience while maintaining the low power consuption of using NPUs.\n\n[2 upvotes] Could it helped handy app to increase speed of local stt?",
  "transcript_chars": 2692,
  "ingested_at": "2026-06-01T01:30:15.015034+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 59,
    "upvote_ratio": 0.95,
    "num_comments": 23,
    "author": "mudler_it",
    "is_self": true
  }
}