{
  "video_id": "reddit_1tpaw6x",
  "channel_slug": "MachineLearning",
  "channel_handle": "r/MachineLearning",
  "title": "AI-generated CUDA kernels silently break training and inference [R]",
  "url": "https://www.reddit.com/r/MachineLearning/comments/1tpaw6x/aigenerated_cuda_kernels_silently_break_training/",
  "external_url": null,
  "upload_date": "20260527",
  "published_at": "2026-05-27T16:35:33+00:00",
  "transcript": "Last month NVIDIA released [SOL-ExecBench](https://research.nvidia.com/benchmarks/sol-execbench), a new benchmark of 235 production CUDA kernels lifted from DeepSeek, Qwen, Gemma, and Kimi. We took several top-ranked AI-generated submissions and tried using them in production workloads. Many of them broke, sometimes in surprising ways.\n\nOne of those kernels is the fused embedding-gradient + RMSNorm backward pass, which runs at the end of every transformer training step. We took the fastest submission on the benchmark for it, and dropped it into the training loop of a small transformer. The kernel had passed the benchmark's verifier with room to spare. But in our training run, the loss diverged and never recovered.\n\nWe started debugging. Replace the dataset distribution with uniformly sampled tokens, the divergence vanishes. Swap SGD for AdamW, also vanishes.\n\nThis is the worst kind of bug for research. Symptoms and masks both look exactly like \"the idea didn't work\". It's the type of bug that can make researchers spend a long time debugging without knowing what's at fault: the dataset? the research idea? the architecture? or the implementation itself?\n\nTurns out, the actual bug is that the embedding-gradient half of the kernel accumulates in bf16 instead of fp32. Embedding backward sums many small gradient contributions into each token's row of the embedding matrix. With uniform random tokens the contributions spread evenly and bf16 precision is enough. In real text, a handful of token IDs end up with thousands of contributions: the small ones round to zero against the growing accumulator, and the high-frequency rows drift. AdamW's per-parameter normalization absorbs the resulting multiplicative bias, so under AdamW the same drift is invisible in the loss.\n\nThe other broken submissions had different bug shapes (all interesting). More examples in [our blogpost](https://www.doubleai.com/research/warpspeed-approaches-speed-of-light-on-blackwell).\n\n\n\n--- Top Comments ---\n\n\n[49 upvotes] >Turns out, the actual bug is that the embedding-gradient half of the kernel accumulates in bf16 instead of fp32.\n\nDamn, that's the kind of thing a lot of people would never find.  \nSome people might see it and gloss over it, since bf16 is used so often.\n\n[17 upvotes] so, the solution was to use AdamW\n\n[7 upvotes] Using bf16 instead of fp32 when it works on AdamW but does not work on SGD does not sound like a bug to me.",
  "transcript_chars": 2440,
  "ingested_at": "2026-05-28T01:30:11.443903+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 136,
    "upvote_ratio": 0.93,
    "num_comments": 12,
    "author": "laginimaineb",
    "is_self": true
  }
}