{
  "video_id": "reddit_1tw0kg0",
  "channel_slug": "singularity",
  "channel_handle": "r/singularity",
  "title": "Someone did an audit on the new DeepSWE, the results aren't pretty",
  "url": "https://www.reddit.com/r/singularity/comments/1tw0kg0/someone_did_an_audit_on_the_new_deepswe_the/",
  "external_url": "https://github.com/datacurve-ai/deep-swe/issues/21",
  "upload_date": "20260603",
  "published_at": "2026-06-03T19:50:12+00:00",
  "transcript": "While this post on the DeepSWE Benchmark github is mainly focused on DeepSeek failing in many places where it shouldn't, it shows many problems with how the benchmark was conducted. It seems that the benchmark was rushed out the door and still needs a lot more work before it can be considered a reliable reference for the quality of the models they benchmarked.\n\n\n\n--- Top Comments ---\n\n\n[53 upvotes] I am calling it now, in the future we will have very profitable AI rating companies similar to credit rating companies like Moody's and S&P Global.\n\n[11 upvotes] In short openrouter suck.\n\n\nThey should use official API from Deepseek.\n\n[6 upvotes] tl;dr the sole outcome of this audit was due to a problem with openrouter deepseek's benchmarks were not accurate. this has nothing to do with any of the other benchmarks at all, and op is simply clickbaiting\n\n[3 upvotes] I used deepseek through open router, it just doesn't work very often, so I'm not surprised.",
  "transcript_chars": 962,
  "ingested_at": "2026-06-04T01:30:18.474491+00:00",
  "source": "reddit",
  "yt_meta": {
    "score": 57,
    "upvote_ratio": 0.88,
    "num_comments": 23,
    "author": "pneuny",
    "is_self": false
  }
}