{
  "video_id": "gFSRIht95h0",
  "title": "How to monitor production AI agents: A simple breakdown",
  "url": "https://www.youtube.com/watch?v=gFSRIht95h0",
  "transcript": "You don't know what your agent will do until it's in production. When you ship traditional software to production, you have a pretty good sense of what's going to happen. Users click buttons, fill out forms, move through paths you've already mapped out. When something breaks, you look through stack traces and logs. Agents are different. The input here is natural language, so the range of things people can ask is basically unlimited. The models behind them are sensitive to small changes in phrasing and give you different answers to the same question. And they reason through problems in multiple steps, calling tools, pulling in context, making decisions you can't explicitly program. So if agents are different, the way you monitor them has to be different too. For traditional software, we use APM tools. They track things like latency, error rates, and throughput. And they work great for systems where you know the possible code paths. When your agent is having conversations with users, the signal you want to track isn't in the code, it's in the conversations themselves. You need to see what the user asked, how the conversation played out across multiple turns and what step the agent took to get there. Whether an interaction went well isn't going to be answered by the uptime and these conversations need human judgment to evaluate During development you can review these yourself but in production you might have thousands or millions of interactions So how do you keep human intelligence in the loop when you can't manually review everything? We found two complementary approaches that work. We built them both into Langsmith. The first is making human review as efficient as possible. Annotation queues let you route specific traces for structured review, set up rubrics and collaborate as a team, instead of hunting through raw production logs alone. The second is using LLMs to automate that review. In Langsmith, you can configure evals and insights that run automatically against your production traces. These can check output quality, safety, compliance, user sentiment, or surface usage patterns and failure modes. You pick which traces to evaluate, and the results feed into reports and dashboards where you can track trends, set alerts and drill down from a high-level metric to the actual conversation. So yes, you don't know what your agent will do until it's in production. But that doesn't have to mean you're flying blind. Capture the right data, combine human and automated review and let every production interaction make your agent better. Thank you.",
  "transcript_chars": 2584,
  "transcript_filled_at": "2026-06-06T14:36:51.227108+00:00",
  "transcript_filled_by": "tk-bulk-groq-retry-20260606"
}