{
  "video_id": "ObTPqBGsEbA",
  "channel_slug": "aidotengineer",
  "channel_handle": "aidotengineer",
  "title": "The Production AI Playbook: Deploying Agents at Enterprise Scale — Sandipan Bhaumik, Databricks",
  "duration_seconds": 2226.0,
  "url": "https://www.youtube.com/watch?v=ObTPqBGsEbA",
  "upload_date": "",
  "transcript": "[music]\n>> All right. Um, thank you for joining my\nsession.\n>> [applause]\n>> Thank you, man.\nUh, I'm Sandy. Uh, I'm a technical lead\nuh, for data and AI at Databricks. Um,\nprior to working in Databricks, I worked\nin Amazon Web Services uh, for 5 years\nas a principal architect for data and\nAI.\nUh,\nin the past few years, I worked\nextensively uh, building and scaling\ndata and AI platforms using distributed\nsystems and technology.\nAnd in the past couple of years,\nspecifically, I've been working with\ncustomers trying to figure out what we\ndo with this new AI technology.\nUh, when I say new AI, AI has been here\nfor a long time, but we all started\nexperimenting quite exponentially uh, in\nin the past couple of years, right? And\nI have learned a great deal of lessons\non from building demos and how to take\nthose demos to production working with\ndifferent customers in\nuh, B2B software industries and then uh,\nregulating industries like financial uh,\nservices.\nSo, in this session,\nI want to share\na playbook, a framework that I put\ntogether from lessons that I have\nlearned working in the trenches\nuh, that you can take and apply on when\nyou think about how to put your AI\nsystems into production. And I think\nthis session is nicely placed in the\nafternoon because what you can do now is\nin this framework, you can fit the\ndifferent\num, knowledge, the knowledge that you've\ngathered attending these different\nsessions throughout the day and see\nwhere they fit in each of these\nyou know, elements in the framework.\nSo, when I started 2 years ago, this is\nthe pattern I noticed in every customer\nconversation, right? So, everyone wanted\nto do something with AI. Uh, there was\nimmense pressure from the top to do\nsomething, to build a demo.\nAnd every conversation started with\nlet's choose the model, right? And it\nwas nobody's fault because the market\nwas like that. We were talking about\nmodels, the models were new technology\nfor us, right? And every conversation\nstarted, shall we use GPT? Shall we use\nClaude?\nYou know, there was huge debate with\nwith within organizations. Then, you\nwould choose a model, you'll build some\nfeatures, offsets of over what features\nto build for that application.\nUh, you would build that in a controlled\nenvironment, so predictable data sets,\nyou know, um, limited scenarios, and\nthen it looked great as a demo, and then\nleadership would get happy, they would\nsign it off, and they'll put it into an\nenvironment in a production environment.\nThen,\nafter a few weeks, people would start\nasking questions that what the hell is\nAI doing?\nRight? Why is it not answering the\nquestions the way we expected it to\nanswer when we were doing the demos?\nUh, it would result in not only\nless you know, no realization in return\non investment, but also loss of money\nand effort in building these demos that\ncan never scale to production.\nThroughout these uh, meetings, I\ngathered three insights that connect to\neverything that you we are talking about\nwhen thinking about taking uh, AI to\nproduction. The first one is the\nobservability gap, right? When we use AI\nand put it into production, if we can't\nsee what it is actually doing, if we\ncan't trace every decision that it's\nmaking, it's no use in production.\nSecond is the evaluation gap gap. A lot\nof these conversations that we were\ndoing, we were not actually thinking\nabout what is what is that one thing\nthat we are measuring. Yes, we talk\nabout accuracy, we talk about latency,\nwe talk about groundedness, but we were\nnot defining what is that exact thing\nlike that matters to the business, and\nhow can we build a system that can\ncontinuously measure that, whether it's\nimproving, whether it's not improving,\nlike what what is that system that we\nneed to build. And that was that\nevaluation gap that I noticed. And the\nthird is the governance gap. Like we\nwere not actually thinking what happens\nwhen AI fails in production. Who's\naccountable? Who do I go to when\nsomething happens at 3:00 a.m. in the\nmorning, right? Who needs to own the\ndata assets that feed some AI responses?\nWhat happens if AI\nyou know, uh, talks um, uh, um,\nnonsense to a customer, right? What what\nhappens, right? So, there is no\naccountability, no governance around it.\nAnd these three insights led me to\nbuild a framework\non how\nI think AI should be taken to\nproduction, and this has been\nimplemented across multiple customer\norganizations, and I think this is\nsomething that you can pick up from\nhere.\nThese are the five pillars, and these\nare absolutely what you need to think\nabout even before starting a project,\nright? Then you start build them\ngradually,\npreferably in sequence, but in real\nlife, I know that this sequence don't\nwork, but these are the pillars that you\nhave to know about and you have to think\nabout when start building. First one is\nevaluation.\nBefore touching any code, before\ndiscussing about any models, any\nfeatures, you have to think about when\nwe build this system, how do we measure?\nWhat does success look like, and what is\nthat system that will help us\ncontinuously measure what success looks\nlike for us?\nSecond is how do we trace each and every\ndecision that AI makes. It's not only\nimportant for the performance of the AI\nsystem, it is also important for the\nregulators. In Europe or in a lot of\ncompanies, especially in regulated\nindustry,\nyou cannot even onboard AI into\nproduction without having tracing and\nobservability in place.\nSo, this is a must-have.\nThe third is the data foundation,\nright? Uh, I I I I think of data\nfoundation in in two ways. One is the\nquestion data, so that is basically\nthe data needed for the AI to answer\nquestions that users ask to it. So, it\ncould be your pre-training data, post\npost-training data, data that you use\nAPIs to hook onto and and get to the\nanswer that the user needs. The other\none is the tracking data, related to the\ntracing data in observability, but when\nyou think from the data foundation and\nand uh, data strategy perspective, this\nneeds to be handled in this pillar\nbecause you need a whole data strategy\nnow with tracing data, especially when\nyou run hundreds of agents in your\norganization.\nFourth is orchestration. One agent would\nwork pretty well. You don't need to\nthink about orchestration. But when you\nonboard five agents, the the complexity\nincreases exponentially, right? You will\nhave multiple coordination patterns\nbetween these agents, they will need to\ntalk to each other in multiple different\nways, they will need to each wait for\neach other's responses, there's a lot of\ncomplexity that comes in. And that's\nwhere orchestration patterns and\nthinking about how you will orchestrate\nyour agents in a particular system\nbecomes really important.\nFifth is governance. This is where\nyou think about what happens when\nsomething fails. Who's accountable? How\ndo we govern data? How do we secure it?\nHow do we secure our systems? How what\ndo we make sure how do we make sure that\nno one injects into our agent and leads\nuh, to you know, misbehavior, right? Or\nloss of reputation.\nSo, in the rest of the session, I will\ndive a bit deeper into each of these\npillars and tell you how how you can\nthink about when you start working with\nthem, right? The first one is\nevaluation.\nEvaluation is basically specification\nfor your AI system.\nYou define success. As I mentioned, it's\nnot like\ntalking about accuracy. You have to\ndefine it with numbers, like what\naccuracy is is is good for your business\nuse case,\nright?\nDefine it in numbers.\nUh, define what kind of you know, false\npositives you can handle. What should be\nthe deflection? So, this is this is an\nexample from a\na retail chatbot, right? A banking\nchatbot where\nwhen you implement a chatbot with an AI\nagent, one of the main goals is to\ndeflect simple queries\num, to the agent so that a human agent\ndon't need to uh, deal with them, right?\nAnd so, you need to uh,\nyou need to track those queries and\ntrack those numbers and put that system\nin place. Second is building those test\ntest cases, like the evaluation data\nset. You've heard about golden data sets\nin evaluation. Talk with the domain\nexperts and find what is actually\nhappening in real life on the ground.\nLike what answer would a support human\nsupport agent\num, give to a customer on a particular\nquestion. Collect those information.\nWhat happens in gray areas, in edge\ncases, like what happens when a human\nsees a customer asking a confusing\nquestion, right? Collect those into a\ndata set, and then automate your AI\ntesting, right? So, you put a question\nto AI, it answers, take that answer,\ncompare against the test set, and\nautomate this whole pipeline so that\nwhen you put AI in production, that\npipeline can actually take live\nresponses and evaluate against the test\ndata set that you're building, and then\ngive you the result in terms of how AI\nis performing against those numbers and\nthe goals that you've defined.\nWhen we talk about evaluation,\nthere are three main layers that I see\nappear across organization, and this is\nan architectural decision that you need\nto make when you build these evaluation\nsystems, right? The first layer is\ndeterministic. These are the easy stuff,\nlike you know, checking formats, you\nknow, checking email formats, phone\nformats, the regular expression things\nthat we have already been doing with our\ncoding systems, right? The uh, the the\nother other is like you know, you could\nuse a classic ML models for name entity\nrecognition to for intent\nclassification,\nfor understanding what is first name,\nlast name, PII detection, etc.\nSo, the these these are easy stuff,\ncheap stuff, you should get them out of\nthe way. We have already been doing this\nfor years. The second layer is the\nnon-deterministic semantic stuff, all\nright? This is where groundedness comes\nin. This is where we implement\ntechnologies like LLMs judges. We all\nknow what LLMs judges are, right?\nEveryone? Okay, I see a lot of nods. So,\num\nAgain, this This is a pretty simple\nversion of how a uh how a prompt would\nlook for an LLM as a judge. Um\nSo, with LLM as a judge, you you you use\na separate LLM from the primary LLM to\njudge the response of the primary\nmodel.\nAnd when you do that, you tell the\nsecondary, the judge model,\non how it should uh judge the primary\nmodel's output. So, it could be around\nsafety, groundedness, you know,\nrelevance to the answer, etc. etc.,\nright? Again, that can feed from a lot\nof uh the evaluation data set that you\nhave created, right? To look at what are\nthe expected answers, and then it can\ncheck against that. This is a sample\nprompt on how these things work, but I'm\nsure you've attended some of these\nsessions where you've seen vendors doing\nthis automatically at scale. Uh for\nexample, in Databricks we in MLflow\nyou'll find automatic LLM as judge, uh\nwhere you can create these custom LLM as\njudges that run automatically on traces.\nThat's your second layer. The third\nlayer is behavioral, right? This is\nwhere uh you think about a tool calls,\nlike is our agents calling the right\ntool? Are they getting into loops? So,\nfor example, um you know, the first\nlayer you you can have a user ask a\nquestion, \"What is my account balance?\"\nAnd\nyou could go and check that, okay, this\nthere is no deterministic problem with\nit. The seman- the agent answered right,\nthat, \"Okay, your account balance is\nthis many dollars.\"\nAnd that was right, and you can see this\nis this is right, but when you go into\nthe behavioral checks, you will see that\nthe agent uh actually made three calls\nto the database to find that answer.\nRight? And that is because it was doing\nmaking duplicate calls for whatever\nreason.\nCalls failed,\nyou know, calls did not work, it went\nand retried and stuff like that. Now,\nthree API calls in demo environment is\nfine, but in production, when you get\nthousands of queries from users every\nday, and there's like duplication in API\ncalls, that's an expensive operation.\nAnd that's where you need to think about\nbehavioral evaluation. And this layer is\nvery, very important. I see a lot of\norganizations, a lot of teams miss them\nwhen when talking about this.\nThe second layer is observability,\nright? Uh and in this pillar, what we're\ntalking about tracing, right? So, you\ncollect [snorts] all the decisions that\nan agent is making. So, I want to\nexplain this with a scenario here,\nright? And this is a scenario from an\nactual project I worked on with a\nbanking uh retail retail banking\nchatbot. Now, obviously, if you've seen\ntracing data, it's not as beautiful as\nthis slide, right? So, I've simplified\nit and made it beautiful for this slide.\nBut what this slide says is basically, a\nuser comes in and says, uh \"You know, I\nhave been charged an overdraft fee, can\nyou waive it for me?\" Because the user\nthinks that the customer thinks that\nthat is not legitimate.\nSo, the agent does an intent\nclassification,\nand you all you know about this because\nyou've enabled observability, you're\ncapturing traces, and you're actually\nseeing what the agent is doing, right?\nWhat AI is doing.\nIntent classification, it is done, this\nit took this many seconds, this was\nconfidence score. Then it goes and\nconnects to the customer's account,\nmaybe in a database, a customer\ndatabase, call calls an API, connects to\nthe customer database, gets the account\ndetails.\nIt retrieves policy documents. It checks\nfrom a rag vector database,\num what is uh what is the policy around\noverdraft, right? Is what the customer\nclaiming is legitimate? So, it checks\nfor policy documents. Then it goes and\ndoes a reasoning on what should be\nuh\nyou know, responded to the customer, and\nthen it does some final guardrail\nchecks, and responds to the customer.\nNow, if you did not set up a system that\nhelps you look visualize all of these\ntraces,\nwhen the customer comes to you and\nraises a dispute,\nyou have no way to check what the AI\ndid.\nRight? You have nowhere to go, and you\nend up saying that I don't have have any\nidea. Let's Let's give the customer a\ndiscount or something, and then make\nthem happy.\nSo, this is why you need this, and this\nis why regulators are are are basically\nmandating, because otherwise there's no\nproduction system if you cannot do this\nkind of stuff.\nSo, this is where um\nyou know, you you you detect this the\nexample that I gave around duplicate API\ncalls. This is where you start detecting\nthis stuff. So, when you when you enable\nthese traces, you can actually go and\nsee duplicate calls, and then take\nrelevant actions based on that. Not only\nthat, you can actually do that in online\nmonitoring. So, when it's happening in\nproduction, in on you can set up online\nmonitoring, and at that point, if it is\ndoing duplicate calls, you can apply\nfallback strategies. Or even if it is\ndoing a call that is failing, you can\nactually go and apply a strategy where\nit will say, \"Okay, go and retry for\nthree times, not more than three times.\nIf it if it is more than three times,\nthen report somewhere, or pass it to a\nhuman to take some action.\"\nThe third pillar is the most important\npillar, in my opinion, is the data data\ndata foundation, right? Uh\nin my typical project projects, I spend\n60% of my time, uh and I see I see a lot\nof organizations spending a lot of time\nhere, because\nno one expected agents to come suddenly\nin the market and start querying data.\nData was always built for humans, and\nhumans are always forgiving.\nYou find the wrong data in a report, you\njust go and ask someone to correct it.\nAgents don't forgive you, right? Agents\nwill go, find it wrong, they'll give you\nthe wrong answer confidently.\nRight? And you wouldn't know what's\nhappening.\nAnd this is why data quality,\nsetting the right data strategy, has\nbecome so important for enterprises now.\nI divide it into two sections. One is\nthe question data, as I was explaining,\nlike data needed for actually serving\nthe AI's uh outcome.\nAnd the other one is the tracking data.\nThis is the observability data, the\ntracing data I was talking about\nearlier.\nYou need a proper plan on how you\ncollect this tracing data, and how you\nserve it to auditors, to regulators, to\ndo online monitoring, to run LLM as\njudges on the tracing, and everything\nelse, right? So, there it needs a proper\nstrategy on how you structure the schema\nand everything on the tracing data.\nUm\nOn Databricks,\num we\ncreate a robust data foundation for our\ncustomers\nusing uh some of the technologies that\nwe provide. If you don't know\nDatabricks, Databricks has been built on\nsome open-source technologies like\nApache Spark, MLflow, and Delta Lake.\nUh we provide a bunch of capabilities on\ntop of it. So, the blue layer at the\nbottom is basically your cloud storage.\nDatabricks works on the three major\nclouds, Google, AWS, Azure.\nOkay.\nI thought it was for me.\nSo, so once you once you store raw data\non your cloud storage,\nuh the data is then um\num we we we bring in a a layer called\nthe Delta Lake layer, which uh which\nbasically brings in database-like\nproperties on top of your raw data. So,\nyou have got images, text files, video\nfiles, or whatever. We we help you\ncreate this um\nyou know, uh table-like structure on top\nof it using manifest files, right? And\nand we help you to uh incrementally load\ndata, do all of those um data management\ntasks in a structured way.\nOn top of that, we bring in Unity\nCatalog, which is a data catalog. Uh\nwith Unity Catalog, you can centrally\napply permissions on top of the data.\nYou can\num you can uh share the data using uh\nDelta Sharing, but also uh what happens\nwith Unity Catalog is uh you you can\nenable discovery and um you know, um\nownership, metadata tagging capabilities\nat the catalog level. What that means\nis, when you apply table a description,\ncolumn description,\nuh tag columns uh like PII columns with\nmetadata, it becomes really easy for AI\nto then get that context when it queries\nthese tables on top of Unity Catalog.\nSo, everything is governed at one layer\nthrough Unity Catalog, and on on top of\nthat we bring in different uh\napplications. So, whether it's AI\nthrough Mosaic AI, so to build LLM, tune\nLLM, or even build AI applications, we\nbring in uh data warehousing\ncapabilities, BI capabilities, and um\nuh and some of the other text-to-SQL\ncapabilities. We have got Genie that uh\nhelps you write natural language to do\nSQL querying, etc.\nAnd one application of that in the\nobservability and tracking tracking\ndata, as I was showing, is is this. So,\nbasically, think about when I was\ntalking about the tracking data\nstrategy.\nOrganizations, especially enterprises,\nwill not be running AI in just one\nframework. They'll be using different\nframeworks,\nCrewAI, LangChain, etc. etc. They'll be\nusing different cloud platforms.\nAnd once they do that, you need a\ncentralized layer of collecting that\ntracing data, so that you can serve sev-\nseveral use cases on the right hand\nside. So, whether it's for operational\ndashboarding, for first line support, uh\na lot of these uh first line\num first line of defense teams need\nhealth monitoring uh sort of dashboards,\nright? These teams can also write SQL\nusing Databricks Genie to do\ntext-to-SQL. But they can also build\nDatabricks apps using coding agents uh\nto create common workspaces or custom uh\nUIs that customers might need for\ndifferent uh different use cases.\nAnd then we've got Agent Bricks and\nMLflow that serves you uh LLM out of the\nbox LLM as judges,\nand uh proactively monitor a\nThe idea is,\nno no no matter where your AI runs, you\ncan create this kind of strategy\nbringing in data in one common place and\nserving uh different teams from one\nshared location.\nThe fourth pillar is multi-agent\norchestration patterns. As I said, one\nagent is good, multiple agents increases\ncomplexity. That's where you start\nthinking about, okay, what pattern is\ngood for my use case. The first one I\ndescribe here is the orchestrator worker\npattern. Where you have one orchestrator\nwhich orchestrates all the work, which\ncontrols all the work from a centralized\nplane, and then distributes this work to\ndifferent agents based on their\nspecialized skills.\nAnd then every request goes through the\norchestrator, so you have got central\ncontrol. If something goes wrong, you\ncan go to the orchestrator logs and look\ninto them and see what has happened.\nRight? So, that's the orchestration data\nuh pattern.\nThere is this choreography pattern where\neach agent is independent, they're\nautonomous, they don't depend on an\norchestrator. All of them talk to a\nmessage bus\nand they listen to the events that they\nare interested in.\nRight? So, think about agents that are\nindependent of each other, right? They\ncan run parallelly. So, they are not\nsequential, like one agent is not\ndependent on another. So, they run\nparallelly, they listen to the message\nbus for the\nfor the events that they are interested\nin. Maybe it's a trigger for, let's say,\na mortgage application, and it says, uh\nyou know, uh the mortgage application\nagent\nuh one of the agent uh looks customer\ndetails, right? The other agent looks at\napproval details and everything else,\nright? They can work in parallel, and\nthe advantage it brings you is the\nlatency is reduced because they are not\ndependent on an orchestrator and sending\nmessages back and forth. Right? So, this\nis the choreography pattern. And the\nthird one is human in the loop, which is\nwhere when an agent crosses a threshold\nor serves below threshold a confidence\nthreshold, then a human is called in the\nworkflow to look into the pattern uh so,\nlook into the looking into what the\nagent has done and then take action\nbased on that.\nI have\ndone a deep dive video on multi-agent\norchestration pattern\nuh for the online track of this\nconference. Uh it's already on YouTube,\nso you can look into it. I talk about\nthe real implications of when you think\nabout multi-agent patterns.\nOne is uh state management, the other is\nfault tolerance, like what happens when\nthings fail, like how do you manage\nthem? I talk about different patterns.\nAnd then talk about how how you think\nabout scaling them in large scale on\nenterprises.\nPillar five is governance, right? Now,\nhere I'm not talking about data\ngovernance at all. That's given, we need\nthat, right? From AI perspective, what\nwhat what are we thinking about?\nRegulatory, right? Audit trails, have we\ngot the trail of every action, every\nuser connection, every request,\neverything that happens in the system?\nAre we capturing everything?\nAre we doing pre-validation of personal\ninformation? Are we using name entity\nrecognition? The the easy stuff, the\nrejects and all of those things, right?\nIn our example, the work that I was\ndoing with the customer that I\nmentioned, we already detected 47 PII\nbreaches during the testing phase by\napplying this layer. So, that's that's\nreally important.\nFourth is um prompt versioning. You have\nto treat prompt versioning as change\nmanagement in enterprise grade solution.\nIt cannot be just change to a prompt and\ncommit to get. It has to be is it has to\ngo through proper change management\nprocesses as you do with code. So,\nbasically treating prompt as code.\nThird is model change management. So, as\nmodels change, the model providers\nupgrade these models, you have to have a\nsystem to understand whether that\nupgraded model will be good for your use\ncase, for your data. Right? Model\nproviders you will put evaluation\nbenchmarks on three uh benchmark uh\nboards,\nbut those are not really useful when you\nput them in your context, in your\nenterprise. So, that's where these\nevaluation data sets come in handy,\nwhere you try these different models on\nthis evaluation data set and try to\nunderstand which one performs better.\nAnd that management needs to be done\nbecause\nfrom a risk perspective, you cannot\nreally rely on one single model. You\nhave to have the flexibility to switch\nto different models and also test them\non your own data.\nThat management needs to be done.\nUh in Databricks, uh we have taken all\nof these these points, these pillars\nthat I've been talking about into Agent\nBricks. We are building Agent Bricks to\nmake uh all of these operations out of\nthe box for you,\nuh so that it's easy to implement\nproduction grade AI applications on uh\non in in your enterprises.\nSo, I wanted to quickly touch upon a\ncase study, just to give you a uh a\nflavor of how these things go, right?\nSo, when I was working with this client\num\nthey were a retail banking they were\nbuilding a retail banking chatbot you\nknow, one and a half 18 months ago.\nUh\ntheir their problem the the problem they\nwanted to solve is they had got around\n20,000 odd calls per month from\ncustomers on their chatbot. They wanted\nto deflect they they they saw that there\nwere like 60% of them were simple\nqueries, what is my account balance, you\nknow, what do I do with my overdraft and\nall of those stuff, like that can be\nanswered simply. So, they wanted to the\nreliance on human agents for those\nanswers. So,\nthey identified those queries and they\nwanted to automate them. Right?\nThey spent around 85K in 6 months doing\na POC which did not succeed. When we got\ninvolved, we found those insights that I\nwas talking like no one knew why things\nwere failing when it was in production\nwhen when when when it was\nin production. No one could actually\nmeasure\nwhy\nwhy it's not succeeding and no one could\nactually understand who is accountable\nfor what when things go wrong. Right?\nSo, the goal we set for them is AI agent\nhandles 60% of user queries, right?\nWhich were simple user queries and then\na way to identify and track them. The\nkey difference in this project that we\nwhen we did is that we selected the\nmodel in week seven, like in a eight\nweeks POC.\nRight? And this is how it turned out.\nFor the week one and two, we built the\nevaluation layer. We collected 200\ncases on their actual human agents\nanswering to their customers on simple\nqueries and understand how they are\nresponding to them.\nWe created that database.\nThen we defined the success metrics.\nWhat does success look like to you? So,\nout of let's say 100 queries, you need\n60 queries or the 60% of the queries\nthat are simple queries\nto be\nuh to be handled by the agent, right?\nThey needed some sort of accuracy. So,\n85% it was around 85% accuracy target.\nThey needed latency, all of the\noperational targets that you need. They\nwere there.\nThen we created this automated\nevaluation pipeline for them. And what\nwhat I mean by that is an automated\nsystem where you can capture a user's uh\na user's question and the AI agent's\nresponse. You take that, compare that\nagainst your evaluation data set.\nYou rate that, and if the rating is\nbelow certain threshold, you get it\nchecked by a human. And you if if\nsomething goes wrong, you make sure that\nyou find the solution. So, it could be a\nchange to the prompt, it could be change\nto a tool calling system or something\nelse. Once you have done that, you add\nthat test case in the test data set. So,\nthat when it happens next time, the test\ncases cases catch them. So, the the the\nthe summary of that story is that your\nevaluation data set is a living system.\nYou start with 200, maybe there is no\ncorrect number here, but once you start,\nas you start building in production,\nthis is a living system. This will keep\ngrowing. And the\nand the bigger it grows, the better your\nsystem will be.\nIn the second week, we talked we thought\nabout the foundational layer, right? So,\nthe the question data, we thought we\nthought that, okay,\nif you have to call the database, have\nyou got the API connections right? Have\nyou got a system in place that can trace\nthe API connections? Are those secure,\nright? We were not talking about MCP at\nthat time.\nRight? It was just direct API calls to\ndatabase to run queries.\nHave you got the distributed storage?\nHave you got the Have you Are you\ncollecting traces?\nAnd this is where when when we started\ntesting after building these systems, we\ncould catch those duplicate API calls,\nright? We could catch why customer\nsatisfaction was dropping and stuff like\nthat.\nAnd then comes in week seven to eight,\nwe started talking about models. Now\nthat we had the evaluation data set, we\ncould run different models on that data\nset to see the responses, compare them\nagainst the expected responses, and\ncalculate a number on on on the\naccuracy, right? That helped us to\ndecide which model to use.\nNow, that decision didn't take long,\nright? We in as I explained in the\nintroduction, like we spent weeks\ndebating on which model to use,\nbut when you took the other approach,\nyou can actually\nuh do that in a very quick way.\nSo,\nonce once that's done, we we stitched\neverything that I was talking about\naround observability, evaluation, the\nlayers of evaluation. Once we had that\nsystem that can make AI visible,\nmeasurable, and accountable, that's when\nwe started launching it to production.\nAnd that's when uh so, this is this is\nthe result uh six weeks post launch,\nuh we we calculated the operational\nmetrics, of course, you know, the\naccuracy, the deflection rate, the\nresponse time, uh the customer CSAT. But\nwhat's important here is\nin few weeks time, when uh there was a\nproblem with uh so, one of the one of\nthe things that happened was that the\nbank changed some uh\ninterest rate related policies. So, when\nthat when they changed the policy, they\nactually sent emails to customers or\nnotifications in the application in the\nin the in the mobile banking app about\nthe policy change.\nBut when the customers came and queried\non the chatbot for further questions,\nthey couldn't get the right answers and\nthey were like\nputting thumbs down on the answers. So\nthey were getting this feedback, right?\nSo feedback decreased.\nThe problem with this kind of system, if\nyou did not have this measurement\nsystem, is that you couldn't actually\nknow what's happening.\nBut because we had the measurement\nsystem in place,\nthese\nthe drop in C set was detected, right?\nBecause we were getting negative\nfeedback from customers.\nWe could actually look into the tracing\ndecisions and see that the agent was\nlooking at a policy policy document that\nwas outdated. So the the new policy\ndocument was not updated in the vector\ndatabase. The embeddings did not come\nthrough.\nBecause it did not come through, it was\ngiving it stale answers. And that's when\nwe went and fixed that. But it it was\nall possible because we built that those\nsystems\nthat\nthat led us\nto to detect this.\nBefore you go, I I generally in these\nsessions I share different\nartifacts that you can take away. I have\na QR code at the end for you to download\nand you will find multiple artifacts.\nOne of the important artifacts that I\nwant to talk about is the production\nincident playbook. This is something\nthat a lot of us tend to miss when we\nwork in AI projects. And this playbook\nis basically a definition of what needs\nto happen when things fail in\nproduction.\nFirst, you detect using your eval\ndashboard.\nThen you diagnose using your tracing as\nI explained.\nThen you contain. So basically, you you\nare versioning your prompts. Is there a\nIs there is a If there is a problem with\nthe prompt, you you take that prompt\nout, right? And start the changes,\ndeflect it to a human. Or in my\nmulti-agent orchestration video, I've\ntalked about multiple fault tolerance\nfailure recovery patterns\naround saga pattern, compensation\npattern,\nand circuit breaker pattern that you can\nlook into the video. I've explained them\nin details on how you can handle them.\nAnd then you use the test case library\nto fix. So you look into LLM's judge\nreports, you look into your evaluation\ndata set reports.\nThen you fix your problem. Once you fix\nyour problem, you put those test cases\nin your data set, right? And and create\nthat eval suite that is a living system\nthat will keep growing.\nAnd\nand and you you keep improving your AI\nsystem based on that, right? But this\nplaybook needs to be in place. When it\nruns in production, you will need to\nintegrate it with your ITSM system so\nthat it alerts the right person at the\nright time. You know, a lot of these\norganizations would have existing\nITSM systems, right? So which\nwhich is used for alerting and you know,\nmaking sure that the downstream systems\ndon't get affected, etc.\nSo\nonce once you have this in place, you\ncan go and stitch it together to other\nsystems.\nSo what can you do tomorrow, right?\nStart with If If you have a project in\nmind, start with defining success.\nSuccess not from the technical sense,\nfrom the business sense. What it means\nmeans for the business, right? Come up\nwith a few examples of what good answers\nlook like.\nAnd and create a data data set of that.\nAnd then build that pipeline using\nsimple Python code. See if you can\nautomate that so that when you run AI\nand get some response, it can become it\ncan go and compare You can go and\ncompare the answer against that data set\nand then that can be delivered\nto to the to the customer.\nNow, these are three lessons that I have\nlearned while doing these things with my\nyou you know, my easily miss.\nThe test case library, as I explained,\nis a growing system. It will grow over\ntime.\nAnd because it grows over time, you need\nsome sort of governance around it. You\nneed a owner, right? You need to You\nneed to figure out which test cases\nrelate to what kind of problem. So that\nwhenever you go back to it, you can\nyou can relate your answers to those\nsort of problems, right? If it is a\nsecurity If it is login, so you can say\nthat the agent did not ask for login\ncredentials when the customer asked the\nanswer. And all those kind of issues can\nbe put under a security category within\nthat data set. So to categorize the rows\nin your data set so that you can pick up\nwhat changed and compare it with them.\nThe second is prompt versioning. Now,\nwhen you start versioning prompts using\nGit, you know, we all know when you put\nGit message commit messages\ntend to be\nsimple commit messages. But you have to\nput governance around what kind of\ncommit messages you are putting in when\nyou're changing these prompts because\nyou need to understand when a prompt was\nchanged, for exact what reason it was\nchanged, right? What was the failure\nthat caused this prompt to be changed?\nWhat kind of failure would it address\nand what would it correct, right? In the\nnext version. That needs to be\ndocumented.\nOtherwise, it becomes difficult because\nwhen you go back and look into prompt\nversioning and look at different\nversions and you cannot trace why why\nthose changes were made, then it becomes\ndifficult to track what's happening.\nThe third, the layer three evals, right?\nSo the behavioral evals that I was\ntalking about around tool calls and\nstuff like that, they can be really\nexpensive as you grow your eval data set\nas well. So when you have a wrong tool\ncall, for example, and you want to\ncorrect that system, when you correct it\nand run it against the eval data set,\nyou have to basically run it against\nlet's say if you have got 300, 400, 500\nrows in the data set, you have to run it\nagainst them. And you do all the testing\nagain and again and again and again and\nagain,\nthat can cost you a lot of money. So you\nhave to put some governance around that.\nSo for example, when in your continuous\nintegration pipeline, when you do the\nprompt\nchange,\nyou can actually put\nsome checks around\njust just selecting a small subset of\nthe eval data set to do the testing. And\nyou only do the full test when you merge\nto the main branch.\nSo you can put these kind of decisions\nin place so that you can reduce cost\naround\naround\nyou know, expensive eval decision.\nIf you scan this QR code, it'll take you\nto a Google Drive link where I have put\nsome examples on\nsome of these how these templates look\nlike, what evaluation checklist should\nlook like.\nI've given you some guide on\nset setting up tracing\nwith open source technologies\nso that you can quickly set up some\ntracing and start testing in the test\nenvironment before you\ndecide on what kind of tools you want to\nuse.\nThank you very much for listening to me.\nThis is This QR code will take you to my\nLinkedIn profile. So I share\nI have a newsletter where I share this\nkind of topics every week. So if you're\ninterested, you can join. It's free.\nI basically share what I learn in in the\nfield working with customers, right? So\nit might be useful for you.\nThank you very much.\n>> [applause]",
  "transcript_chars": 36290,
  "ingested_at": "2026-06-18T16:30:31.636835+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": null,
    "like_count": null,
    "channel_id": null,
    "categories": null,
    "tags": null
  }
}