{
  "video_id": "_QAVExf_1uw",
  "channel_slug": "aidotengineer",
  "channel_handle": "aidotengineer",
  "title": "Agents Learn Best From Failures — Raj Navakoti, IKEA Digital",
  "duration_seconds": 4095.0,
  "url": "https://www.youtube.com/watch?v=_QAVExf_1uw",
  "upload_date": "",
  "transcript": "Uh\nThank you. Maybe we can get started. Uh\nfirst of all, thank you so much for\ncoming for the workshop.\nEspecially ones who didn't get the seat.\nUh\nI I promise you I'll do my best to make\nit entertaining especially for you for\nsitting.\nUh thank you so much.\nvolunteers, right? Yeah.\nUh actually it it makes sense. So now I\nknow like why the tickets got sold out,\nright?\nUh which workshop actually sold out the\ntickets.\nSo\nuh\nLet's start with uh my introduction. So\nI'm Raj.\nUh I work as a staff software engineer\nuh at IKEA.\nUh I work for a domain called delivery\nand services.\nUh basically we are like almost more\nthan 100 engineers and six product teams\nall together. It's like a mini company\nwithin the company itself.\nUh\nI'm very interested with architecture,\nneuroscience, and linguistics.\nAnd now AI. So if anyone have some cool\nprojects because everyone is building\ncool projects these days.\nPlease find me after this meeting.\nSo quick pulse check uh with the\naudience. Uh who is visiting London for\nthe first time?\nOkay, cool. Welcome to London.\nUh who is here uh from the engineering\nbackground?\nAlso with live coding and uh\nprototyping.\nNo, all of Okay. So uh who actively uses\nagents like Copilot or\nOkay, this is going to be tough for me\nthen.\nSo extensions?\nOkay, so everybody is pro. Okay, fine.\nNot so much as I want. Okay, so much\ntension now.\nUh\nSo you're going to sit here in this hot\nroom for more than an hour. So, first I\nwill uh give a bit of introduction of\nwhat I'm going to present today.\nUh it's basically on agent and the\ncontext management. Uh I I divided into\nthree parts. Uh one is the situation,\nwhich all of you already know, so I'll\nkeep it tight and short like 5 minutes.\nUh then I'll talk about the problem.\nThis is where I'll spend a bit more time\non the slides because\nI think like uh nobody is actually\nseriously looking into the problem.\nUh and I want to bring it up.\nThen less slides and more into some kind\nof a hands-on. Uh how the uh actual\ndemand-driven context actually works.\nAll good?\nOkay. So, let's start with the first\none.\nHow many of seen this movie?\nMemento.\nOkay. Okay. Okay. Cool. So, I'll I'll\ngive the gist of the movie. So, this guy\nis very skilled, uh very talented. The\nonly problem he has is uh he can't hold\nmemory more than 15 minutes. So, every\n15 minutes he has to take his notebook,\nuh watch his tattoos that he put it on\nhis one, and figure it out, \"Okay, what\nI was doing before the 15 minutes?\" And\nhe does it again and again. If you\nrelate to the AI and AI agents things\nand all, it's actually fits exactly how\nthe movie is and how the agents are\nright now.\nIf you go and watch this movie,\nyou don't need to watch YouTube or blogs\nto understand agents and MCP. This movie\nactually tells you about everything.\nLiterally. Uh\nand as this guy has a memory problem, uh\nin the same way the AI uh that we got\nintroduced couple of years ago uh is\nvery good with reasoning, computation,\nuh code generation. It's It's\nbenchmarked as like uh above the par.\nThe only problem is the institutional\nknowledge, right? The domain knowledge\nthat you have. That's That's the only\nthing we have to be uh a bit\nproblematic.\nSo,\nfrom AI to agent, if you look at the\nevolution,\nit exploded. So, it started from prompt\nengineering first of all.\nThen there was rags,\nMCPs, then multi-agent. Now it is deep\nagents. Uh\nI recently found out like\nuh using Replit actually, you can build\na full stack app in 10 minutes. That\nmeans the\nby the time you make the instant\nnoodles, you already have a\nmillion-dollar app already working on\nyour laptop. So, we got it to this\npoint.\nLike it's extraordinarily good.\nNow, that's AI and agents. So, let's\ntalk about enterprise AI.\nOkay.\nUh I don't know how how many have you\nhave this question, but most of the\nenterprises I see the question is,\n\"Okay, AI is pretty smart. It's doing\nuh code generation, full stack apps,\nreviewing your PRs. Uh\ndoing incident management, all those\nthings.\nRight? So, if AI is doing that much, why\nis the Jira tickets or epics are not\nmoving on the dashboard, right? Why do I\ndon't see the delivery actually? So,\neverybody is speaking about look at like\n3 minutes, everything is ready. Yeah,\nokay, fine.\nWhy are my Jira epics are not moving?\nBecause that defines the business\ndelivery and that defines the return of\ninvestment, right?\nUh\nand as you see, like uh it's it's from\nthe McKinsey this year, like 88% of all\ncompanies use AI, but they only see like\nuh 6% of value creation.\nOkay, so I think this is the problem\nuh that we have.\nUh I have four Jira tickets, different\nones,\nuh sample ones. And you can see the\ngreen ones that I have uh marked as\nbasically which LLM is or already\ntrained on like APS standards or like\nthings very they already know it's a\ngeneral knowledge, right? That's fine.\nThose tasks from the ticket it can pick\nup and it can do it. Now, there are\nsecond part orange ones which we have to\nteach them actually. So, you have you\nknow this but do this in this way. So,\nall this kind of an orange color things\nwill fit into, you know, your agent\nextension like skills or like uh\nBut, the red ones\nthat's what the institutional knowledge\nis which sits within the company and\nwithin the people.\nSo, unless if it picks a task uh if it\npicks a ticket, it has to fulfill all of\nthem.\nIt is so good at with uh green ones and\norange ones but it struggles with the\nred one with the institutional\nknowledge.\nAnd what I believe is uh\nright now the coding agents are getting\nso so better.\nI feel like if there is an AGI coming,\nthe first AGI will be a coding agent for\nsure.\nUh\nSo,\nto fix the uh\ngiving the institutional knowledge to uh\nthe agents, so we have an industry\nsolution already. So, this is basically\nyour return of investment on AI pipeline\nwill look like, right? So, you have LLM\nmodel quality,\nuh you have agents, and you have agent\nharness.\nAnd your your institutional knowledge\nsits under Confluence, Jira, SharePoint,\nGitHub, all those things. And basically\nretrieval layer is what industry is\ntelling us will fix that issue. So, you\nbuild that a retrieval layer and then it\nwill fetch all those things and give it\nto an agent and the agent should be able\nto do it, right? So, basically uh\nSo, 40% uh\nactual factual accuracy can be achieved\nthrough rag or like knowledge graphs\nactually.\nUh\nbut with a documented knowledge base.\nNow,\nbasically if you build a retrieval layer\nit has to work, right? Now, let me ask\nyou a question. How many have you built\na retrieval layer things like rags and\nMCPs?\nOkay, cool. All of them. Okay, now the\nquestion is how many did you build?\nHow many MCPs did you build? How many\nhave you built more than 20 MCPs at\nleast?\nOkay.\nOkay, so nobody beats my record then.\nUh\nso\nwhat I see is mostly in the enterprise\norganizations and all people are\nbuilding like 10 to 15 or like 20 MCP\nservers or like rags,\nknowledge graphs on top of their\ninstitutional knowledge and\nto the agent, right? So, the assumption\nis if we if I can build all those MCP\nservers and give that agent I don't need\nto work anything like it will do.\nBut the thing is all when you\nplugging this MCP servers basically all\nthis data coming out is mostly\nundeterministic.\nIt's unreliable and it's untested,\nright? So, especially in engineering\nnobody does e-vals. Actually, it's it's\nmore like a data machine learning\nconcept but we don't do e-vals. So, for\nme if you\nSorry,\nuh\nif you plug an MCP or like a rag and all\nwe see whether the output is coming or\nnot rather than is it really valuable\nactually? Is it really solving the\nproblem or not? Uh that's the main\nproblem that I see. I'm not saying\npointing other people because I was that\nperson.\nI was like, \"Okay, let me build all MCP\nservers, plug in my institutional\nknowledge. I'm going to prove the point\nthat\nagents can semi-autonomously can\ncontinue and fill those Jira tickets and\nfinish it, right?\"\nBut every time when I build those MCP\nservers,\n10% 20% 30% of time it was accurate, but\nrest of the time I was doing the data\nentry job for them actually. So, I was\nfilling the gaps, answering the\nquestions. So, basically I'm doing more\nwork than actually doing less work.\nUh\nSo, I think this is the main problem.\nAnd I I actually was in this fourth\nstage where I literally started to write\nthe domain context with handwritten\nactually. So, okay, let me write\neverything and prove the point, but I\ngot really exhausted of doing it.\nOkay. So, how I don't know how many can\nyou relate with this pie chart? Uh but,\nmost of the enterprise\nuh\nthe institutional knowledge is kind of\nsomething like this. So, 20% if you see,\nit's\noutdated. 20% it's unreliable. Uh 20%\n10% is always duplicated with different\nplaces.\nAnd the major problem is 40% of the\nknowledge is always uh tribal knowledge,\nwhich means people know how things work.\nSo, it's it's never documented actually.\nSo, in this situation of an enterprise,\nand you build like 100 MCP servers and\nplug into that monolith, it doesn't\nmatter how many you build, it won't work\nbecause basically your whole\ninstitutional knowledge is a is a\nmonolith.\nUh I think like because you're all from\nthe engineering background, so you\nalready know uh the transformation of\nmonolithic legacy system to\nmicroservices, right? So, in the same\nway, unless we break down that monolith\nknowledge base into some kind of a\ncontext blocks which are useful for\nagents, then only we can actually\nmake it useful for them.\nUh for the agents and actually make them\nsemi-autonomously can actually do the\ntasks.\nSo, that's it. Uh we are going to talk\nin this workshop mostly on that\nmonolith, how to break it, uh what is\nthe approach to break it? And how it is\nuseful when once we break it. And this\nis a job we need to do because\nthe LLM providers will\nfocus on the LLM model quality. The\nagents will focus on the harness things\nand there is a big\nretrieval market of 9 billion. They're\nfocusing on retrieval. But nobody is\ngoing to come to your company and fix\nyour knowledge base. You have to fix it\nyourself, right? So, how can we do it?\nOkay. So,\nthe demand driven context is what the as\na solution I was trying to propose.\nRight? So, basically, if I have to give\nan abstract of it, what it is,\nlike we have Mike monolith\nservices and we have this process of\nbreaking them to microservices. We have\nwaterfall model which we transform into\nagile. In the same way, when you have a\nmonolith of institutional knowledge, how\ndo you transform into a context blocks\nusing an approach? So, this is an\napproach of how we can do it.\nUh, before starting,\njust not an idea. So,\nwe already tried with some data sets and\ntry to prove this approach works. And in\nthe March, we have published a preprint\nin RXP. So, if anyone interested in\nreading up academic papers, you can find\nit with the demand driven context or\nlike I can also give you a link after\nthe\nworkshop.\nOkay.\nSo, how does it work actually?\nUh,\nwhen we are giving institutional\nknowledge to agents, basically, what\nwe're trying to do is we're trying to do\na push strategy, right? So, we build\neverything and we push it to to it.\nSo, in in this approach, it's more pull\napproach.\nUh, which means, for example, let's say\na new joiner has joined your company,\nright? How do you onboard a person? So,\nyou onboard them for a 1 2 days. You\ngive some initial orientation and then\nyou tell them like okay, these are the\nconference links, these are the GitHub,\nthis is the some some kind of a\ndocumentation you have to follow things\nand all. Then you just assign a task to\nthe person.\nSo, but you're not going to tell okay,\ngo and get graduated on on this\nknowledge and come back then I'll give\nyou work, right? So, you'll just assign\na work item.\nAnd when you assign the work item, the\nperson will start asking questions, fill\nthe gaps. If the person is very much\ninto documentation, he will also fill\nthe documentation for you.\nHe gradually\nget his knowledge of of the\ninstitutional knowledge, right? In the\nsame way\nwe don't push all the knowledge to the\nagent rather than we start giving\nproblems to the agents like work items\nand let them actually pull the\ninformation from us.\nAnd once\npull the information\nalso ask them to document it.\nUh in a in a better way rather than in a\nmonolithic structure.\nSo, if you\nSo, that's the four layers. So, you have\na monolith\na framework and it actually pulls and\nactually creates a good better context\nblocks. You can actually relate it to\nmore into a\na legacy monolith to microservices\ndirectly if you have to have an analogy\nof it. So, this is how it works.\nSo, this is one cycle of uh how\na problem to an agent and in the first\nattempt the agent will fail to do it.\nSo, it will say you know what?\nYou gave me a problem, but the most of\nthe documentation I couldn't able to\nfind anything. I couldn't able to do it.\nThen these are the things I need to do\nto finish this task and it gives a\nchecklist of things. So, we fulfill the\nchecklist like we fulfill the checklist.\nSo, once it is\nuh, given, the problem is solved, it\nwill take that knowledge and also it\nwill update, that means curate the\nknowledge in a particular place so that\nit can reuse or like other regions can\nalso reuse. This is one cycle. So, the\nidea is if we can do it in multiple, uh,\nsessions\nwith multiple problems, so it will\ngradually, uh, curate your knowledge\nmonolithic knowledge base\nuh, and also document it for you.\nUh, you can also relate it to TDD. So,\nhow many are\ndo TDD or like\nNobody hates TDD, right? Before before\nI, uh, yeah. Okay. Okay. So, uh, in the\nin the same way, right? In in a TDD\napproach, what we do, we just write the\nfailed test cases. We don't build the\nproduct first of all. We just write the\nfailed test cases.\nUh, we see what is a code that is\nmissing uh, for the failed test case to\npass and we just give that code and we\ngradually build the a product based on\nthe failed test cases. In the same way\nwe give problems that agent will\ndefinitely fail\nand we gradually, uh, fill those gaps\nand at a certain point it becomes\nsemi-autonomous with a good, uh,\ninstitutional knowledge already.\nOkay.\nSo, I think I can jump into, uh, some\nkind of a,\nuh,\ndemo already. Uh,\nSo, I will use terminal, so don't hate\nme.\nUh,\nI think like all of you're from\nengineering background, so I think like\nyou'll like terminal.\nUh,\nLet me switch to terminal.\nOkay.\nOkay. So, on the on the far left what,\nuh, right what you see is how, uh, under\nthe hood it works actually. So, when you\nhave given a problem,\nhow does the agent will fail? How does\nit demand for the knowledge\nthat the problem has to be solved? And a\nhuman like a domain expert and all\nfills those gaps and then it will curate\na new knowledge base for you, which is\nwhich is much better. And then the agent\nsucceeds and you can repeat on the next\nproblems. So, that is how one cycle of\nthings are done. So, how I it can be\nimplemented it can be implemented using\nany agent. There is no it can be\nimplemented on cloud or co-pilot because\nit's an approach you can do in any way\nyou want. At work I use co-pilot. So, I\nimplemented this using co-pilot\nbut because everybody I I believe loves\nmore cloud code. So,\nI created this demo with cloud code and\nyou can see it's just a combination of\nskills, rules, agents and hooks and some\nkind of a place to save the knowledge\nbase.\nOn the middle pane what you're seeing on\nthe top is your monolith basically.\nThis is a representation of your\nconfluence, Slack, GitHub and all but\njust for a sake of demo, I just put some\nflat files that look like them. So, that\nis how your monolith knowledge base will\nlook like. On the down what you're\nseeing is on a live. So, when it's\nsolving a problem, how it is actually\nadding the new knowledge to it. So,\nthis is how\nlet me\nSo, let me\nSo, what I'm going to do is I'm going to\ngo to the agent.\nOkay, I'm going to basically give an\nincident problem to do the root cause\nanalysis, right? So,\nOkay, what I did is you remember in the\nprevious slide there is a Jira ticket\nsamples that I showed, right? It's a\ncombination of\nknowledge that is documented, not\ndocumented things at all.\nSo, this incident also represent the\nsame kind of combination. So, there is\nsome knowledge that is documented\nalready on your monolith, some it is not\nthere or outdated or things like that.\nAnd most of it, it doesn't couldn't\nwouldn't able to find because it's never\nwritten down actually. So, when I gave\nthis problem, so it uses those\nskills that I have developed using this\napproach. And it will try to actually\nfirst go to your monolith actually on\nthe knowledge base and try to find\ninformation on what is\nthere.\nSo,\nthink about it like this. So,\nfirst part is retrieval. That means it's\nalready doing what RAG and MCP is doing.\nFirst part. But what else it is doing is\nafter it fetches the data, what it will\ndo with the data actually. So, that is a\nmissing part. For example, when you give\na new conference links to a new\nemployee, the employee goes there, looks\ninto it, but doesn't find information.\nBut it he doesn't stop there actually.\nHe continue asking questions\nso to solve the problem then then just\nadding more knowledge and things and all\nlike Those are the next steps missing\nright now. It's we just stop at\nretrieval.\nSo, this is the next three steps it does\nactually. So, you can see the confidence\nscore is almost one to five because it\nsays\nThese are the particular terminologies.\nI don't understand actually these\nterminologies. And these business logics\nis not needed. So, one thing you need to\nlook at here is whatever it has said,\nthis is the undocumented information.\nThat means it was never written down.\nSo, unless you don't do this way, you\nwill never know what is not documented.\nFor example, if somebody says like,\n\"Okay, there is documentation missing.\nWe need to write.\" Okay, what do you\nwant me to write actually? So,\nthere is so much in the people's head. I\ncan't write so much. It's somehow it has\nto surface.\nSo, when you give a problem, it actually\nsurfaces what is not documented. And it\ntells me, \"Okay, this is missing. I need\nto have a new information there.\"\nUh so, what I will do is so\nit does all the three steps. So, then\nwhat I will do is I already have a\npre-prepared answer.\nVery high-level pre-prepared answer I\ngave it to it. Uh\noff like what is the missing\ninformation. So, okay, this is the\nmissing information you asked me uh to\nsolve this problem. Can you solve this\nproblem now?\nUh\nI didn't expect this one. Okay. Uh\nNotification is yes. I'll just say yes.\nIf that's the sole what fictional name\nshould I respond? Uh okay, I didn't see\nit.\nSee, when I did the test and it didn't\nask me the questions.\nLet's see. It knows it is a demo and\nI trained it to recognize.\nOkay.\nNo, it is already what it is doing is\nalready\nSo, you can see on the live it is\nalready adding the entities that the new\nknowledge base has been come into the\nplace.\nSo, the knowledge base is\nmanaged as a file system, right? Or as a\nsystem files? For the demo, I'm just\nshowing it as a file system, but it's\nbasically your MCB servers\nthe data will stay in confluence, Slack,\nor things. Uh you can just plug in and\nuse the same MCP servers or rack and\nall. So, it don't need to be a flat\nfile.\ntreat this as your system slide for for\nthis agent.\nRight?\nUh It's like a map. Do do you use any\nlike a memory tool for for this? Yeah, I\nwill show you on the next slide how I'm\ngoing to save. Uh\nOkay. So, it started from 56 entities or\nsomething with the this one, right? Now,\none problem actually surfaced\nsix entities that are never been\ndocumented.\nAnd when I gave that information to it,\nit is able to actually discover, curate\nanother five or six\nnew entities that were never documented.\nSo, it does discovery of the gaps. It\nalso gets information from me and also\nstores information, new information, and\nall.\nUh\nthis is one, okay? Next, let's see.\nThis is a busy window. I tried to\nactually do things, but uh it didn't\nwork out. Okay. So, what you're seeing\non the on the window is like 14\nincidents.\nYou have seen one problem that I solved\nwith an agent, right? The communication.\nWhat if I took like 14 incidents uh and\nI just go and have 14 cycles of this\nthing and how it does. So, if you see on\nthe left side, it was the first\nincident, right? So, right now, it has\n1.5 confidence.\nAnd everything is critical. Every So,\nbasically, nothing is documented. So,\neverything is critical, high, and the\ndata is missing. So, I started giving\nanswers to in the first incident, then I\nrepeated for the same second and third\nand continuously for like 14 incidents.\nBut, on the 14 incidents, it basically\nactually able to go to a confidence\nlevel of 4.4 because first it discovered\non every instant, it got the list of\nanswers for me and also it documented\neverything for me. So it gradually from\n1.4 to almost like five range of\nknowledge.\nIt improved.\nSo if you look at the traditional way in\ntraditional way what we do is we solve\nall the context problem.\nRight? We have to deal with it first\nthen I have to give it to agent. In this\none\nwe are moving agent from consumer to a\nknowledge manager.\nSo you just don't consume from me. I'm\ngoing to tell you but the whole\nknowledge management is also your job\nand you have to do it for me.\nOkay, I think we can get back to the\nslides a bit.\nOkay, so what we have seen is\nwe have I have run one cycle and also I\nhave shown how it look like when I run\nin like 15 or 16 different cycles,\nright?\nBut if you have want to do it manually,\nit would be really painful because I\ntried it after 15 cycles like nobody\nwould like to actually sit with an agent\nand you know you have an incident but\nyou won't be sitting with your agent and\nkeep actually asking questions and\ntelling you about your problems, right?\nSo that is super painful.\nSo\nbut the thing is we can automate this\nprocess. So this is where actually it's\nit's really good and gets interesting.\nSo here is the thing.\nYou all we already have all the work\nitems, right? We have Jira. We have\nincidents.\nWe have\ncustomer support tickets like that. All\nthose kind of a work items already\nthere. Right? Sitting in the archive.\nSo why can't we take\nthem and actually use the framework\nand validate across your knowledge\ndatabase, run an automation and see\nactually what is the state of your\nactually right now.\nOkay, let me see let me show you how it\nlooks like. So,\nrather than actually doing it manually\nat a scale if we do this approach, so\nhow does it look like? So,\nthe demo that you're seeing is almost\nlike everything is preset.\nFor example, I have the demo. I have\nlike a platform operations agent and uh\nI'm saying, \"Okay, these are the recent\nincidents. Let's say I have 20 recent\npast incidents I have.\" Uh like an MD\nfile or a JSON file, right? It has all\nthe details of description of it, things\nand all comments and everything. And the\nrest of the files are your knowledge\nbase. So, it's it's a file system, but\nyou can also actually connect with the\nsame way confluence and things and all.\nJust for a demo purpose, I'm just\nshowing it as a flat files.\nNow, what I'm trying to do is\nI'm going to take all these incidents\nand validate each incident across my\nknowledge base\nand ask the agent, \"Okay, tell me\nuh how much of the document is good, how\nmuch of the documentation is I can't\ntrust it or like old or outdated, and\nhow much is actually missing, not\ndocumented as per this incident?\" So,\nlet me\nrun it.\nOkay, it will take some time. So, it\nwill take three steps actually. So, one\nis uh it generates probes, which means a\nbasic test it will write to actually\ntest your knowledge. Uh\nThen it will run those tests and then\nanalyze the gaps actually.\nOkay.\nIt's a little bit hot in the room\nactually.\nI'm going to identify the\ngaps. it's just like a clever problem,\nso I imagine.\nUh okay, for example, let's say let's\nsay you have an incident called the\nnotification service is not sending\nuh customer\nuh messages to\nuh\nSMS service, right? So, the notification\nservice then you mention that the agent\nsees is there a documentation related\nnotification service.\nSo, it doesn't find, that means you\nnever wrote a documentation on\nnotification service. I do understand\nwhat is the customer\nSMSes things and all. The customer\nnotification service when you mention,\nit's a gap because it's never\ndocumented. Or\nit takes the customer notification\nservice, goes to confluence, and sees\nlike the documentation how old it is. If\nit is says like uh\nit's like 1 year old, it will tell you,\n\"Look, I looked into it. It's like 1\nyear old. I don't know whether I need to\ntrust this documentation or not.\" Or\nlike incomplete uh documentation. So,\nif you see it's called each incident it\ntook it, and it looked at all the\nknowledge uh base that you have\nconnected, and have consolidated list of\nlike scoring of like okay, partially the\nagents can handle uh the basic\nedge cases of the incidents that you\ngive\nuh because your knowledge base is not\ncomplete, and it will show you how much\nof the tribal knowledge is missing,\nsystem information, business process,\nwhat are actually you're missing from\nyour\ninstitutional knowledge, when whatever\nis not documented.\nUh this are the probes.\nAnd it will also identify uh what is\ncritical and what is high. This is\nreally important because\nuh let's say there is some kind of an uh\nexample of notification service which I\nmentioned, right? It is repeatedly\nuh appearing in like 20 incidents, and\nyou don't have\nThis is the first that you need to you\nneed to as per your documentation. So,\nit will also help us actually understand\nwhen you're\nuh\nbreaking down your knowledge base\nyou need to understand what is critical\nactually, what I need to focus on first,\nwhat makes value for me.\nSo, we'll organize into critical, high,\nmedium.\nSo, this is what like I showed the\nflat files, but you can also connect it\nto the various data sources that we\nhave.\nSo, the step one is basically what it\ndoes\ndemand extraction. That means every\nincident it will extract\nthe checklist of information what is\nmissing.\nOn the second step is what it will\nconsolidate everything what is missing.\nSo, it will create like systems and APIs\nand all and how many are clean, how many\nare stale, which is incomplete, what is\nentirely missing. Uh something is\ntribal.\nUh those kind of a classification it\nwill also do.\nAnd it will create a Kanban board for\nyou. So, what happens is So, if you want\nto fix your institutional knowledge\nbase, basically, you just just like Jira\ntickets, we finish it. We actually has\nto document these missing pieces and\nall. And the the moment you started to\nSo, it also saves in the context like\nSo, it also has to build its own\nknowledge base.\nAnd you can see the performance. So,\nonce you're fixing the tickets on the\nKanban\nuh\ninstitutional knowledge.\nSo,\nthe how So, what we've seen is one is\nthe approach first of all, which means\nnot the pull push approach, but the pull\napproach, how to do it.\nOne cycle or multiple cycle, how it look\nlike. But, if you put it in a scale of\nautomation, then how much valuable it\nwould be.\nOkay. Now,\nthe important question is\nShould I do a voice over later,\nactually?\nI have the patience. Who has the\npatience? For a second.\nSo,\nthe question is so I was all all the\ntime I was talking about, okay, receives\nthe context, we give the information, it\nwill store it. But, the question is\nwhere does it actually store it? So, I\nhave a very opinionated opinion.\nHear me out. I prefer it has to go to a\nGitHub repository because eventually\nsomebody will actually come up with a\nyou know, 20 million seed funded SaaS\nsolution for you. But, before that, I\nprefer\nto actually put it in GitHub as a\nrepository. Why? Because if you look at\nit at a scale, if you want to do this,\nthere will be multiple agents, multiple\nteams actually contributing to the same\nknowledge base. And there will be\nconflicts and resolutions, right? So,\nthe\nthe easiest way to do is using GitHub\nbecause it actually comes with inbuilt\nuh\nuh PR processes, review processes,\nthings and all. So, if multiple domain\nexperts are sitting and uploading the\nfiles or like agents are contributing to\nit, the most efficient way to manage is\nin a GitHub, something like a structure\nlike this.\nAnd the other advantage is also if you\nput it on GitHub, you can also publish\nit to Confluence later or like Slack\nwherever you want to publish it to on\nthe\nsolution that you want to use.\nUh so, I prefer to have it on GitHub,\nbut if you want to directly integrate it\nto Confluence and all, you can also\ninstall do it.\nNext is\na meta model.\nHow many are aware of the word meta\nmodel in the\nOkay, maybe I can quickly show you how\ndoes it look like.\nSo, meta model is basically something\nlike this, right? So,\nin how does your\nuh\nuh domain actually structured around\nlike\nis a business process are how it is\nrelated to a system. How systems are\nrelated to\nuh APIs.\nAnd how is this\nbusiness jargon or like a tech jargons\nare actually linked it to which one. So,\nthese kind of a relationship\nmeta model is really important. It's not\nnecessary\nfor the approach that I have proposed,\nbut it's an add-on. And why uh you need\nto have this one is right now think of\nit is like a map. Right now, your agents\ndoesn't have any map actually to\nnavigate with your knowledge base.\nBasically, what you're doing is you're\ndumping like\nthese many number of files and it need\nto figure out which file I need to need,\nright? But, your file structure is\nactually\na representation of your meta model. It\nactually knows how to navigate. For\nexample, let's say can you fix this\nsystem? It will understand if I change\nmake changes which business processes\nwill be affected and which APIs I need\nto change or like touch these kind of\nthe things.\nSo, it's also important to have a meta\nmodel.\nIf you have it,\nuh\nthen it will produce more value. So, I\nstrongly prefer to have a meta model\nalong with this approach.\nOkay.\nSo, the last part is what is the value\nit created? So, it\nThere's a lot of slides that you have\nseen, a lot of demos that you seen. So,\npersonally I need to also share like\nwhat's the value that I see when we\nI was using it or like the other people\nwho I shared with already were using it\ncame back with a feed feedback and told\nme. Uh first, the most valuable thing is\nknowing the unknown.\nSo, what is never documented is\nsomething can be surfaced only by this\napproach actually.\nUh otherwise, you will just end up in um\nan endless Miro board of like putting\ntickets on like okay, this is missing,\nthis is missing, I need to add it, I\nneed to add it and keep on doing it. So,\nthis is the fastest and better way to\ndiscover uh with your previous work\nitems and all what is never documented\nuh things and all. Uh second is uh\nbasically\nI can now\ngive work to agents rather than I do all\nthe things. Like rather than I become I\ngive the agents all this information.\nLet it manage my knowledge management. I\ndon't want to be the knowledge manager\nof it. So, let let it do it. So, those\nare the two big values that I have seen.\nIf you want to use it, I think like you\nwill also see the those two as the most\nvaluable. Uh\nbut these are the other things what I\nseen.\nNow, okay. So, I also need to\ntell you like what is the\nuh uh drawbacks of also using it, right?\nFirst of all, if you are coming from a\nsmall team or like if you say like no,\nno, no, my documentation, my knowledge\nbase is really good.\nI'm like super happy for you. Uh\nyou're the lucky ones in this world\nright now with agents. Uh\nfor you it might not be really relevant\nunless you have a very\nvery complicated\nuh documentation that you have.\nUh second is I already mentioned the\nmanual manually doing is it's very\npainful. I don't prefer anyone to do it.\nIf you want to just try it for testing\npurposes, you can also do it. But, uh,\nautomation is the most best way to\nactually use this one.\nUh, this is very early, this approach.\nSo, by tomorrow morning on YouTube,\nsomebody would have already posted\nsomething differently,\nuh, better than me. So,\nuh, in the in the\nera of AI, nobody knows like, uh, how\nlong a thesis or an approach or an app\nproduct going to survive. So, for now, I\nsee this is the best approach.\nOkay, so\nthe whole workshop, so we started with,\nuh,\none pipeline, right? On the ROI. And so,\nthe demand-driven context actually sits\nbetween this monolith and also the\nretrieval layer, actually.\nAnd what it does is,\nuh, it actually helps you\nbuild curated context blocks for you.\nYou can also think of it like a\nuh, cache database that you have. So,\nevery time your agent doesn't need to go\nand, you know, boil the ocean for your,\nuh, fixing an issue. Rather than if you\nhave a good context block of\ninformation, most of the time, 80% of\nthe time that can be usable. Because\nwhat I also believe is it's always the\n80/20 percent rule. So, 20% of your\ndocumentation is most useful. 80% is\nsome corner cases you have to look into\nit. So, rather than giving 100% of\nthings, you need to figure out of what\nis my 20% of that, uh, that is super\nhelpful for agent and have it like a\ncache database, uh, the context block of\nit using it. And rest of it, you can\nleave it like, uh, links. So, whenever\nagent feels I need more information,\nthen only it can go and check the, the\nwhole, uh, monolithical, uh,\ninstitutional knowledge.\nOkay, so from here, what you can take\nfrom this workshop is three things. One\nI hope I makes I made sense of this\napproach. So\nthere is a GitHub repo\nwhich I detailed it out and also a\nstarter guide on it one if you want to\ngo home and try with it and you can try\nit. You already have know how the\nframework works. So you want to go home\nand just remix the whole approach you\ncan do it and let me also know.\nI'll leave this one and I'll join with\nyou for contribution. You have a context\ngap gap scanner that I showed you which\nis live already with presets. I think\nlike added like $20 on it. So hit it as\nmuch as possible. You all right? Okay.\nOkay. So after $20 so first come first\nserve.\nSo all these three you can use\nyou can take away from this workshop.\nOkay, so\nbecause this is a workshop so I also\nwould like to\nwant you to try something. What you can\ntry is three things. One is\neither\nif if you say like you know what I'm so\nso tired already it's almost like four\nit's almost about to go for a party. I\ndon't want to do it. So you can just go\nto the context gap scanner.\nEverything is a preset here. You can\njust try it out hit it and see how it\nworks. If you think it can be done\nbetter let me know so that we can work\nwith it.\nUh\nor otherwise let's say now I'm I'm very\ntechnical. I want to know how it works\nunder the hood. This is a GitHub\nrepository. Uh it's it's under maybe\nI'll just take this out.\nUh this is a GitHub repository and it\nhas all the information.\nUh plus there's is guide also if you\nwant to if you want to try it out. But,\nuh, if you still feel like no, I want\nmuch more simpler.\nYou can also try this one. So, you don't\nneed to do anything. Basically, take\nthis prompt,\nuh, take one of your Jira ticket or\nincident that you have right now. If you\nalready built MCP servers, uh, or like\nany other kind of a things, you just use\nthat prompt, give it to your agent, uh,\nwith the incident or a Jira ticket, and\nask it, uh, give me the quality of the\nknowledge base that I have as per this\nincident or\nJira ticket in this way, and see how\nmany how much of it comes in the red,\nwhich is never documented. So, you can\nyou can try also this simple one.\nI just leave it like this. Maybe you can\ntake a picture. Maybe I can switch to\nthe slide if any anyone want to.\nCool.\nThis is your slide, you can go.\nYeah, so I see some problems in this\napproach, but I find it very\ninteresting. So, my my first question is\nhave you already used this way of\nworking at scale? Or because we've seen\nmostly toy examples, right? Yep, yep.\nUh, I used it, uh,\nnot at a scale. I started with simpler\nbecause you also need to see what is the\nscope of it. Let's say I have an\nenterprise, and I try it at enterprise\nlevel, I can't do it because it's\nmultiple domains, things, and all. Even\nif If do it at domain level, I need to\nunderstand I tried it at domain level.\nThen even at a domain level, there is so\nmuch of a domain expertise I need to\nfill it up and\nfill those gaps. So, again cut down into\nmaybe what is the smallest team that I\nhave and the smallest team's Jira\ntickets, the smallest team's instance,\nand the team's Confluence page.\nUh with a bit of a scope. Then if I\ndrill down the scope, then I feel like\nit's more\nfast, more useful. But if I do it at a\nbigger scope, what happens is\nnot one person has the whole\ndomain expertise. So, basically it again\nbecomes like\nsomebody has to come and\nyou know, five or six people has to sit\ndown and start doing these things. Yeah,\nI'm I'm a bit concerned that this might\ndenial of service attack your your team\nmembers in a certain way because our\nLLMs are fine-tuned to keep eliciting\ninformation, to keep\ngetting more information out of us, to\nask follow-up Okay. Um so, I think it\nwill be hard on the engineers that have\nto do the question answering.\nUm and secondly, the the scanner is\nnice, but that's still built on\nthe assumption that all of your team\nmembers and the rest of the enterprise\nare still using your enterprise IT\nwell, as planned, that they're actually\nfilling in their tickets with all the\ndetails and etc. And I know from\npractice that that is most of the time\nnot the case.\nUh that is true. That is true. I agree\nwith you. Even if I\nmy my assumption is also even if I go to\na leadership to buy in, like, \"Hey, can\nyou give me a bandwidth or like\nyou know, I need these people to\nactually sit and fix the context?\"\nI don't think right at this point of\ntime nobody will do. But I think it will\nhappen because slowly I think we are\nslowly moving towards in agent managers\nwhere agents are becoming\nsemi-autonomous or autonomous, and we\nmanage them. But at the certain point of\ntime, somebody has to fix that knowledge\nbecause it's not going to come from\nanywhere. You have to. So then the\nenterprise focus will shift towards the\ngap. That's what I started saying, I\ndon't think nobody is looking into the\nproblem yet. Everybody is very focused\nwith agent how good the agent is, how\ngood the retrieval is, but how good the\ncontext is,\nyou're not solving. It I think like in\ndown the line in a year or so,\nI think people will realize importance\nof it and uh\nthe Kanban board will definitely come\ninto reality actually very soon.\nYeah, thanks.\nYeah, I think actually they're going on\nthe same price. I think when we look at\nlarge which applies actually the\nsource of truth is not actually the\ndocumentation, it's actually the code.\nJust wondering have you applied it to\nthe code base?\nUh I did. Uh I also applied it the code\nbase. Uh\nbut I got a mixed result when I So\nSo there here is the thing. What\nhappened is when I only use code base,\nuh it is particularly good or when I\nonly use Confluence or like textual data\nuh like uh\nuh it gives a good results, but when I\ncombine it, somehow actually uh it\nconflicts because it it creates a theory\nout of the GitHub repository, but the\nsame GitHub repository documentation is\nalso on Confluence.\nSo there it gets a conflict of Okay,\nwhat is the source of truth? Code says\nthis. Should I implement it this way as\nper the documentation? So then again I\nneed to create an additional skill or\nrules like Okay, what is the ranking\nthat you need to give? If you see it in\na GitHub, that means that is the source\nof truth.\nOr if you see it if you don't see it,\nthen you have to uh\nlook the information in Confluence\nthings and all. But those are still\nI'm trying to fix those things actually.\nSo seeing the gaps and fix those things,\nbut I definitely see uh that issue\ncombining those two. And the second\nquestion is\nBut interestingly, it's actually\napplying the same approach and skills\nbecause what we find out is actually\nlike you have your um\nkind of like your um\nlike the the process starts like by\nrunning agents, which is bringing a\ncontext and identifying the right skills\nthat need to be used, right? Mhm. Then\nyou go and do the task and you fail,\nMhm. right? Then once you fail, you\nidentify what you need to solve. Mhm.\nYou go back, you curate, you you fix,\nbut then you're fixing the knowledge\nkind of base.\nWhat we find out actually is we go back\nand fix the skill.\nOkay.\nThat's the increment of the\nthe next situation.\nSorry. So, I'm not sure if skill could\nalso be a skill that skill that Yep.\nskill's part of the\nthe iteration loop or something.\nUh I think right now the skill that I\nhave built is static, but what you're\nmore proposing, if I'm not wrong, it's\nlike uh evolving skill, right? If the\nskill fails, it has to evolve, right? Uh\nI agree with you. I never tried it, but\nI think like it has to be uh like that.\nBecause I'm also more concentrating on\nhow to do it at scale. Uh the reason is\nalso\nuh\nI I want I want the context to be fixed\nbefore retrieval itself.\nNot during operational. So, first when I\nstarted with it, I I started doing with\nwhen operational, which means oh, I have\na work item, I will assign to it, it\nwill fail, then I'll start giving\ncontext and all, but it takes a lot of\ntime. It takes a lot of patience for me.\nSo, rather than doing it, you know what?\nI'm going to fix the context, but before\nretrieval.\nSo, if I can uh\nwhile I was answering your question, if\nyou take a team,\nthe context that you need to fix is very\nsmall.\nSo, you can use a\nuh context gap scanner uh kind of a\nthing.\nAnd maybe if you're good have a good\ndomain expert, I think like couple of\nweeks you can actually fix your\ndocumentation. Not like 100% at least\nlike 60, 70, 80% of a good quality that\nyou can already build it.\nSo, my proposal would always be don't do\nit at an operational level.\nUh at an real-time level, but do it\nbefore retrieval uh itself. That is much\nbetter in this approach.\nYep. Uh yeah, I have another question\nthat is relevant to\nespecially SK. If you have a lot of\ndocumentation and if that documentation\nis sitting in a GitHub repo.\nYou may have situations where you ask\nquestions\nthat may need I don't know, like five or\nsix different docs. So, you will have\nlike\nI'm going there,\nreading all the docs. Like this takes\ntime for\nretrieval of information and then\nit takes a lot of context also for the\nUh\nRight now, after Claude code announced 1\nmillion of tokens in the context window,\nI don't know No, I don't have any\nproblem. So, I calculated it.\nUh at an average, it's like uh 96K\ntokens because I tried with different\ndomains actually. Per domain, I see like\naround 96K tokens uh if I consolidate\neverything like confluence, things and\nall. Uh so, easily it fits in the\ncontext window actually. Uh I've tried\nto do some experimentation around, you\nknow, a graph rag, put them there rather\nthan just take all the files, use a\ngraph rag, understand the intent. But\nfor me,\njust putting the whole context right now\nin the window\ngives you more results than actually\ndoing uh uh\nrag. Unless you have a very big uh\nalmost around a million tokens of a\ncontext that you want to fit in. Maybe\nthen you have to use a bit more\nretrieval mechanisms between it, but\notherwise I think like it should be\nfine.\nUh I have one question. Uh\nI opened your paper, and could you\nexplain this graph like\nuh comparison between different\ntechniques like domain knowledge\nstrategy, knowledge access.\nUh\nUh which one? Yep, sure.\nThis one.\nOkay. So, I also did the citations from\nother papers. Okay. Uh, so, not directly\nrelated, but you have the paper of AS,\nuh, which is also\ndoes a similar thing. So, but AS is not\nexactly into\nUh, how do you say, discovery and\ncuration actually.\nIf I remember correctly,\nuh, maybe I need to refresh my memory.\nmean between\nthe difference between domain knowledge\nand strategy knowledge?\nOkay. So,\nstrategic knowledge, okay. So, what AS\nand all\nare doing is when you are trying to have\na conversation with AI,\nuh, you can see in the cloud code and\nall, it updates its memory. Or like the\nrelationship with your things like that,\nright? So, and also from the chat\nhistory, it understands what is the most\nimportant context I need to remember,\nthose kind of the things. So, when you\nare in communication with it, that\noperational conversations with AI\nimprovement they propose. So, what I\nmy proposal is not based upon your\nconversation with an AI, but rather than\nyour domain knowledge which is\ndocumented actually.\nYeah.\nUh, somebody else has a question.\nSorry.\nWhen you're going to remote knowledge,\nso things on Confluence and that kind of\nstuff, um,\nhow do you ensure that your agent only\npoints to the updated or like the\nthe relevant documentation as\nbecause in your local file system, you\nyou have tags like outdated\nand stuff like that, but remotely that's\nharder to tag, I guess, right? So,\nOkay. So, when I wrote a pipeline for\nextracting from Confluence, it also\nallows actually to give you a date and\nalso last updated who created kind of an\nanalytics on on the space. So, you can\nuse it to actually put a threshold of\nlike, okay, on this particular date,\nwhatever is is old, consider it as an\noutdated one. And let me know. Don't\njust consider it as an outdated one, you\nlet me know because sometimes\nthe document can be stale for so long,\nbut it could be an important document\nactually. So, it lets you know, but not\nlike take decisions actually right now\non this one. So, you decide which one is\nstale and which one is not. You don't\nhave like an intermediate layer where\nyou in the repo you store\nthis is still a data dump. Okay. So,\nwhen it is curating the context, also it\nupdates with a\ndate and also the state of the document\nlike stale, active, and clean. So, it\nalso looks into, okay, this is stale, I\nknow I'm not going to touch it, and I'll\njust go to just look for any other new\nother documents are there in this one.\nDid you think about how to manage access\nor permissions later to this knowledge?\nLike if you have some Mhm. knowledge in\nthe company which is sensitive and only\nand only specific people can get access\nto it.\nCurrently, I guess you just have all the\nknowledge and everything is accessible,\nbut there Okay. So, because it's not a\nproduct or a SaaS solution, it's just\nbasically GitHub.\nFor me, right now, permissions and\nthings are not difficult to implement\nbecause GitHub\nout of the box gives me\nwho I can give the permission to this\nGitHub, who can have right, read access,\nthings and all, who can merge, those\nthings and all. But in case if it\nevolves into a product, and for example,\ncontext gap scanner as a product, and I\nwant you to test it because the reason\nwhy I was using presets for this\nworkshop, not actually asking you to\nupload the files, is because I don't\nwant to take your IP data on this one,\nright? So, unless it becomes a product,\nyou don't have any problem\nGitHub and all, but if you have a SaaS\nsolution for this one, Uh, it's between\nhow the SaaS solution will manage it\nright right now. But the approach has\nnothing to do with\naccess things and all. How you implement\nthose access on on the knowledge is up\nto you.\nYou have a question?\nSo this is of course about\ndocumentation, but\ndid you give any consideration about\nusing it on some like central\ntooling that a company would use? Like\nlet's say that you have a platform team\nand you have a CLI that the different\nteams are using.\nMhm. And so now it's used by different\nagents, right? Okay. And so the agents\ncan also be like, well, this action is\navailable for a resource, but I don't\nwant to do 500 calls just because I have\na list of 500 resources. Okay.\nbe nice if the tool could do that.\nI don't know if if you've given any\nconsideration to this. I I think that is\nhow it has to work in an organization.\nYou need to have a central solution for\nit. But how you want to do the solution\nis up to the organization. For example,\nwe are doing Agile, right? So Agile can\ndo by Scrum, Kanban, or like Lean, or\nsomething. And also you can do different\napps to do it. The The process is the\nsame, but how you do it, which method\nyou will choose, and which app that will\nyou choose in your organization is\ndifferent. In the same way, what we have\ndiscussed is the approach.\nIf you want to put it in the\norganization, you can use the approach\nand you do you can do it in whichever\nway you want.\nMy point is more like, so with this you\ncan identify gaps in your documentation.\nUh-huh. Right?\nCould you use it to identify gaps on\nyour tooling?\nOkay.\nUh, when you say tooling, it's the agent\nInternal tools that I don't know, maybe\na team is building for the rest of the\ncompany. Just infrastructure in general,\nright? Maybe. Yeah.\nUh, can you give me an example of uh,\nlike\nhow it could\nthat uh, I don't know, you build some\nsort of abstraction on top of\nKubernetes. Uh-huh. Okay.\nYou don't want your developers to\nnecessarily know what to do with that.\nAnd then you a different CLI or you have\nsomething, right?\nUm\nbut then like I say, I don't know, maybe\nyou thought that they would lease one of\nyour custom applications or corporate\napplications one by one.\nBut a team has grown into using more of\nthat and suddenly they have a lot and\nthey don't want to do that many calls or\nperhaps even the agent is like, \"Well,\nthis is inefficient. I would like this\ninternal tool to work in a different\nway.\" Okay.\nAnd in that way it would identify like\ngap in the tool or performance\nimprovement, kind of like these tasks\nfor documentation.\nCould be extended, actually.\nSo because\nwe have seen the business processes\nalso, right?\nSo it can also document business\nprocess. The business process is nothing\nbut how\nuh the process in the application it\nactually runs and does things, right? So\nyou can extend it to also find out the\ngaps in the business process or like how\nit works.\nIt could be an extension to it. Okay,\nyeah.\nThank you.\nHow would you ensure that maintenance\nwon't kill you?\nI'm sorry?\nHow would you ensure that maintenance\nwon't kill you?\nBecause the knowledge knowledge for a\ncompany changes with time. Yeah. So if\nthe answer for a question today is B,\ntomorrow could be one.\nOn Friday it could be C, right? Mhm. So\nyou have to see whether you you have to\nidentify B one and B\nand overwrite them. If it's stored as a\ntext, that's a problem.\nOkay. So when I showed the context gap\nscanner, you also saw like an indicator\nof a duplication, right? So if today you\nhave a document, tomorrow you have\nversion 2.0 and something else actually,\nit will find out the same information is\nhaving in three different It It also\nwill find it.\nIf you have only one it is changed, it\nwill take the latest updated one because\nas a human you changed it. So it will\ntake it as a source of truth, right? But\nif you have\nthree versions of the same document,\nthat's a duplication, and it will flag\nit as a duplicate.\nBut, but\nit's a search problem. How how do you\nensure\nperformance? Well, let's say you have a\ndocument of 100,000 words. Uh-huh. And\nyou just change a word, like a\npassword, let's say. It won't be there\nabout for example, right? If you just\nchange Okay. So, you have to find this\nspecific word, compare those few in\nthree documents,\nMhm. find them and replace them, so on.\nHow is it feasible?\nWhich tools would you use to make to\nensure it will it will it will not kill\nyou cost-wise?\nUm I didn't quite get your question,\nactually.\nUh Is it like the token usage you're\nworried about, like uh that many tokens\nthat we used\nHow How is What is the cost saving?\nYeah,\nprecisely.\nIf if if it will grow Mhm. precisely you\nneed, and if there are a lot of changes,\nyou would have to maintain maintain that\nwhole database, right? So, you have Mhm.\nOkay. the structure.\nI don't know many grammar I won't talk\nabout, right?\nUm\nbut my question is, well,\ncuz what you presented is sort of a\nhappy path, where you have a gap, you\nfill it, and then you reuse it.\nBut, in a while, you will have a bigger\nproblem, where you pretend to have that\ngap filled,\nbut actually it doesn't contain\nup-to-date information\nto be done. It contains wrong\ninformation, right? So, you want to\npreserve that.\nUh okay. Okay. Uh\nso,\nit can flag as per when it is created or\nthe last updated. You can set such kind\nof a filters.\nBut, let's say you have a latest\ndocument which has a wrong information.\nRight?\nBut, you don't know that, right? No,\nthat's true. But, as for example, as a\nhuman being, right? So, you go and look\ninto documentation. You told somebody to\nlook into documentation. And the person\nlooked into the documentation and as per\nthe documentation, this is being\nimplemented in this way. The person will\ndo it, right? It's not an agent or a\nhuman issue.\nOkay.\nMhm.\nOkay.\nUh for for example, sorry, you're saying\nthat\nBut I don't think it will cost that much\nas a\nAs I said like\nwhen it tested it, there is none of the\ndomains which cross more than a 100k\ntokens actually.\nSo, I don't think we will for example\nconduct gap scanner, right? You I don't\nthink like you don't have to\ndo it like on a daily basis or anything.\nEven if you run daily basis like 100\ntokens and do one scan. For example,\nright? If you try to start hitting all\nof them,\nall of you, the context gap scanner, I\nthink you can't even burn like one $1. I\nthink so. If I'm not wrong. It already\nhad like Oh, you're ready, okay.\nOkay, I'll I'll cancel the subscription.\nBut I\nMhm.\nspecific domain Mhm. So, the moment you\nscale, you would have to solve this uh\ncost question. Okay. Which is going to\nbe depend also how fast the to to his\npoint, like how fast\nthe the the data change. Mhm. Which I\nthink most of the time not that much.\nSo, the moment you get to like 80% 90%,\nyou just continue performing the task.\nYep.\nUh I see it's different use case by\ndifferent use case. Yep, use case by use\ncase, yep. Yep.\nUh any other questions?\nYeah, I was curious, so I I ran this\ngame maker for like one of these uh\ndomains, Okay. and it has a bunch of\nrecommendations. How do I know that\nthat's enough\nUh\nIt it\nIt actually tries to detail out as much\nas possible. Right now, I haven't\nactually exposed everything what it did\njust for the UI purpose. Uh but the all\nthe per ticket what it actually found,\nlike uh it writes like 100 or like 150\nlines of markdown files and save it\nsomewhere. So, that gives you more\ndetails in case if you want to know\nactually. So, for for the demo, I just\nput the you know, nice UX stuff on on\ntop of it, but you also have a detailed\ninformation at this.\nAnyone else has any questions?\nCan I have another one? Uh yeah, sure,\ngo on. Right, so you Easy one, huh?\nI'll just try.\nYeah, I think you started with the job\nfor clients or that's\n40%.\nYeah.\nAnd if I call the translation right,\nyour claim is\nwith fine lines.\nNot this You said a couple of weeks,\nlike a couple of weeks, and you will\nhave filled that\nNot fill the gaps,\nuh but discover. But if you scope down\nto a team,\nwithin weeks you can do it. Right, cuz I\nmean, if you wouldn't fill it, and you\nwould have to keep asking those\nquestions, then\nYeah, so you do it one time. Uh first of\nall or like multiple times at first, see\nthe whole picture first of all. What is\nthe state of your knowledge base?\nFirst fix it at that level. Then you go\ninto operations, right? You still can\nactually also\nYou you can still continue doing it with\nthe agent with skills.\nYep.\nUh-huh.\nMhm.\nOkay. So, you mean like you can also\ngive all the transcripts rather than\ndoing the cycle, you mean?\nUh that can also be done.\nIf you're only have the\nall the time you have a discussion in\nthe meeting. Everything is documented\nmeeting's transcript itself. But I don't\nthink like this\nsame case for everyone at least uh\nuh for\nI think the amount of time people\nspending in teams\nif you do use the transcripts, actually\nthose are the ones actually who which\nhave more tokens actually. There are so\nmany useless meetings. Uh\nthe transcripts actually\nCould be. Again, it depends upon\ninstitution to institution, right? So,\nare you like more into meetings have\nsolving problems within the\nconversations? And those conversations\nhas a data? Or like your conference or\nthings has a data. If you have it, use\nthe those transcripts as your knowledge\nbase. And at the same time like the\ncompression actually that works\nactually. That that is more useful.\nYeah.\nAnyone else? Any questions? No? All\ngood?\nUh then\nthank you so much\nfor attending this session.",
  "transcript_chars": 56514,
  "ingested_at": "2026-05-12T00:32:31.903983+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 11081,
    "like_count": 206,
    "channel_id": "UCLKPca3kwwd-B59HNr-_lvA",
    "categories": [
      "Science & Technology"
    ],
    "tags": [
      "ai",
      "ai engineer",
      "ai engineering",
      "software development",
      "tech",
      "startups",
      "software architecture",
      "machine learning"
    ]
  }
}