{
  "video_id": "ynJyIKwjonM",
  "channel_slug": "aidotengineer",
  "channel_handle": "aidotengineer",
  "title": "Full Workshop: How Agents ate Search — Leonie Monigatti, Elastic",
  "duration_seconds": 3793.0,
  "url": "https://www.youtube.com/watch?v=ynJyIKwjonM",
  "upload_date": "",
  "transcript": "Everyone ready?\nYes.\nAwesome. Welcome to AI engineer. Thanks\nfor joining my session.\nWe are going to be talking about agentic\nsearch for context engineering today.\nMy name is Leonei. I work at Elastic,\nthe company behind Elastic Search.\nAnd usually I like to talk about\nretrieval on Twitter. Today I'm super\nexcited to be doing this in person. Uh,\na little bit of housekeeping. If you\nwant to access the slides and the code\nwe will be looking at, you can scan the\nQR code.\nSo\nlet's start with why I'm excited about\nsearch and retrieval and hopefully why\nyou are excited about it by the end of\nthis workshop as well.\nWho here has built an agent or some form\nof it before?\nAwesome. Then you've probably then\nyou're probably not intimidated by this\nuh image. You've probably seen some some\nalternative to this one before. This is\nessentially what context engineering\nlooks like. So context engineering when\nwe talk about it is the art or\nengineering techniques about how from\nall of the possible context sources we\nhave, how do we actually decide what\ngoes into the context window so our LLMs\ncan uh generate the best responses.\nOften when we talk about this, we talk\nabout context curation and we mean this\nlittle this little arrow from context\nsources to context window. But we're not\ngiving this little arrow right there\nenough uh credit in my opinion because\nwhat's powering this is the search tool\nor search tools that actually decide\nwhat goes from context sources to the\ncontext window. And today we're going to\nbe looking at the different search tools\nwe have. So this is my personal hot\ntake. I like to say that context\nengineering is about 80% agentic search\nbecause it's this little box right here.\nAll right, let's start with a little bit\nof history. And when I say history, I\nmean the last three years.\nRag. Um when we started with rag the\noriginal idea was that we had a fixed\nretrieval pipeline. So the user message\nwould\nusually more or less ver verbatim be\nused as a search query to be used\nusually as a vector search query to pull\nsome data or chunks uh from a database\nand together with a retrieved context.\nit would the user message would go into\nthe context window and then it would be\nfed to the LLM.\nNice.\nThis has clearly many limitations.\nSo since this is a fixed pipeline\nwhether or not you actually need any\ncontext, you're still retrieving\nadditional information and in the worst\ncase that can actually confuse your LLM.\nRight?\nOn the other hand, if you're only\nretrieving once,\nlet's say you need some multihop\nretrieval, you're asking your LLM\nsomething more complex, then if you're\nonly retrieving once, maybe the\nretrieved chunks um reveal some\ninformation about another search query\nyou need, then you would actually might\nwant to have a second round of search,\nright? So that's why we then moved on to\naentic rag.\nSo we replace the fixed pipeline with\nnow a search tool. So\nnow the agent can decide by himself by\nitself uh whether or not to call the\nsearch tool and retrieve some\ninformation. So we don't have the\nproblem anymore of do I actually need\nany information and when I actually\nretrieve information is this even\nrelevant? Do I need to retrieve more?\nUm,\ndo I actually have to retrieve\nsomething, rewrite the search query? Um,\nyeah.\nSo, we still only have one context\nsource in this case, one database. Now,\nwhen we look at context engineering, the\ncontext lies in many different places,\nright? So, we have context sources in\nlocal files. So when you think about\nyour coding agent, you probably have\nyour um coding project in or code files\nlaying around in your local file system,\nmaybe you're using um some kind of\nworking memory like a scratch pad. So\nyou're planning with your um agent what\nyou want to do. Then you probably have\nsomething like a plan MD file.\nWhen you have agent skills, you also\nhave them usually in a local folder\nsomewhere. We still have databases\nbecause many enterprises have their data\nstored in databases.\nUm, we have the web as another context\nsource. And I know this is a super\ncontroversial uh image here because I\ndid not commit to having the long-term\nmemory in the local file system or the\ndatabase. This is I think a currently a\nvery big discussion. We can get into\nthis in the Q A if you like. But we also\nhave long-term memory as another context\nsource. Right?\nSo how do we actually retrieve context\nfrom these? Usually we have a set of um\nlet's say context source native search\ntools. So for the local files you\nusually have something like um a search\nfiles um tool. For skills you usually\nhave a skill loading tool. When you\nthink about databases,\nwe have a little bit more custom tools.\nSo, um something like a semantic search\ntool. Maybe you also have something more\num general purpose like a tool that lets\nyou execute entire search queries\nagainst a database like SQL for example.\nFor web you have web search tools and\nfor memory you have something like a\ndedicated memory tool.\nIf that's not overwhelming enough, we\nnow also have something called a shell\ntool. Um, Langchain calls it shell tool.\nAnthropic calls it the bash tool. If\nyou've experienced uh if you've played\naround with open cloud, it's called the\nexec tool. But what all of these tools\ndo is they let your agent run commands\nin the terminal. And that actually makes\nthem super uh versatile because now you\ncan let your agent um use CLIs to\nnavigate your um and explore your local\nfiles. So you can just run ls and grab\nto find data in your local file system.\nIf your database has a custom CLI, you\ncan actually let the agent also use the\nshell tool um and interact with the\ndatabase.\nYou could also let your agent write an\nentire search entire script from scratch\nlike connect to your database, run a\nsearch query.\nUm, if your database is exposed via\nHTTPS, you can just run a curl command,\ninteract with the database through the\nshell tool. Speaking of curl commands,\nyou can also um do web searches if you\nlike. So this shell tool is super\nversatile, right? So the question and\nthe topic of today is\nwhat search tool do we actually need? Do\nwe only need a s a shell tool? Do I need\nall of these?\nAnd if you take home only one thing from\ntoday is that doing good search is\nincredibly difficult and that's why we\nhave many different techniques to do\nsearch. Right? We have vector search, we\nhave keyword search, even in vector\nsearch, we have dense embeddings, sparse\nembeddings, multi vector embeddings.\nThen we have many different indexing\ntechniques. So\ndepending on what kind of search\nrequirements and latency requirements\nyou have,\nyou will need to curate your own stack\nof search tools. Right?\nSo today we're going to be looking at a\nfew of these. Unfortunately, we only\nhave one hour, so I cannot show you all\nof them. Um,\nbefore we get into some code, I want to\num give you a few fundamentals um of\nbuilding good search tools because\nagentic search at the surface level\nseems very straightforward.\nThe user makes a request. The agent\ncalls the right tool with the right\nparameters.\nI see someone laughing. Um, the\nretrieval tool gives you the tool\nresponse and then your agent uh responds\nto you with the correct answer.\nAt Elastic, we help a lot of internal\nand external teams build uh agents based\nto interact with elastic search data.\nAnd the reality is that this can break\nin many different ways. I'm just going\nto show you three um today. So the first\nis the agent doesn't call any tool. So\nthis means the agent decides I actually\ncan answer this question based on my\nparametric knowledge. I don't need to\nuse any um context retrieval tool. And\nthe other problem is that the agent\ncalls the wrong tool. I was recently\ntalking to a colleague of mine was\nasking him what was your the most\nchallenging aspect of your project and\nhe was like you won't believe it but it\nwas really difficult to get the agent to\nactually not call the web search tool\nbut call the database search tool\nright\nand then depending on how complex your\nparameters are for your search tools it\ncan also be quite challenging to get\nyour agent to generate the right um\nsearch parameters, right?\nThere's many more failure cases, but\nwe're going to limit this to to these\nthree today. So,\nI personally hate this slide because I\nfeel like everyone in this room probably\nknows um that the tool description is\nthe most important aspect, but anytime I\nsee a tool description, it's like the\nleast effort, one sentence, and then\nyou're wondering why your agent isn't\ncalling the right tool. So, arguably,\nthis is a very long tool description.\nI'm not saying you have to write it like\nthis. I'm just saying if you just start\nwith a core purpose, if it works fine,\ngreat. But if you add more parameters or\nmore tools and your agent is starting to\nstruggle with calling the right uh tool,\nthen maybe add some trigger condition.\nWhen should this tool be used? When\nshould this tool not be used? Especially\nif you have multiple tools. Um, adding\nsomething like relationships is super\nimportant, like first call this agent\nskill before you actually call this tool\nor get some confirmation before you call\nthis tool.\nIf you have the perfect tool description\nand your agent still doesn't call the\nright tool, then reinforce it in the\nagent system prompt. That should\nactually um help out in most cases.\nThen I want to quickly touch on some on\nthe parameter complexity.\nIf you have a search tool that's just\nvery simple in in the sense that\nsomething like get customer by ID, it\nshould be fairly straightforward for the\nagent to generate an ID parameter given\nthat it's a valid ID. Um same for if\nyou're doing a semantic search, right?\ngenerating some valid string should be\nshouldn't cause any any issues. But\nlet's say if you want to have a semantic\nsearch tool and instead of just giving\num a topic, you now also want to give it\nsome filter conditions, maybe you want\nto define the top K. Then you start to\nhave more parameters, right? This isn't\nvery complex here, but the longer the\nlist of parameters you have, the more\ndifficult um it's going to get for the\nagent to generate the right ones.\nAnd I think a very complex one for an\nagent is to when you have something\nthat's more general purpose like letting\nthe agent um execute um entire search\nqueries against a database. So here I\nhave ESQL which is the elastic search\nquery language could be could be SQL as\nwell. So letting the agent write an\nentire SQL query from scratch can be\nquite challenging. Most are pretty good,\nbut um some aren't. So just keep in mind\num let's say the the complexity of the\nparameter also is kind of a failure mode\nand you can you might need to help the\nagent out with um a few of these if the\nthey are more complex.\nGood. Let's look at some code.\nOkay,\nso\nquick show of hands. Who here has built\nsome sort of agentic rag, agentic surge\nbefore?\nOkay, about half. That's good. Um,\nso we're going to be looking at three\nthings. Uh, I'm going to give you a\nquick recap or intro to the very vanilla\nagentic search demo.\nAnd then I'm going to show you how easy\nit is to break this usual demo. And then\nwe're replacing the semantic search tool\nwith something more general purpose. So\nwe're letting the agent write an entire\nsearch query from scratch.\nAnd for these two examples, we will be\nusing um a local elastic search cluster\nas a context source. And then for the\nthird part, I'm switching gears and I'm\ngoing to be showing you how search over\nlocal file system works um with the bash\ntool. And then I'm also going to show\nthe shell tool. And then I'm also going\nto show you um some limitations of the\nshell tool and how you can expand it\nwith custom CLIs.\nAll right. The example that we'll be\ndoing today is I have the conference\nsession data of this conference here.\nAnd let me let me start show you this\none. So just a quick recap.\nWe have elastic search database and\nwe're going to be writing a semantic\nsearch\num search tool. And for in the database\nI have the conference session\num already chunked. You probably already\nknow how to chunk and store data in a\ndatabase. So we're skipping this part.\nThis is not the important aspect.\nSo, what do we need for an agent, an\nLLM? Oh, sorry.\nWe're going to be using lang chain for\nthis um session just because it wraps a\nlot of the complexity and uh I don't\nlike it lets us concentrate on the high\nlevel concepts. also has some nice\nbuilt-in features like uh the shell tool\nis built in and it has some uh code\nsamples for skill loading tools which we\nwill be looking at later.\nOkay, switching back. I'm using GPT 5.4\nNano for this demo.\nThen we're defining a very simple system\nprompt. So the usual you are a search\nagent tasked with answering questions.\nUm you have access to different context\nretrieval tools and before answering a\nquestion oops decide whether or not you\nneed to retrieve additional context\nto help the agent a little bit I have\nsome information about how the data is\num structured in elastic search. So here\nwe have a text field. It's comprised of\nthe title of each session and the\ndescription of each session and the text\nfield is what actually gets embedded um\nas the vector embeddings for semantic\nsearch and then I also have some\nmetadata fields. So for example the day,\nthe time, the room, um the speaker's\nname. So since the metadata is not\nembedded, I can only run um filters over\nthem but not any semantic search just\nfor your information.\nOkay,\nnow the interesting part let's build a\nsemantic search tool. So how that works\nin lang chain is I have to first define\nan embedding model. In this case I'm\nusing the new genome embeddings v5\nmodel.\nUm this is used to embed the search\nqueries at query time and the embedding\nmodel I'm putting in with putting it\ninto my um elastic search store together\nwith the elastic data to create a vector\nstore and then\nI can create a search tool. So in this\ncase I have all I have to do is I call\nthe similarity search method. It takes\nin a search query and in this case I'm\nsetting the limit or the top K to three.\nThis is a little bit of foreshadowing\nbecause I'm limiting the capabilities of\nthis tool to just returning three search\nresults. Right?\nWhat's nice in Langchain as well is that\nwhen you use the tool decorator up here,\nit lets you convert any Python function\ninto a search tool uh sorry into an\nagent tool.\nSo by default, it takes the\num functions u Python functions name as\nthe tool name and the dock string down\nhere is going to uh convert going to get\nconverted into the tool description. You\ncan see I'm breaking my own rule by\nhaving a very short tool description\nhere. Why this works is because I only\nhave one search tool here, right? So you\nwill see I'm adding a few things later\non, but it's not going to get very\ndescriptive in this demo here.\nSo now we can um run a test and test it\nfor a search query of regulatory\nconstraints. And you can see it finds a\ntalk by my friend B on engineering AI\nsystems under server constraints and it\nalso finds some more talks. One by TAS\nand one by Pedro.\nAll right, let's plug it in.\nSo we're plugging in the LM, the system\nprompt, and the search tool. I'm leaving\nout memory. Obviously, this would be\nanother core component of an agent. In\nthis case, I'm leaving it out to keep it\nkind of concise.\nNow I can\nuh run a simple question like which\nsessions discuss regulatory constraints\nin AI systems.\nAnd you can see the agent first uh calls\nmy semantic search tool. It wrote a\nquite extensive\nsearch query in my opinion, but it\nworks. Um, it finds the right talk by\nBili.\nThen it decided that that apparently\nwasn't enough. So, it rewrote the search\nquery. Uh, but decided but got the very\nsimilar search results back. So after\nthat it decided that sorry it decided\nthat it's now able to um respond with\nthe right talks.\nThis is where most agentic search demos\nfail. But this is very brittle. Does\nanyone have an idea how we can break\nthis?\nYes, that's a good idea.\nAnything else?\nAsking it something that's not in the\ndatabase would be something great. What\nabout asking it um something where\nsemantic search actually falls short?\nMaybe something where we want to look\nfor a keyword, a specific keyword.\num also doing something like filtering\nbecause we're in this search tool we\ndon't have any filters implemented right\nso\nmy\nmy choice of search query is which\nsessions should I visit to learn more\nabout GEA\nI'm not even sure like I've heard people\ntalk about Ga I'm not even sure if I'm\npronouncing it correctly sorry that's\nwhy I need to definitely\nattend this session.\nSo what you can see is the agent now\ncalls the search semantic search tool\nand this time it's looking for GPA. So\nfar so good\nbut now you can see it's actually\nreturning a talk for um deep minds Gemma\nmodels. I guess from token from a\ntokenization perspective it could be\nsimilar to GDPA or JPA I don't know then\nit returns something on harness\nengineering not sure if that's\nnecessarily related and then a third one\nI clearly none of these are related to\nGPA\nspoiler alert I know there's a talk\nabout GPA or JPA again I think it's\nright after this one So\nwe can see the search tool we just\ncreated. It's not very useful or at\nleast useful only for a very narrow\nscope of use cases. Right?\nWhat if we let the agent now write an\nentire search query from scratch? Let me\nshow you how we can do this.\nSo\nwe're now replacing the database tool\nthat we had with an execute query tool.\nSo we're letting the agent not only take\nin a like a search like a topic, but\nthis time we're giving it an entire\nthe search tool an entire search query.\nAnd I'm going to show you because this\nis quite difficult for an agent. Uh\nwe're also combining it with a skill\nloading tool.\nSo\ndoing the same thing. I'm setting up my\nLLM.\nYou've probably noticed I'm switching to\na little bit more powerful model here.\nSo I'm switching from the GPT 5.4 Nano\nto um the mini because I am now\nanticipating that writing search queries\nis a little bit more difficult. So the\nnano is probably not powerful enough.\nI'm using the exact same system prompt\nas before.\nAnd now I'm creating a general purpose\ndatabase query tool.\nSince I'm using elastic search, I'm\ngoing to be using the elastic search\nquery language, which is a pipe query\nlanguage for filtering, transforming,\nand analyzing data.\nIt looks something like this. Maybe it\nreminds you of SQL. It's a little bit\ndifferent. It has different\ncapabilities. Not important for this\nsession. Um, but you can see when I\nconnect to my client and then I use this\nquery method from the SQL class and run\nthis query, you can actually see that\nthere is a session by Samuel which talks\na lot about GPA. You can see here is a\nmatch.\nHere's another match.\nSo let's wrap this into a search tool.\nYou can see um this time I just use the\nthe query method again here and the\nagent takes in the ESQL query um as a\nparameter\nand I exchanged the tool description\nwith something that's said that's called\nexecute an ESQL query against the\nconference schedule index in elastic\nsearch.\nnotice anything different about how I\nwrote this search tool versus the other\none.\nThis time I added a try except block\nhere for error handling. Generally\nspeaking, you should have error\nhandling. But since I'm anticipating\nthat writing a good ESQL query or a\nvalid one is gonna cause more problems\nfor the the tool, um I don't want the\nagent to just fail and then the whole\nsystem to crash. So instead of instead I\nreturn the error response to the agent\nso it can kind of self-correct, rewrite\nthe query.\nGenerally speaking, super important to\nhave this, right? So the agent can\nself-correct. So when we we can test\nthis here, this is not important. Uh\nthen I'm plugging in the LLM, the system\nprompt, and my not new search tool into\nthe agent again. And when I now um ask\nit the exact same question as before,\nwhich session should I visit to learn\nmore about GDPA?\nYou can see it calls the execute ESQL\nquery tool and it generates something\nthat looks like valid ESQL.\nI'm not expecting anyone to be very\nfamiliar with ESQL. What's wrong with\nthis is that\nESQL doesn't use the percentage sign as\na wild card character. And in ESQL, you\nwould use the asterisk. So in this case,\nit's actually looking for percentage\nsign, GPA, percentage sign in the data\nas an exact match. So that's why it's\nactually returning zero search results.\nAnd this is when you're working with\nsearch tools also super important to\nthink about. Is returning zero search\nresults\nactually a valid response or is it a\nfailure mode? Right?\nOkay.\nHow could I overcome this? I could\nprobably write\num a more descriptive tool description,\ngive it a little bit more help on how to\nwrite better um parameters.\nI could re reinforce it in the system\nprompt, give it more instructions there.\nOr I could use an agent skill because\nyou need more documentation than just\nlike a oneliner, right?\nSo now I'm going to show you how to add\nan agent skill.\nSo in this case\nI'm going to be writing my own very\nshort custom uh agent skill. Quick\nquestion. Who has you used and played\nwith agent skills before?\nOkay, good amount.\nSo\nI'm going to be writing a very short one\nhere. Um\nthere is official elastic search agent\nskills available if you want to play\naround with it. In this case I'm just uh\nusing my own custom ones. So how that\nworks is you have the um the skill name\nand then also skill description which\ngets injected into the system prompt. So\nonly the uh if you write it in in\nmarkdown it's the I think the front meta\nright that gets injected into the system\nprompt and then when you need it um more\ninformation on the agent skill is loaded\ninto the context window right so it's\ncalled something like progressive\ndisclosure where you kind of add more\ninformation about the skill as you as\nneeded\nso in this case I have some minimal\ninstructions like here's the basic um\nstructure of an ESQL query.\nUm ESQL uses double quotes for string\nliterals.\nJust some very basic syntax rules. And I\nalso added some more information about\nthe wild card pattern so it's not making\nthis mistake again.\nAnd then as I mentioned, Langchen has\nsome boilerplate code you can just copy\nand reuse for um using agent skills. So\nI'm skipping over this. All you have to\nknow is we have a tool and skill loading\ntool and it's get it's getting inject um\nsorry it gets combined with a something\ncalled a skill middleware.\nSkipping over this because this is not\nrelevant for our session.\nAnd now\nall I have to do is\num add it edit the tool description of\nmy general purpose search tool. So this\nis the exact same tool that I had before\nexcept this time I'm now adding some\nrelationship to I'm saying always use\nelastic the elastic search ESQL skill to\ngenerate the ESQL query before using\nthis tool because otherwise if the agent\nthen still uses this tool first without\nwithout um using the agent skill then\nthat would be a shame. Right,\nI'm doing the exact same. So I'm\nreinforcing this now in the system\nprompt. I'm saying the same thing to use\nthe um elastic search a agent skill\nfirst before calling the general purpose\nsearch tool. And now I'm plugging it\ninto the agent again. So this time LLM\nsystem prompt for the skill loading\ntool. I have the skill middleware and\nthen my general purpose ESQL query tool.\nAnd now when I let the um ask when I ask\nthe agent which session should I visit\nto learn more about Ga you can see it f\nfirst loads the skill.\nSo it actually loads everything that's\nkind of in the body part of the skill\ninto my context window.\nAnd then it generates\nthis time a very valid SQL with the\nasterisk as my percentage as my wild\ncard characters. And you can see it\nactually finds the right session.\nAnd now it tells me that at 10:40, so\nafter this session, I should be going to\nthis session to learn more about GPA and\nlearn how to pronounce it correctly.\nOkay,\nwhat's also cool about this is now the\nagent can do a lot of things, right? It\ncan also do aggregations.\nSo if I ask it something like how many\nsessions are on April 8th, you can see\nagain it loads the um elastic search\nESQL\ntool.\nUm and then it writes an ESQL query.\nWhoops.\nWrites an ESQL query that's using a\nfilter. So it's filtering for April 8th.\nAnd then it also does an aggregation,\nsome some counting and tells me today\nthere are 27 sessions.\nThis is nice because\nif I just do a search, let's say I ask\nit to tell me which sessions are on\nApril 8th and it only runs a filtered\nsearch, right? So just imagine it would\ngive me a list of all 27 sessions that\nare today and we let the agent\ncount how many sessions there are.\nThat would probably not be so good\nbecause we all know agents or LMS are\nnotoriously bad at counting things. Um\nand also it would um fill up your\ncontext window, right? So by letting the\nagent do its own calculation, so letting\nlike outsourcing the calculation part\ninto the search tool, it's actually\nquite an efficient way to do this,\nright?\nAny questions so far.\nOkay,\nlet's switch gears. Um,\nthis is a very prominent topic at the\nmoment. Maybe you've heard the\ndiscussion about all an agent needs is a\nshell tool and a file system.\nSo, I work at Elastic, but I don't\ndiscriminate. Let's look at file systems\nand how to do this because I think it's\na very interesting um topic in general.\nSo what I did here um I prepared the\ndata this time in a local file system.\nSo I have a folder called session data\nand in here I have for each type of\nsession like keynotes and workshops. I\nhave another folder and in there there's\nper session one file looks something\nlike this. So with a title, some\nmetadata and the description.\nAnd now um I'm going to show you how you\ncan use the shell tool with this.\nSo I'm switching back to the GPT 5.4\nnano because LM are just generally good\nat um navigating file systems, writing\nuh shell commands. So GPT 5.4 nano is\nsufficient. In this case,\nI define another system prompt. So the\nfirst part of the system prompt is\nuh exactly the same as the one we had\nbefore.\nAnd what I'm replacing this time is\ninstead of explaining how the data is\nstructured in elastic search, I'm\nexplaining how the data is structured in\nmy local file system.\nOkay, so\nlet's use the shell tool. I have to give\nyou a disclaimer. Using the shell tool\ncan be risky since giving your agent\naccess to a terminal\ncan make it delete files or do other\nthings you don't want it to do. So\nalways recommended to uh use it in a\nsandbox environment. Also in langun it\ndoesn't have any safeguards by default.\nSo please be careful when using this.\nBut other than that it's very easy to\nuse. So you can just use uh import the\nshell tool and instantiate it here. And\nhere you can see how you would use it.\nSo it takes in the commands parameter.\nSo when I say echo hello world, you can\nsee down here it actually prints hello\nworld into my terminal.\nAnd then all you have to do again plug\nin the LLM, the system prompt and the\nshell tool. And now you can ask it this\nexact same thing we had earlier. So are\nthere any sessions about GPA and you can\nsay you can see here it's called the\nterminal here but it's the it's the\nagent calls the uh shell tool and it\nactually writes a few commands. So first\nit's looking at the folder structure and\nthen it runs um some grab commands. So\nit's looking for GPA in the session data\nand I think it's looking for the first\n50 entries.\nSo you can see it saw the um the folder\nstructure and then it also found the one\nsession we were talking about earlier.\nBut since I was only looking at the\nfirst 50 and only found one session, it\ndecided that it should probably look at\nthe entire session data. So it finds the\nexact same session again. So this time\nit decides, okay, then I should probably\nlook at the contents of this session.\nSo now it reads the entire file content.\nAnd here you can see the\nthe session information\num as a tool response.\nAnd then at the end the agent tells me\nwhich session I should visit.\nOkay.\nGrab works\nbased on exact matches, right? And um\nreg.\nAnd I just want to show you this because\nI think it's funny how like surprisingly\ngood agents are with bash because they\nkind of can cheat at semantic search. I\nLet me show you this.\nSo when I ask it which sessions discuss\nhandling regulatory constraints, this\nwas our semantic search query from the\nbeginning. You can see it again looks at\nthe folder structure and then it the\nfirst command or the first search it\ndoes it's looking for regulate.\nIt's fair. It's looking for regulation\nfor regulatory.\nI guess that's that's a fair start. But\nthen it goes ahead and now it just\nchains a bunch of synonyms together. So\nit's looking for compliance. It's\nlooking for constraints. It's looking\nfor GDPR. It's looking for governance.\nYeah, I guess that's fair.\nUm, I think it actually finds the it\nfinds a bunch of sessions. So, it's\nlike, okay, let me try a bunch of other\nsynonyms. So, now it's looking again for\nuh regulate, compliance, GDPR, PR,\nsoenity.\nUm, I think yeah, the list goes on. It's\njust looking at a bunch of different\nsynonyms and it actually is successful\nwith this and it finds the session by\nbuilding and returns the session\ninformation and then um is able to\nrespond correctly.\nI guess it works. Is that the most\nefficient way to do this?\nProbably not. I mean just as an example.\nSo, let's say you you want to search for\nsomething like movies with animal\nsuperheroes or something. Do you really\nwant to do your agent to search for a\nlist of all the animals possible? Will\nyou find all the superhero movies with\nanimal superheroes?\nProbably not. So,\nit works.\nIs it the best? I let you decide.\nSo at the moment there's many different\nsemantic search alternatives to grab. Um\nI think there's one by llama index\ncalled sam tools. There's a really cool\none by light on which is called coal\ngrap based on multi vector embeddings.\nAlso there's one by our own Gina that's\ncalled Gina Grap. Um, today I'm going to\nshow you how easy it is to actually um,\nuse this together with your agent.\nSo, all you have to do is go ahead and\ninstall the Gina CLI\nand then\nall all you have to do is tell your\nagent that it now has access to this\ntool or to the CLI.\nSo, this is the exact same system prompt\nas I had earlier. Now with the\ndifference that I'm explaining to it um\nthat it has gen\nworks how it should use it.\nHere are some examples of how you would\nuse Gina Grap.\nJust a disclaimer Gina Grap has many\ndifferent modes. You can use it for um\nclassification. You can use it for\nre-ranking. Today I'm just showing you\nhow to use it for semantic search.\nAnd at the end I'm also explaining to\nthe agent when it should use grap and\nwhen it should use gina grabs. Just so\nit knows for exact matches you probably\nstill want to use grap. And for um more\nsemantic search or fuzzy queries use um\ngina grap.\nAnd then plugging this in to my agent\nagain.\nAnd when I now run the exact same\nsemantic search query we had earlier. So\nwhich oops\nwhich sessions discuss handling\nregulatory constraints. You can see the\nsame behavior. So it calls the terminal\ntool. It first explores the folder\nstructure and then it actually on the\nfirst try is able to correctly use gina\ngrab. So, it's looking for regulatory\nconstraints\nand boom, it actually finds um the\nsession by Bill on the first first try.\nFinds a few others because it's looking\nfor 10. Um it says returned top k of 10\nand then it's able to answer me\ncorrectly. Nice.\nAll right. Any questions so far?\nGood.\nThen I'm switching back.\nSo we were looking at a bunch of\ndifferent tools today. We saw how big\nthe tool landscape is. Um I showed you a\nfew of the um search tools we have. Now,\nsome practical recommendations on when\nshould you actually use what.\nSo,\nmaybe let's start with this. If you're\nlooking for just one silver bullet tool,\nthat's probably not the right way to go.\nAgain, if you're if you think about it,\ndoing good search is incredibly\ndifficult. So ideally you want to have\nor curate the right set of search tools\nfor your agents search behaviors\nand you want to have a combination of\nspecialized tools and a combination and\nuh general purpose tools. So specialized\ntools are something that the agent can\nuse out of the box. Something with a\nvery simple parameter,\nsomething where you're not where you\ndon't need a very powerful LLM. You\nknow, the agent isn't going to make a\nlot of mistakes. The agent can just use\nthis tool out of the box. So at Elastic,\nwe like to think about this about of\nhaving a low floor. So this is a concept\nfrom user experience where the agent can\njust\nyou use a tool doesn't make many\nmistakes. It's also efficient so it\ndoesn't have to run your tool multiple\ntimes. You can think about this as the\nsemantic search tool we had earlier.\nMaybe you need to look up customers by\nID a lot of the time. Then having a\nspecialized tool for that exact\noperation would be helpful.\nBut then you also want to give the agent\na high ceiling. That means\nfor unexpected queries, for complex\nquestions, you want the agent to still\nbe able to handle these questions,\nright? And not be have like these\nlimited uh specialized tools and be\nlike, I I cannot solve this. So for this\ncase, something like a shell tool or the\nvery um general purpose one we had\nearlier of a query execution tool would\nbe very helpful.\nBut the problem with the query execution\ntool or the shell tool you saw earlier\nis since it's so general purpose, the\nagent sometimes might need more\niteration to iterations to actually get\nto the right answer. Right?\nSo this is my my practical\nrecommendation of having a balanced set\nof search tools um of a low floor and\nhigh ceiling.\nThis is all nice when you already know\nyour agents behavior,\nbut if you don't know your agents query\nbehavior yet, then I would recommend to\nstart with a general purpose tool.\nThen log your agent's behavior.\nGenerally speaking, logging your agents\nbehavior recommended. Um, but if you\nnotice, maybe your agent is taking\nfour or five tool calls per question.\nThat's too many tool calls. Then you\nprobably that's probably an indicator\nthat the tool your agent has is too\ndifficult for it to use. then definitely\nlook at what the agent's actually trying\nto solve, maybe scope out something more\nspecialized in that case, right?\nAlso, if you notice specific um query\nbehaviors, this is what I personally did\nwith my with my test open claw. I was uh\nit has the exec tool and I um started\nlogging its behavior and obviously I was\nplaying around with databases. So after\nthree days I was asking it what kind of\ninteresting patterns do you see and was\nrecommending me actually to um implement\nsome specific uh search tools uh to\ninteract with the database because it\nwas out of the box only using the the\nexec tool.\nAll right. Start with general purpose\ntools if you don't know your users\nbehavior yet. Log breaks and app\npurposeuilt interfaces.\nYeah, that's that was a lot to take in.\nUm,\nI'm sure you have lots of questions, so\nI'm opening it up for Q A. And\notherwise, on your way out, don't forget\nto grab yourself some stickers. And then\nthank you for joining my session.\nI think there's a mic coming.\nThank you. Um, so would you say the tool\nstack you need is also mostly dependent\non the model you're willing to use. So\nif you were using a very good model, it\nmight be fine using the shell tool or\nthe search tool, but a very small model\nand light agent might need more\nspecialized tools or\nYeah, actually we I think in our\ninternal testing we noticed that a more\npowerful tool actually reduces the error\nrate of um for the parameters by I don't\nknow the numbers exactly but it was a\nvery big amount where it reduced the\nerror rate. So having a stronger model\ndefinitely helps\num for the general purpose tools but I\nthink you cannot expect uh just because\nyou have a very strong model that\nthere's going to be no no errors if that\nmakes sense. Yeah.\nThanks for a nice talk. Um I have one\nquestion. Maybe it's slightly off topic\nbut um so now we are talking about a\ngentic rack but it comes with the\ndrawback of having higher latency\nagainst typical rack. So would you\nrecommend of having like a second\npathway for simple rack for fast answers\nand how would you like\nguide the agent to actually choose the\nright one because I think it's hard to\nsay which question should be answered by\na gent or by simple rag.\nThat's a good question.\nI I don't think I have a have a good\nanswer on like on the top of my head\nright now. I'm thinking I was maybe\nsomething related. I was asked recently\nif you have a rag system, should you\nreplace it with a gentic rag? And I\nguess this is kind of going in the same\ndirection of when do you actually need a\ngentic rag, right?\num\nprobably for a lot of use cases. I know\nrag has been killed many times, but I\nthink the reality is that rag is still\nvery effective for many use cases. Um\nhow would you actually switch between\nrag and agentic rag? I'm not sure\nbecause I assume it's again it needs\nsome kind of almost agentic logic of\nswitching between them. So\nI'm not sure. I'm sorry.\nIn such cases uh like um when you have\nuh the wild card in the GAPA\nuh example uh why can't we just um\nperform a hybrid tool that maybe search\nand replaces uh common wrong uh wild\ncard symbols coming from SQL with the\ncorrect ones for instance.\nI'm not sure if I understand your\nquestion correctly. I mean um\nthere are cases in which uh the agent\ndoes doesn't know the how to write the\nuh correct query because maybe he thinks\nuh the placeholders uh coming from SQL\num apply to ESQL\nbut uh so why don't we perform a hybrid\ntool that um determine significantly um\nuh search and replaces the wrong um\nplaceholder\nthe the percentage symbol with the\nasterisk.\nYeah, actually so the the example I\nshowed you of using the agent skill\nwasn't necessarily the the necessary\nsolution for it. You can also um add\njust some very simple instructions on\nfor ESQL don't use um the percentage\nsign as a wild card character. It\nactually works. I tried it when I was\nbuilding the the demo. Um but then when\nthe agent now runs in the next issue,\nthen you start adding the next piece of\ndocumentation. then you can kind of\nstart writing the entire ESQL\ndocumentmentation from scratch into your\nsystem prompt. And yes, for the demo\npurposes is it would have worked, but\nit's probably not how you would do it\nnecessarily when you're building\nsomething more robust, right? Because if\nyou just add like little like band-aids\nevery time you run into an error, then\nwhat happens when you run into the next\nedge case? Does that make sense? Yeah.\nHi, thank you for the really wonderful\num presentation. I have a question. Uh\nin the demo we have walk through the\nagentic search with DB curious tool and\nalso another one with shell tool. Would\nyou recommend in the practical use we\ncan also kind of uh use combine both\ntool and then we validate the result\nfrom each of tool and then we kind of\nadd the confidence of the result from\nthe LM. Would you like recommend doing\nthis in a practical use?\nYes. Yes. That's a great question. Also\nagain I'm kind of cheating in this demo\nright because I'm only showing you one\ntool per demo. In reality, you would\nhave something more like a bunch of\ndifferent tools where you then have to\num decide which or the agent has to\ndecide which tool to use. I think there\nwas a very interesting blog post by\nVersel I believe and they did an\nexperiment. I think it's called, if you\nwant to look it up, I think it's called\ntesting is if bash is all you need is\nthe title I think of the blog post. And\nthey actually kind of benchmarked or\ntested an agent with a bash tool, an\nagent with a just file search tools, I\nbelieve, and an agent with database\ntools. And in the end, they also had one\nagent with a bash tool and the database\ntool. and they noticed that was super\ninteresting.\nUh for a specific set of queries um\nwhere you have analytical queries,\nthis is a specific use case. Actually,\nthe database tool was more effective,\nbut on the other hand, the file search\ntool is very effective as you saw for\njust quickly finding things. But the\nvery interesting aspect was the hybrid\nagent with the bash tool and the um\ndatabase tool was actually achieving the\nhighest um like highest accuracy because\nit first I believe it was first using\nthe database tool and then verifying the\nresults with the shell with the shell\ntool and that led the agent to actually\nachieve better accuracy. So I think that\nwas a very interesting way um and\nbehavior to see in in agents.\nThanks for sharing.\nUm one second question. Um, so if we use\nthe semantic search tool, I think in\npractice you probably would use some\nkind of threshold to cut the results to\nnot get something if there's no answer.\nBut in the agentic regime, would you\nthen say, okay, let's put a conservative\nthreshold such that we don't confuse our\nagent or would you say the agent is\nsmart enough even if we retrieve results\nthat are not really relevant, it will be\ngood enough to to to notice that?\nYeah, that's a great question actually.\nUm, so in the examples you probably saw\nsome where the agent was returning. I\nthink in the last genograph example, you\nsee it's actually returning the top K\nresults where only the first one is the\nactual relevant one. And I think um\nbecause the agent does a little bit of\nreasoning over whether the search\nresults are relevant to the search\nquery, I think it's much better or\nthey're much better today at kind of\nweeding out what's not relevant. But\nthen you kind of run into the risk if\nyou have longer running conversations\nthat kind of these search results sit in\nyour context window longterm could have\nthe problem of confusing your agent long\nterm. So I think it kind of\nit depends on your use case of how\num your like how your agent can handle\nlike irrelevant search results.\nGenerally speaking, based on search\nresults, it can filter out what's\nirrelevant.\nUh thank you for the talk. Amazing. Um\nare you utilizing sub agents for this\nsearch queries? Because yeah, if we let\nthem decide to is it relevant to use a\nquestion like yeah it's done in reggas\nframework for example for evaluation.\nI mean sub agents would help a lot. Do\nyou have any experience with them?\nUnfortunately not. I have not played\naround with sub aents yet. I can only\ntell you that I know for example I\nbelieve in cloud code they're using sub\naents for doing specific search tasks. I\nthink there was a blog post on how\nthey're actually using a sub agent to um\nanswer specific questions about claw\ncode because it's kind of like a niche\nquestion a user would ask. So in this\ncase they kind of outsourced um their\nexpertise to a sub agent. So having a\nsub agent for specific niche questions I\nthink would be interesting but I don't\nhave too much like experience and\nOkay. Thanks. Uh can I ask another\nquestion?\nSure.\nUm\ndamn I forget. Sorry\nI try to catch up later.\nUm yeah, I kindly off topic but um you\ntalked about skills and the big benefit\nof skills is just have the description\nin the uh system prompt and whenever\nneeded we need to load this the full\nskill. Um do you have any recommendation\nwhen and how to clear the system prompt\nagain? So because we want to keep the\ncontext window small and maybe for a\nlong um session we might have up to 10\nskills full skills in the context and\nyeah\nI'm not sure Joe do you have a better\nanswer like have an idea.\nSo what we're doing is that we learn the\nskills like\nbut then we\nto like offload things. So when the\ncontext\nvery small\nbut\num but yeah so like the way that we are\ndoing it behind the scenes is that we're\nwe're we're providing like that kind of\nprogressive disclosure of skills. So\nwe're providing those um the skill names\nand descriptions the location within the\nfile store and then from the file store\nwe're loading into the context window\nwhen we need that skill and then we\noffload it once it's once it the prog\nyou know the context window progresses\nahead of time around that. So we have\nthis kind of more on demand one around\nthat. And that's the same with our like\ncompaction like comp of context and\nthat's what I would advise you to do\nlike some of the questions do try and\nuse the file store as much as you can\nand have those tools as well like being\nable to grap the file store for when you\nwant to see previous tool results and\nthen and then use it from that.\nThanks as my colleague Joe from Elastic\nas well.\nAwesome. If there are no more question,\nI will let you guys go into the coffee\nbreak. Again, don't forget to grab\nyourself some stickers and happy to\ncatch up in the halls if anyone's\ninterested. Thanks so much.",
  "transcript_chars": 44568,
  "ingested_at": "2026-05-12T00:31:29.295816+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 13445,
    "like_count": 401,
    "channel_id": "UCLKPca3kwwd-B59HNr-_lvA",
    "categories": [
      "Science & Technology"
    ],
    "tags": [
      "ai",
      "ai engineer",
      "ai engineering",
      "software development",
      "tech",
      "startups",
      "software architecture",
      "machine learning"
    ]
  }
}