{
  "video_id": "Xel_mF5CCAo",
  "channel_slug": "deeplearningai",
  "channel_handle": "deeplearningai",
  "title": "AI Dev 25 x NYC | Scott Yak: Building MCP Servers That Make Agents More Effective",
  "duration_seconds": 1637.0,
  "url": "https://www.youtube.com/watch?v=Xel_mF5CCAo",
  "upload_date": "",
  "transcript": "Good afternoon everyone.\nIf there's one message I would like you\nto take away from this talk is that eval\ncan become a source of joy.\nYeah. If you consolidate your agents\ntools into an MCP server. Okay. I'll\nexplain more.\nUh I'm Scott Yak from Data Do. What does\nData do? Right. So, data dog is an\nobservability platform. So, you send us\nmetrics, traces, logs, and this nice\ntelemetry data. We provide the tools to\nhelp you figure out whether your service\nis running as expected, whether there's\nan outage and how do you fix it. So, we\nhave a nice website. You can look at the\nmetrics, check on your service, nice,\nthere's a nice dashboard to get an\noverall sense of how your website is\ndoing, some traffic information. We have\nsome really fancy tools to visualize how\nyour service is running. So this is all\nvery nice but it's a bit intimidating\nfor newcomers and we don't just want you\nto know uh to see what's going on with\nyour service. We want you to be able to\nact with your service. So the dog builds\na bunch of agents and we have learned a\ncouple of lessons along the way. What we\nlearned is that in the process from\ngoing from demo quality to production\nready agents, you when you want to\nimprove the scores so that you can\nactually accomplish your tasks, you have\nto deal with a whole bunch of failure\nmodes. Things like hallucination,\noutput formatting, to call failures and\nso on and so forth, right? And the\nanswer that everyone is going to tell\nyou is that you need evals, right? And I\nagree with that. You definitely need\nevals,\nbut they're also a real pain, right? So\nwill I like to help with that, right?\nJust evals are still going to be\npainful, but at least we can take away\nsome part of it. Tool call failures and\nthat's where MCP servers can help.\nNow what are MCP servers? So without MCB\nservers, what you would have to do is\nthat for every single agent that you\nbuild, you have to uh you typically have\nto build the tools to connect to the\nback end. And what usually happens is\nthat because each agent is doing\ndifferent things, they need different\ntools. They call the tools differently\nand they expect different things from\nthe tools. The tools fail, the tool\ncalls fail in different ways. And that\nmeans that each agent team have their\nown tool uh tool called failure evals\nand that leads to a lot of duplication\neffort.\nWhat MCP servers allow you to do is that\nyou can consolidate all your tools into\none server. And so these tools can serve\na whole bunch of different agent teams.\nUh [snorts] but you could always have\ndone that, right? But what's so special\nabout MCP servers? Well, the nice thing\nis that um MCP servers can also be\nremote. So not only are they just a\nbuilding block towards or stepping stone\ntowards better agents, they also talk\ndirectly to customers. So the your ids\nlike cursor can use your tools directly.\nThird party agents like cloud code can\nuse your tools directly and to build an\nagentic experience directly from the\ncustomer. So what this means is that\nyour tools are no longer just a stepping\nstone. It's self a product. And when you\nhave a product, you want to come up with\na good product experience.\nAnd what we want to get out of this uh\nproduct is that for our users when they\nconnect to our MCP server, the agents\nbecome data. Power users.\nSo to give you a quick sense of what our\nMCP what it's like to connect to MCP\nserver. So here's the tab, the agent tab\nfrom cursor, right? So you might ask a\nquestion like do you have any are there\nany HTTP errors in service MCP blah blah\nblah okay then cursor is going to talk\nto LM give you some u some feedback and\nthen it makes a tool call to our search\nlocks tool it chooses search locks then\nit gives the search locks tool gives\nback the results and then it summarizes\nwhat it found yes I found some HTTP\nerrors great right it is acting like a\npower user as a user you didn't really\nneed to know data do search lock syntax\nit's Good. Um, so let's see what\nactually is going on in there. So we\nhave a better sense of what our MCP\nserver actually needs to do.\nSo here what just happened. You have the\nuser, you have the agent here which is a\ncloud code agent panel. Uh you have the\nMCP server, our data do uh MCP server\nand that's our back end that serves uh\nthe search locks tool and in the back.\nSo the agent has a really important job\nwhich is to manage the context window.\nIt starts with the system prompt and it\nmakes a call to MCP server to list tools\nso that it can fill its context window\nwith the tool descriptions and that way\nit knows what tools are available and\nhow to call them. Then it receives a\nrequest from the user and it adds that\nto the s to the context window. It then\nmakes a it then calls LLM to decide uh\nto see what to do next. it may decide to\ncall a tool and because of the\ninformation from tool descriptions and\nthen it makes a call to the MCP server.\nThe MCP server takes that request\ndecides what to do with it. Maybe call a\nparticular back end, maybe cause that\nother back end. The back end returns the\nresults and the MCP server does some\nother uh business logic, maybe it does\nsome filtering, maybe it does some\npageionation, maybe it postprocesses\nsome JSON into CSV, whatnot, right? Or\nerror handling. and before it passes\nback the response to the agent.\nSo now the MCP server uh so MCP server\ngives back the response to the agent.\nThe agent loads it back to the context\nwindow. Now it's part of the context\nwindow when it decides what to do next.\nIt may have it might decide that it\nneeds to do something uh it needs to go\nback and ask more questions. So it keeps\nlooping or looping until it's complete\nand then when [snorts] it's complete it\ngives you back a result. Yes, I found\nsomething and returns it back to the\nuser. So that is a whole that's the flow\nof the agentic system or most basic\nagent loop.\nSo I often get this question what\nexactly is there to do in MCB server?\nDon't you just wrap around the API and\nyou prompt the tools and the prompt the\ntool descriptions all that stuff. So yes\nthe tool descriptions is a is an\nimportant part right it tells you it\ntells the agent how to call tools and\nwhat tools are available. But besides\nthat uh it's also important to note that\nthe tool call response also gets fed\ninto the agent context window and that\naffects what it will call what we will\ndo next. And so in the MCP server we\ncontrol the MCP we control the MCP\nserver and we because we are in the same\ncompany we also we also can talk to the\npeople who work on the back end to\ninfluence what comes back out of the to\ncall response. So that means that we\nalso want to optimize the tool\nimplementations.\nSo and this is a massive search base,\nright? It's not just the text in the\ntool description. It is the code base of\nyour entire back end as well, including\nyour MCB server. And how do you optimize\nthat? How do you know what to\nprioritize? For that, you need EDOS,\nright? And the thing is that um it's a\nbit so eels for MCB servers are\ndifferent because there's a lot we don't\ncontrol. When you're agent you control\nalmost everything. When you are an MCB\nserver you control almost nothing. And\nthat feeds into our eval philosophy.\nWe don't optimize for any particular\nagent. We are agent agnostic and we only\ncheck the final result. So we don't care\nabout which tool it calls. Okay. So I'll\ngo into more detail about why we choose\nthis as our philosophy for eatals.\nSo why agent agnostic right and the\nreason for that is really because of\nhelplessness. We don't really know\nenough about the agent which agent needs\nto use us. So we cannot actually\noptimize for any agent. Um a different\nway to look at it is that we want MCB\nserver to be ergonomic enough so that\nany agent can use it well not just the\nsmart ones. The nice side effect of that\nis that you actually can, you know,\nprioritize simple agents. Um, and what\nthat means is that our evals, we don't\nneed to use the expensive ones. We use\nthe cheaper ones. We use the faster\nones. And that means that our evals are\nfast and cheap to run. That's nice. Nice\nbonus.\nSo why tool agnostic though? So what\noften happens when people make a tool\ncalled evals for agents is that they'll\nwhen they when agent reaches a\nparticular step they will say okay make\nsure you call this tool and make sure\nyou call this tool this way and I think\nthere's a better approach than that and\nthe reason is that agents are creative\nlike there are often a few ways to solve\nthe same problem for example if you want\nto check which service went out of\nmemory there are probably three\ndifferent three tools in data dot that\ncan give you that answer. But if you\nwere to force a tool to say, \"Okay, you\nhave to call search locks then and the\nagent goes ahead and try something\nelse.\" Then you're going to the EDOS are\ngoing to fail uh like needlessly and you\nhave too much stuff to add into your\nEDOS. It becomes brutal.\nAnd the other thing is that agents are\nsurprisingly resourceful. So one thing\nthat I've accidentally uh noticed when I\nwas using cursor with our MCP server is\nthat sometimes if you once I misspelled\na service name it actually uh zoomed\nout. So it added some wild card tags. It\nremoves certain filters. So it\neffectively is zooming out and then it\nsees okay what service names are in your\nare in your logs in the first place and\nit finds one that is actually similar\nenough to the service name that you had\nand then it zooms into that and says\nokay this is we didn't find the exact\nthing you want but this is probably what\nyou want like if you are in some you\nknow very tight agentic loop with very\num rigid eels you're probably going to\nmark that as a failure but if you are a\nif you're a human trying to do something\nmessing things up once in a That's\nactually a nice experience. So we don't\nwant to erase that by overly rigid eels.\nAnd the other thing is that because we\nare the team that develops the MCP\nserver, we want to keep upgrading our\ntools. If we if we make it too hard to\nchange our tools, then when we want to\ndo big changes, big improvements like\nconsolidating five tools to three tools,\nlike switching to a different query\nsyntax or adding one tool that just\nmakes all tools better. If you have\noverly rigid tool um evals, that is\ngoing to prevent you from making those\nchanges in a way that you can see the\neval scores before and after, right?\nBecause once you change those tool once\nyou change your tool surface you have to\nrewrite your evals and you don't get an\napples to apples comparison anymore\nright your tool specific evals become\ntoo brittle it becomes change detectors\nand you don't want that you want to keep\nimproving your MCP server so that it\njust when you connect to it the whole\nexperience is just better you don't have\nthe user shouldn't have to think about\nthe individual tools\nand the effect of that is that eval runs\nare fast right we from the time you\nchange your code and you run the eval\nrun and you see it in the dashboard\ntakes only two minutes. We run it\nlocally. It's nice when you see it on\ndashboard it looks like this. uh we have\nit instrumented with data. LLM\nobservability and so you can see at a\nglance right which uh which two calls\nfailed which two calls succeeded and\nthen you can zoom into that one of those\nscenarios and say okay like show me the\nspans of of the of the tool calls the LM\ncalls and all that that happened and\nthen you this each of these things have\na trace ID which you can then take a\ntake the URL copy it put into your group\nyour group Slack and see hey this thing\nfailed why did it fail can you help me\ntake a\nIt's a very nice debugging experience.\nSo it makes running evals actually fun.\nLike we change something, you rerun the\nevals, two minutes later you see stuff,\nyou can share with your team. You can\nyou try something new that makes\nsomething that didn't work before work.\nYou share you can now uh take this trace\nID, share with your team.\nOur evals now become integrated into the\ndev cycle of our MCP server. And now\ndevs want to write EDS, right? If they\nare trying to do something with an MCB\nserver that is challenging its existing\ncapabilities and the evals don't\nactually capture that new cap new\npossibility they want to write an eval\nso that they can see hey before this\nfailed now this passed I did something\ngreat\nnow how does this help agent builders so\nthe nice side effect for other people is\nthat tool call failures are now handled\nby a central team so the agent builders\nthey don't have to handle tool call\nfailures with their big end to and eval\nruns right so to call failures are\nhandled by this lightweight process\nthat's done by a small central team this\nmeans less work for them\nand another good side effect is that\nlet's say you have a few agent teams and\none of the agent teams give you some\nfeedback that hey this tool doesn't\nreally work very well maybe your you\ndidn't describe it properly then so the\nfeedback from your agent team can lead\nto MCP server improvements from your\ndevelopers and that leads to\nimprovements to all your agents because\nNow they are using the same MCP server\nwith the improved tooling.\nThat's very uh motivating for people who\nare building MCB servers.\nBut so it was nice to run the MCB server\nbut how do you make the EVAL scenarios\nto run in the first place? Right?\nRunning EVAL scenarios was never really\nthe hard part. The hard part is making\nthe eval scenarios. Right? I just told\nyou how to do the easy stuff but not how\nto do the hard stuff. Right?\nSo let's go through step by step how do\nyou actually make a single eval scenario\nfor MCB server.\nSo first we think okay if you are a\ndata.p power user you should know how to\nsearch for logs you should understand\nlog search syntax right. So you look at\nthe documentation page you see okay we\nneed to understand how to put tags\nservice tags basic things. Okay, so show\nme recent logs in a service name full.\nSimple question, reasonable request,\nright? Can you make that an EDL\nscenario? Um, so the the unfortunate\nthing here is that if you call this\nthing, um, your actual answer is a whole\nblob of JSON, right? And if you want to\nput that in if you put that into your\nevals, it's going to be all variables.\nYou don't want that. Actually, you can\njust ask it for counts. Uh, because\nyou're trying to test whether it\nunderstands the lock search syntax,\nright? If it knows how to make a count\nand it gets back a specific number,\nchances are it actually knows how to\ncall that. It actually understands that\nsyntax. Just changing from counts to\nlist is pretty trivial.\nSo can we make this the case? Uh not yet\nbecause the word recent is ambiguous. So\nwe need to make this more specific,\nright? Recent means before, it could\nmean five minutes before, right? It's\nnot like a different LM call might\ndecide that it means different things\nand you get a different answer. So this\nneeds to be made unambiguous, right? Is\nthis okay now? Uh still not yet because\nlast hour depends on what time you're\ncalling it. So now you need to fix the\ntime. There's this little thing called\ntime travel uh which you can pass in a\ntime stamp into your server and the\nserver pretends what time it is now. And\nnow finally this is actually\nunambiguous. you can as a human or you\ncan help you you can get a script to\nhelp you generate this get this answer\nuh put this in the eval scenario and\nthen no matter when you run it you're\ngoing to get back the same answer as\nlong as the two calls are correct right\nand it also doesn't really matter if the\nif your MCB server uh if your LLM makes\na different call to a different tool or\nyou change your tool surface and you got\nand you just get back the same answer\nit's fine it's still going to pass\nso how many evals else do we need to\nmake? So let's go back to our our\ndocumentation, right? So we have wild\ncards, we have two wild cards, we have\nfull text, we have free tax, we have all\nthis special u we have all this special\nsyntax, we have all these edge cases\nthat we need to deal with, right? So we\nneed maybe hundreds of eval scenarios\njust to cover such logs. And then we\nhave other products too. Not only do we\nhave logs, we also have metrics, we have\ntraces, right?\nAnd then not only do you want to make a\nquery on one thing, you want to make\nqueries across different products. And\nhow do you do that? How many evals do\nyou need now? Maybe thousands, right?\nAnd that's going to be several days of\nnon-stop ground work just to create eval\nscenarios.\nAnd what's worse is that if you work for\na company that has data retention limits\nfor your logs, then every time the data\ndisappears, you have to repeat this\nwhole process all over again. Because\nonce the data disappears, your new data\ncomes in, then last hour and your now\ntime needs to change and the answers are\nall going to change. So you need to do\nthis all over again. So is there a\nbetter way?\nThe good news is that you can actually\ngenerate those emails. You don't have to\ndo this by hand. Uh it's a bit\nunintuitive that this is actually\npossible. So I'll walk you through that.\nSo if you start with a question, a\nnatural language question and you try to\nget the answer, this is not really easy,\nright? If you are trying to if you're\njust using a simple agent that tries to\nmake a tool call, this is not easy.\nThat's why we're trying to make agents\nin the first place. But if you start\nfrom a CQ, you kind of know what the\nanswer should look like and then you\nturn that from the CQ to the answer.\nThat's easy. you just pass that query\nexactly into your API call and then you\ncan make a script that just gets that\nanswer\nand then to go from a CQ query to the\nquestion uh it's not so straightforward\nbut what you can do is that you can use\na parser parse this thing convert it\ninto a and then from a you traverse the\nthe tree for each node you map it to a\ntext template and that will lead you to\na natural language string that preserves\nthe semantic meaning of that CQ query\nSo this is this is does not this does\nnot involve an LLM. This is a\nstraightforward um just you can write\nPython to do this thing for you.\nHow do you get a seat query in the first\nplace? Right? We just made the problem\nuh we just pass the problem to another\nplace. So here is where the LLM or the\ncoding agent comes in. The coding agent\nwith the help of a product documentation\nthat Yeah.\nHello anyone? Okay.\nAll right. So, with the pro with the\nhelp of your um your coding agent and\nyour product documentation and some\nelbow grease because your coding agent\nis going to look at your is going to\nlook at the placeholder service names.\nWhat's going on? It's going to look at\nthe placeholder service names and just\nfill in say API, but it doesn't exist,\nright? You work for this company. You\nknow what services actually exist. So\nyou have to fill in say from service API\nto service full right. So you still have\nto do that but you can generate a whole\nbunch of seed queries at once\nand with those sequeries\nrepeat that for each process for for\neach question to for each CQ query to\nget a question answer pair and then zip\nthem up to get labeled eval scenarios.\nSo these labelled eval scenarios you can\nput them all into a Python list and you\ncan all run them locally.\nSo we just went from one docs page to\n200 something eval scenarios. That's\npretty nice.\nBut we now have a new problem.\nToo many evals. Too many eval scenarios,\nright? If you just so the basic the\nfirst thing to do that you want to you\nprobably would do is that you print them\nto your console log because we're\nrunning it locally. But then when you\nhave 200 eval scenarios and you know\nbefore your change you got the you got\n0.7 as a score as average score after\nthe change you got 0.71 as your eval\nscore. So, was this a no-up or was it\nbecause something improved and something\ngot worse? At a glance, it's not easy to\nsee. But what we can do then is uh going\nback to the previous example, you can\nuse your LLM to also help you add some\ntags because when you use that process,\nyou were referring to the product\ndocumentation and a product\ndocumentation probably has some sections\nheaders that tell you what capability it\nrefers to. So you can add the capability\ntag. Then you can pass these tags along\nto your instrumentation to part into\nyour stance that you pass to LM\nobservability or your favorite\nobservability tool. Then you can group\nthem by capability. So at a glance you\ncan see what capabilities you're doing\nwell at and what capabilities you're not\ndoing so well at. So here we can see\nthat count to C has the lowest uh\naverage metric score. And so if we\nshould probably do something about that.\nSo as a as a human you look at this\nthing and you think okay I probably need\nto prompt the tune the prompts for my uh\nfor my lock search syntax so that I can\nso that I know how to deal with the C.\nMaybe I didn't think about that when I\nwas writing the tool. Or\nyou can ask cloud code to look at that\nand help you optimize it.\nBecause by\ninstrumenting this thing and putting it\ninto LM observability, LM observability\nbeing a data do product is also can be\nexposed through our MCP server. So now\nyou can ask a coding agent like cloud\ncode to point to it and to ask some\ninformation. So let's start with just\nrunning the evaluation, right? We got\nthe score of like 0.04, a low score. We\nwant to improve that.\nSo then we can ask the coding agent to\nanalyze the edction result with the help\nof the MCP tool for like LM observably\ntool in MCP server.\nSo it analyze this thing and now that\nthis analysis is in the context window\nof the coding agent you can point the\ncoding agent at the piece of code that\nyou want to optimize. So in the next uh\nslide I you're looking at the code at a\ndiff like the red part is is looking at\nthe tool description for our MCP server\nuh for search data.logs box and the red\npart is the one that is is what is going\nto delete and the green part is what is\ngoing to improve. It's going to add you\ncan see that it is trying to um it's\ntrying to deal with the with this IP\naddresses now\nso you can optimize the code and then\nrerun restart restart the server and\nthen rerun the evaluation and get a\nhigher score. So you can repeat this\nprocess again and that's the\nselfoptimization loop. So to recap what\nI just showed you. So we generated 200\nsomething label eval scenarios. You can\nrun evals locally in two minutes. You\ncan visualize results on the dashboard.\nYou can analyze the failure patterns on\na dashboard or you can do it with the\nhelp of a coding agent. And you can\npoint the code at the place where you\nwant to optimize\nwhether we the business logic be the\nprompt the tool description or somewhere\ndeeper in the back end to optimize the\ncode based on what we just put into the\ncontext window.\nSo as a recap,\nthere's a huge search space that we can\noptimize by being as MCP server\ndevelopers.\nAnd with all this automatically\ngenerated evals, they can become a\nsource of joy not just for uh MCP server\ndevelopers, but also for all the agent\nbuilders that built on top of our MCP\nserver. But the first thing you have to\ndo is to consolidate your agents tools\ninto an MCP server.\nThank you.\n[applause]\nThank you, Scott. Uh, we have time for\ntwo questions.\nUh, very nice presentation. Thank you\nvery much. Uh I'm wondering if you are\nexperimenting also in fixing the issues\nthat data do finds in a similar manner\nusing MCP server then feeding that to\ncloud code and then fixing the issue\nautomatically.\nWait, can you can you repeat what are\nyou feeding into the thing?\nUh basically how you are fixing the\nevils.\nYou can also use MCP server for data dog\nto see the issues that data do found and\nthen automatically fix by using MCP\nserver and cloud code. Are you\nexperimenting with this or\nwe I definitely plan to uh yeah this is\na very some of this stuff is b coded uh\nit happened in a pretty short amount of\ntime\nokay yeah we are looking forward we are\nusers and it's just amazing tool thanks\nthank you\nwe've got time for one more if anyone\nhas a question\nyeah okay\nyou mentioned um at a point where you\nneed the back end to return different\ntype of response so that agent can\nhandle them better.\nOkay.\nCan you be more specific?\nAh yes. So the thing is that the our\ntool surface is not the same as our API\nsurface, right? When you have an API,\npeople expect it to be stable, right?\nPeople build code on it. They don't\nchange it very much. Every time you\nchange API, you break anyone who hasn't\nlike updated their their binaries. But\nwith MCP, you can change your tool\ndescription uh every time because\nthey're going to have to call the list\nservices every time they restart their\nthey initialize uh and restart their\nagent. So you can actually you it's more\nforgiving. You can change your tool\nservice more often. So what we can do is\nthat we can we can actually have a\ndifferent API. We can have a different\num interface um compared to our API. So\nfor example, what we can do is that we\ncan do some spelling correction. We can\ndo some checking before it reaches the\nAPI call. We can perhaps if we think\nthat this is a clustering query instead\nof a list query, we can actually just\ncall the the clustering API instead of\ncalling the list API.",
  "transcript_chars": 25186,
  "ingested_at": "2026-05-15T10:45:19.936431+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 848,
    "like_count": 10,
    "channel_id": "UCcIXc5mJsHVYTZR1maL5l9w",
    "categories": [
      "Education"
    ],
    "tags": []
  }
}