{
  "video_id": "C_GG5g38vLU",
  "channel_slug": "aidotengineer",
  "channel_handle": "AI Engineer",
  "title": "Harnesses in AI: A Deep Dive — Tejas Kumar, IBM",
  "duration_seconds": 1226,
  "url": "https://www.youtube.com/watch?v=C_GG5g38vLU",
  "upload_date": "20260517",
  "transcript": "[music]\nHello everybody.\nEverybody's head turned up. Hello, hi.\nHow was lunch? Was it good?\nYou didn't like it, no?\nIt's like\nBritish food. Anyway, hi. I'm Tejas. I'm\nI'll be your first speaker this\nafternoon. Tejas, that's pronounced like\nTejas.\nDon't worry, I'm not\nHopefully my my joy in AI is and I've\nhad the privilege of working at a number\nof different places over my career in\none form or the other. It's just been an\nabsolute joy to learn from the best.\nToday, I'm a AI developer advocate at\nIBM\nwhere we we do things with AI, believe\nor not. We train frontier models, we\nbuild harnesses. It's really it's a fun\nlab to work in.\nBut that's not what I'm here to talk to\nyou about today. Today, I'm here to talk\nto you about AI harnesses. AI harnesses.\nBefore I move forward, I would love to\njust have a show of hands.\nHow many of you are like confident in\nyour understanding of AI harnesses? Like\nyou're like I could present this on\nstage today.\nLook around. Look around. No, seriously,\nlook around. That's why we're doing this\ntalk. Okay, this is my hope. I want you\nto If I ask you this at the end of the\ntalk, right? I want you to be like, oh,\nI I I get it now. That's the whole\npoint. I have literally nothing to gain\nfrom this other than I I shared\nknowledge, okay? Because also this term\nis kind of everywhere. You may have\nheard it used like 52,000 times today.\nAnd it means different things to\ndifferent people cuz like in the machine\nlearning world, it means like a\nglorified test suite for machine\nlearning models. But in the AI in the AI\nworld, it means something different. And\nso today, we're going to understand this\nin detail. It's a deep dive, but it's 18\nminutes long. So let's let's move\nforward. Um\nI want to start by talking about why\nharness. Like why do we use harnesses?\nAnd the reason for this is because we\npay rent to companies that give us\ncompute, give us inference, give us\ntokens in return. Some of you maybe work\nfor companies that have frontier models\nlike Anthropic or Google or whatever and\nyou maybe What was the term? Token\nbillionaires, yeah?\nUm\nI'm not that. I am maybe with Watson\nmodels, but but the vast majority of us\naren't token billionaires. We we pay\nrent. We literally $20 a month for\nClaude Pro. And then you get a context\nwindow that's limited and you get like,\nyou know, you you don't get the full\nhog, so to speak. And the model you rent\nis is a black box. Like they could at\nany time, I'm not saying they do, but\nthey could if Opus is somehow not\navailable, they could serve you Sonnet\neven though it says Opus. You would\nnever know, right?\nAnd so it's just a big There's too many\nvariables that we cannot control. So why\nharness? Because the name of the game\nwith harness is reliability. Um\nI really hope I'm not supposed to stand\nin front of this white line and then I'm\njust not in the camera. Anyway,\nwhatever.\nIt's reliability. It's it's making sure\nthat the agents we build do what they\ndo, period. Irrespective of the black\nbox model, irrespective of the of the\nthe thing we rent and so on, okay? Now\nthat we understand why harness, let's\ntalk about what a harness even is from\nfirst principles. Like let's let's take\nit all the way back to harnesses that we\nknow and understand. If you've ever, you\nknow, climbed a mountain or something or\nyou've seen someone This is a harness.\nIt's like mountain climbers literally\nwill like harness themselves to what? To\na mountain because it's stable. And they\ncan't go off the rails, literally.\nThey they\nanchor themselves in something stable so\nthat they can't drift too far.\nOkay, that's that's what a harness is by\ndesign. When you have any dog owners\nhere? You have dogs? You you walk your\ndog on a harness, okay? That's why?\nBecause your dog doesn't go and bankrupt\nyou with tokens. Okay?\nThat's what a harness is.\nBut the problem is if we think about\nwhat harness, there's really two types.\nThere's one from the machine learning\nworld, which as I mentioned is kind of\nlike a test suite and a test runner.\nYou would give a model some inputs and\nyou see the quality of the outputs.\nThat's not This is not ML engineer\nEurope. We're going to talk today about\nthe agent harness that is common in AI\nengineering. Okay, so what what is an\nagent harness?\nAn agent harness and and this is kind of\nthe money shot here. The agent harness\nis I'm not making money off this. It's\njust an expression. The agent harness is\neverything around the model that gives\nit grounding in reality. It's literally\nthe thing that ties it to a stable\nenvironment, okay? An agent So Claude\ncode, for example, can be considered an\nagent harness. And some of you would\nsay, oh, no, it's a coding agent.\nAbsolutely, it's a coding agent. But\nit's a harnessed coding agent. An agent\nharness has more or less the same\ntypical suspects, moving parts. Number\none, it's got\na tool registry. Almost like so Claude\ncode, cursor, codex, they have tools to\nread from the file system, to write, to\nexecute bash commands, right? They have\na tool registry. They have a model and\nsome of them allow you to choose a\nmodel, some of them allow you to not.\nThey have a model. They have primitives\nfor managing context. Almost every\nharnessed agent runtime today will\ncompact its own context, right? That's\nthat's that's the job of the harness.\nGuardrails are another part of a\nharness. For example, max steps. Anyone\nusing max steps? Do not do more than\nfive tool calls. That's a guardrail. And\nso if if you do that, you just kill the\nkill the run, right?\nAn agent loop is another part of an\nagent harness, which is crazy. This is\nwhat some people I've spoken to\npreparing this talk\nwill say,\nwait, isn't a harness just the agent\nloop? No, it's the stuff around the\nagent loop. In fact, it could be a loop\naround your agent loop. It could be an\nNM loop. And we'll look at that a little\nbit in some code. And then finally,\nthere's a verify step. This is, for\nexample, in a coding coding agent, after\nthe work is done, a verify step would\nbe, hey, let's let's run lint, let's run\ntests, let's make sure nothing broke,\nright? So almost every I'll use code\ncoding agents as an example, but you\ncould have a harness for anything. And\nit's it's amazing cuz it really grounds\nblack box models in a stable environment\nthat you control, okay? I'd like to show\nyou\na demo. And what we're going to do\ntogether is we're going to build a\nharness, a bare bone baby's first\nharness. Let's call it a poor man's AI\nharness together so we understand from\nfirst principles how this works. We're\ngoing to build a computer use agent that\nhas a job. The job is go to Hacker News\nand upvote the first post, okay? It's a\ncomputer It's a browser use agent. We're\ngoing to use a really bad model\nintentionally. We're using GPT-3.5\nTurbo, which is like 2023, right? But\nwe're going to harness it so that it can\nactually do the job. And we're going to\nsave money. So let's I've spoken too\nmuch. Let's just get into the demo.\nUh\nAnd and so let's Welcome to my project.\nThis is my project. Hello everybody.\nThis is the entry point. Can you see\nthat? Is it too\nYeah? You want it bigger?\nLet's do bigger. Okay. So this is not\nActually, this room is too bright. Let's\ndo light mode. It's I It's not my\nnature, but sometimes. That's better,\nyeah? Okay. So we have we have a model\nand we're\ntrying an old LG Sorry.\nWe\nWe shouldn't have seen that. No, we'll\nwe'll try an old model. And this is the\nprompt. This is the This is the task.\nThis is literally my prompt. Upvote a\nstory I just described it. For the\npurpose of this demo, we will not change\nthe prompt at all.\nBecause a lot of us think, hey, my agent\nis not doing what it's supposed to do,\nso I just need to prompt it harder,\nright? That's not always true. I need to\nchange the system prompt. We're not\ngoing to touch any prompts here. We're\njust going to build a harness and the\noutcome will change.\nWe we log some things to the console and\nthen we start a browser session. Okay,\nwhat's a browser session? It's literally\njust Playwright. Not Playwright MCP,\nlike Playwright Playwright. Where this\nis just a class I made with an open\nmethod that launches Chromium and gets a\ncontext and makes a page. And then\nnavigate We're just literally calling\nthe Playwright functions, yeah? This is\nThis is just traditional engineering. So\nwe create a session, we open the\nsession, meaning a browser window in a\ncontext. And then we create our tools\nand we give that browser session to the\ntools. And we create a context and we\ngive the task, meaning the prompt here,\nto the context. Now, create tools is\nliterally what it sounds like. It's\nhere. There's just some types and create\ntools is a function that takes a browser\nsession and gives you like tools. And\nthese tools are not I didn't invent\nthis. This is from OpenAI's SDK, okay?\nSo you have the name, the description\nparameters, and execute, the way you\nactually call the tool in your runtime.\nAnd and there's just tools for I made\nthis. It's very easy. Um\nSo that's my tools. And then create\ncontext, you may think, whoa, context\nengineering. Absolutely not. It's This\nis my context. There's nothing here.\nIt's just a system prompt.\nLiterally, the most basic system prompt\nand the user's task. This is basic\nbasic. And then we have run loop, which\nis just running the agent in a loop. So\nwhat it's doing here, we can actually\njust look at this, too.\nWhile true, so it is an agent loop. And\nwe get a response from the agent and we\nsee if the response says stop, meaning\nif the LLM says, I'm done, then we\nreturn the value.\nIf we get any other response, we don't\ndo anything except\nadd these events into a trace. So we\njust push history into a big list of\nhistory. Does that make sense? And so\nthat's all we're doing here. This is\njust a loop where we just collect events\nuntil we're done, okay? So this is super\nbasic. Now let's see how it works. So\nI'm going to come over here and I'm\ngoing to do Are you okay, sir? Do you\nneed water? I'm going to\nnpm run agent. Um And so it's going to\nopen Chromium. It's going to Okay,\nHacker News, so far so good. Click\nupvote. Oh, no. So we we hit a login\nscreen and then it kind of panicked and\ncrashed. But look, it it lies. You see\nthis?\nUm this is a problem. And so what's the\nsolution? Prompt it harder? No. Change\nthe system prompt. Always login with\nthese credentials included in the system\nprompt. No.\nSo how do we then solve this? And look,\nwe because of my logging, we can\nactually see it just clicks the upvote\nbutton and then considers it a success.\nIt doesn't verify. This is the job of a\nharness, okay? So now incrementally,\nwe're going to slowly start building a\nharness. Um\nAnd so, let's just move I'm not going to\nwrite code here. I'm not going to live\ncode because we don't write code\nanymore. We inspect diffs.\nRight? Anyone write code by hand? You\ndon't Maybe actually you do belong here.\nAnyway, so um\nI'm kidding. So, this is um\nThis is the first change we're going to\nmake. This was our index file.\nAnd we have this run loop that I showed\nyou, but now we're going to add one\nthing to it, which is default\nguardrails. We're going to create some\nguardrails, okay?\nUm what do our guardrails look like?\nWell, let's go and look at it in the\neditor um with guardrails over here. And\nso, we have some types, but these are\nour guardrails. We have two. Max\niterations, meaning if you do more than\nsix steps, I'mma kill you.\nAnd max messages, meaning if you have\nmore than this many messages, I will\ncompress the context. These are just\nguardrails, okay? A little utility to\ncombine them, and we just we can compose\nthem here. We could do like as many as\nwe want. So, now\nlet's go back to our changes. That's the\nguardrails. We if we go back to the\nagent loop we actually use the\nguardrails here in this diff. And so, we\ninclude the guardrail functions, and we\ncan see that here what we're doing is\nwe're checking how many messages have we\naccumulated, and we just like trim the\ncontext if it's too much. Um but what I\ndid want to show you is here at the end\num we we push context size, which is\nsome more metadata about what we've done\nwith our guardrails, okay?\nUm our context compressor is extremely\nbasic and extremely naive. This is what\nit does. Um let me actually open this\nwith syntax highlighting to spare Um\nthis is what it does. So, what we're\ndoing is if we always keep the system\nprompt and the user prompt and the most\nrecent two messages. So, if the\nguardrail is triggered, we always remove\neverything after the system prompt and\nthe user prompt in the middle, and we\nkeep the last two messages. This is\nsuper naive. Don't do There's better\nways, but this is where babies first.\nWe're We're getting there.\nSo, we we're starting to have a harness,\nbut it's not called a harness, but this\nis really like\na pregnant harness. Like it's almost\nborn, okay? And so, what we're going to\ndo is let's just call it a harness now.\nSo, I'm going to show you another diff\nwhere we\nHere, check this out. Index, we've\ndeleted almost everything.\nUm and we've moved it into this file\ncalled harness. Let's go look at our\nentry point now.\nIt index, it's it's all gone. So, the\nprompt is there. But this is it's like\n19 lines of code, and we just have run\nharness. We've taken all the logic from\nhere and hidden it in a function called\nrun harness. And as you would expect,\nrun harness does exactly the same thing\nas we did in the index, okay? Nothing\nnew is here except maybe like a print\nfunction, which is just console log. Is\nthis clear so far? Yeah, we just moved\nstuff. Now that we have something called\na harness, we can actually use it.\nAnd let's solve the problem of lying\nfirst before we solve the problem of\nlogging in as me. Yeah, because it says\nI I upvoted, it did not. I want to know.\nSo, what we're going to do is we're\ngoing to add some guardrails and and\nhave it tell the truth. Like if you\nfailed, tell me the truth. Um\nhow might we do that? Well, we'll check\nit out here. So,\nmany many things changed. Um\nOr not. I don't know. Hang on a second.\nYeah, okay.\nDid Many many things changed. So, we run\nharness and we added a third argument\nhere, which is a verify step and max\nattempts. Max attempts goes to our\nguardrail. So, if if you took more than\nthree tries to do this, just give up.\nAnd if we go to the harness, we added a\nlot of things um that are just manual\ncode. This is not different prompt. This\nis my logic. Um the main logic is run\nharness no longer wraps over the code we\nmoved, but we moved that to a different\nfunction called\nrun harness attempt. So, if we if we\ncome to run harness Let's go here. I\nneed to check the branch out, sorry.\nYeah. So, now if we go to run harness\nattempt, we'll collapse this. I'll\ncollapse this.\nI'll collapse all of these. And if we go\nto run harness attempt, now this is the\nsame thing from our index. We just moved\nit into a function called run harness\nattempt because our main run harness is\njust a loop that runs no more than three\ntimes, okay? Is this clear? So, we're\njust enforcing the max steps, but at the\nharness level for safety. Um\nthen we have run harness attempt that\ncalls it. We have this function called\nverify successful upvote. I wrote this.\nThis is deterministic. That's what I\nwant to show you. What does this do?\nWell, we see if You remember we were\ntracing in the agent loop, we're just\nadding history events. So, we reflect on\nthat, and we see if there was a browser\nclick\non the upvote and if it's successful,\nbut really successful, then we say true.\nBut there's a huge butt here, which is\nwe have now cases for failed login. If\nthere's a tool named harness auto login,\nand if the message starts with failed\nthen we return early and we say no no,\nthis failed. We're We're removing the\nlie, okay? Similarly, unrecovered login\nredirect. We look over our agent loops\ntools that we've been pushing into.\nUm and if we see that the harness auto\nlogin didn't run\nand now we're on the page that is the\nlogin URL, then again, we just fail.\nOkay?\nUm and so, we're what we're doing is\nwe're just adding like if this happened,\nif this happened, you just just fail.\nReturn early. Is this clear? This is\nwhat a harness does. And so, let's run\nthis now with the harness.\nUh npm run agent.\nAnd\nnow it's going to go on Hacker News, and\nwe're going to repeat the same cycle.\nOkay, it's going to come here, and now\nit's still failed, but look, it stopped\nlying because our harness checks the\ntool history and actually sees what\nhappened. This is what a harness is\nsupposed to do. Great. This is already\nlike half the battle won because step\none to solving a problem is admitting\nyou have one, okay?\nTest-driven development vibes. So, now\nthat we we're failing correctly, we can\nsucceed. And I'd like to show you that\nin the last diff, and then we'll finish\nthe talk here. So, number four.\nUm we have a whole new function. It's\ncalled login handler. Uh I'll add some\nsyntax highlighting here so you don't go\nblind. Uh but here, create login\nhandler. This is This is all it does. It\nruns every agent loop just before we\npush to the traces, and it This is what\nit do It checks the browser session's\ncurrent URL.\nAnd if we're not on a login page, it\njust says cool, I don't I return I have\nnothing for you. This computationally is\nnot costly at all, right? If you're not\non the login page, but if you are on the\nlogin page\nthen it we fill in\na temporary This can be an environment\nvariable. It can be secure, you get the\nidea. But we fill in credentials and\nsubmit the button programmatically from\nthe harness, not from the agent,\ndeterministically and securely because\nthis file has access to any secrets I\nwant it to, right? And so, this How How\nis this called? Well, this is called in\nthe agent loop. So, if we go back to our\nagent loop and notice we were pushing\ntraces, yeah? This is where we push the\ntraces.\nJust before, if we have a login handler\nwe call the login handler just before\nthis in the agent loop. What does the\nlogin handler do? Well, if we're not on\nthe login page, it does nothing.\nIf we are on a login page, then it\nquickly will inject credentials and\nsubmit the form and then take you back.\nIt will also add, as we can see here, it\npushes a message into the queue saying,\n\"Hey, I'm the harness. I logged in.\nYou're good now.\" Is this clear? Yeah?\nSo, the the harness is is is literally\nharnessing the agent to something\nstable, something deterministic. That's\nwhat it's for, okay?\nLet's run this now and see what happens.\nSo, npm run agent.\nIt's going to open Hacker News, and when\nit gets to the login, now that harness\nstep, it logged in and it upvoted the\nfirst one, and it closed.\nAmazing. So, successfully upvoted a\nlittle snitch for nilux, uh rank two, uh\nsucceeded after six iterations, and I\ncan click this and go into Hacker News\nand actually see indeed it was upvoted,\num and it I can unvote now, which means\nit was upvoted, right? So, um the agent\nused the computer, logged in as me with\nmy harness that I just made here on\nstage. That's the purpose. Is this clear\nso far? Do you understand the role of a\nharness? Look at you nodding. This is\nmusic to my ears. Fantastic.\nSomething to my eyes. I don't know the\nIt's beauty to my eyes, kind of weird.\nWe don't have a expression for that.\nLet's land the plane. I'm done. I think\nmy work here is done. What does this\nlook like in practice? Why Why do I care\nso much about harnesses? Because they\nrun the world. Models are\nnon-deterministic. And you want to do\nmore with less. You want to use a cheap\nmodel. Use like Quinn or something, or\neven something smaller. Use GPT-OSS.\nIt's free. And with a great harness, you\ncan go very far. That's why. At IBM, we\ncreate a open-source project that we\ndeploy in the enterprise that allows\nvery large companies, huge companies, in\ntheir private like data-sensitive areas\nto perform rag operations on all kinds\nof things, teams, calls, and PDFs, and\ninvoices. Um We We build It's called\nopen rag, and it's it's rag I don't know\nif rag is cool or not anymore, but\nopen rag has a hell of a harness that\nprovides enterprise-level security to\nlike asking questions with internal very\nvery siloed data. And And that's kind of\nwhere the harness engineering comes in.\nSo, let's summarize. We covered a lot of\ncontent. Was it a deep I think it was a\ndeep dive. It was a deep dive in like 18\nminutes or so. Um\nwe went pretty far. I\nIt's not It should not be lost on you\nthat I did not touch the prompt once.\nI did not change the system prompt. We\njust built a harness\nand the outcome radically changed. And\nof course, we can add secrets, we can\nadd tokens. Um yeah, we did a lot. In\nthe end, I hope you understand what a\nharness is, the value it can present,\nand how you can use it. What's next? Um\nLook, I\nI don't have a crystal ball like\neveryone else here, um but it's not lost\non me that 2025 was the year of agents.\nYes? Uh 2026\nis the year of harnesses, I'm pretty\nsure. Everybody How many times is this\nword used here? Um I think\nI would hope I think it'd be pretty cool\nif 2027 was the year of dynamic\non-the-fly generated harnesses. How cool\nwould that Like you tell an agent, \"Hey,\ndo this for me. Buy me a flight ticket.\"\nWhatever it may be. And then before\ndoing the work, the agent creates a\nharness. This similar to plan mode. Any\nof you using plan mode? But But on\nsteroids. The The agent creates an\nactual harness, self-aware. It knows,\n\"Oh, I can maybe hallucinate here. I can\nmaybe\" Creates a harness, does the job,\nand returns back to you, guardrailed and\neverything. That is so cool. Dynamic\non-the-fly harnesses. I would I I this\nis honestly the next logical step\ntowards AGI, and I would love to see it.\nI don't know if this is just me being uh\nyou know\nweird guy with ideas, but um I think\nthat's kind of the direction. So, with\nthat um I'm almost out of time. I would\nbe really remiss if I didn't spend the\nlast like 30 seconds saying thank you so\nmuch. The slides are on GitHub uh as uh\nam I uh and so I'd love to chat more.\nThank you.\n[music]",
  "transcript_chars": 22023,
  "ingested_at": "2026-05-21T19:20:38.663732+00:00",
  "source": "retry-no-transcript",
  "yt_meta": {
    "view_count": 52006,
    "like_count": 1831,
    "channel_id": "UCLKPca3kwwd-B59HNr-_lvA"
  }
}