{
  "video_id": "mR-WAvEPRwE",
  "channel_slug": "aidotengineer",
  "channel_handle": "AI Engineer",
  "title": "Build Agents That Run for Hours (Without Losing the Plot) — Ash Prabaker & Andrew Wilson, Anthropic",
  "duration_seconds": 4540,
  "url": "https://www.youtube.com/watch?v=mR-WAvEPRwE",
  "upload_date": "20260518",
  "transcript": "Nice meeting you guys. Um I'm Ash. Uh\nthis is Andrew. We both work in uh as\nengineers in our applied AI uh team here\nat Anthropic. Um\nand the kind of topic for this session\nwas uh inspired by a blog post we put\nout uh\njust a couple weeks ago actually\nabout how to think about building\nuh agents that can actually run for\nreally long extended periods of time.\nYou know, we're talking 5 6 hour plus\nkind of runs.\nUh I think we've all seen these kind of\ndemos, you know, of like companies being\nlike, \"Hey, we've like one-shotted a\nbrowser.\" For example, but not\nnecessarily sharing like some of the\ndetails into what goes into the harness\nand that's what we kind of want to talk\nabout today. So, the first off um my\namazing quick Andrew will talk about a\nlittle bit about basically how we've got\nhere, some of the primitives that we've\nshipped in code code um and, you know,\nwhere we are today. Um and then I'll hop\nback on stage to talk a little bit about\nsome of the more experimental stuff that\nwe're playing with with harnesses um as\nwell as, you know, a few examples of of\nwhat we've seen. But, over to you.\nSounds good. Thank you, Ash. And yeah,\nthanks everyone for joining uh first\nsession of the AI Engineer conference.\nSo, glad you're spending it with us. Uh\nmy name's Andrew. I'm on the applied AI\nteam based out of London working as a\nsolution architect with a lot of our\ndigital native and industries customers.\nSo, um yeah, I'm going to give a little\nbit of a history tour uh trip down\nmemory lane, but really with the focus\non all the things that we've shipped\nthat lead to agents being able to run uh\nfor multiple hours or even days at a\ntime. Um and then I'll hand over to Ash\nto do more of the the state of the art.\nRight. Okay, so um\nlittle quote from or on Twitter from\nBoris, the creator of Cloud Code. This\nwas on the one-year anniversary of Cloud\nCode. Uh, basically saying a year ago,\nCloud was struggling just to write bash\ncommands and escaping strings. Um, and\nit could run for, you know, maybe 20\nminutes at a time. And then, we're now\nat the point where almost all of Cloud\nCode is being written by Cloud Code, and\nit can run effectively for days at a\ntime. Uh, so sort of a a big big swing\nover just the course of a year, and I'll\nwalk through that history uh, a little\nbit now. But, just to Let me uh, zoom in\nhere. Um, just to sort of frame the\nproblem. I'll play why why is it that\nit's really difficult for these agents\nto run for extended periods of time? Um,\nI think broadly there's three big\nbuckets. Uh, some are more intuitive\nthan others. So, firstly, context. I\nthink we all understand context windows\nvery much finite. So, you start a new\nsession, there's like amnesia. The agent\nhas to start from scratch, so you need\nsome sort of memory components. Um, also\nas you're working through a context\nwindow, there's this notion of context\nrot. So, uh, there's less coherence as\nyou're you're getting deeper into that\nsession. Uh, also, you might get to the\npoint where uh, the model actually\nexhibits what's called context sense\nanxiety. So, it gets kind of nervous as\nit reaches the end of its context\nwindow, and it just quickly hurries up\nto finish what it's doing. Um, this kind\nof leads into planning. So, uh, in\ngeneral, models are not that great at\nplanning just out of the box. Uh, they\nmight try and do everything in just one\nshot. Or, for example, they might build\nhalf a feature and then stop, or they\nmight just run out of context altogether\nand sort of leave a half-finished app\nbuilt. Um, but then maybe less\nintuitively, um, models are really bad\nat judging their own output. So, I know\nwe all know that models can be\nsycophantic and sort of tell you what\nyou want to hear, but this applies as\nwell to to coding tasks. So, it might\nlook at a feature and see that it's sort\nof half half-baked or a little bit\nimplemented and say, \"Yeah, okay, uh,\nthat looks done.\" and then it'll move on\nto the next thing. Or it might build a\nfeature like a button, but actually the\nback end, you know, it doesn't exist for\nit. There's sort of no nothing behind\nthat, but it looks like the feature is\ndone. So, um I know Ash will talk quite\nextensively of so some of the new\ntechniques we have to to help with this\num specifically so models can become\nbetter at judging their own output.\nSo, there's there's two ways really we\ncan we can fix these things. Uh the\nfirst one is obviously the model. So, um\nbaking it all into the model weights\nthemselves. And I'm sure you've all seen\nthis this meter chart. It's basically\nhow long can an agent run for with a\nminimal scaffold uh where it's\ncompleting 50% of the tasks. And you'll\nsee from Opus 3.7, it's around 1 hour\nand up to Opus 4.6, 1 year later, it's\nat 12 hours. So, an entire day. Um and\nwe've of course, you know, managed to\nget that running much longer. Other\npeople have as well, but this is just a\nsort of a very minimal scaffold.\nThe second thing that you can do is, of\ncourse, make changes to the harness\nitself. So, this is the scaffolding um\naround the model. And we have the agent\nSDK which ships with all of the\nprimitives that we've been building over\ntime. So, there's the core agent loop\nitself where you have Claude model\nthat's determining what to do, what\ntools to run, uh maybe it's pulling in\nsome tools from MCP servers. Uh it might\ndelegate some tasks to a sub agent. It's\nbringing in all the context from things\nlike claude.md or the skills that are\nloaded or slash commands. And there's a\nwhole permission system. And the this\nthis will change over time as well as\nthe models get better and improve. But\nthese are sort of the the core\nprimitives that we're working with. And\nthen of course, you use this framework\nto to build your own harness for\nwhatever it is you're trying to do such\nas some of the things that Ash will show\nuh later on when we're getting to more\nlong-running agents.\nUh I think what's also interesting is\njust looking back at the last year of\nreleases is that when we've released a\nmodel we've always also released a lot\nof harness changes alongside the models.\nSo really these things are like\nco-evolving together. So we'll just look\nback um\nsuppose firstly just prehistory um\nbeyond you know one year ago. I think we\nall remember that that period where\nClaude had the artifact section of\nClaude.ai and and\nuh Sonnet 3.5 was the first model that\nreally showed promise when it came to\ncoding. And it could now verify that it\ncould look at what it had built and sort\nof iterate from there. That was quite an\naha moment sort of pre-Claude code. Uh\nbut then also we shipped computer use so\nit could start clicking around taking\nscreenshots um testing its own code as\nwell as MCP spec uh which enabled it to\nsort of use tools.\nSo then getting into Claude code uh this\nis February 2025. So this is about just\nover a year ago. Um Sonnet 3.7 was\nreleased and this was sort of state of\nthe art on Swebench. And Claude code was\nreleased in research preview. And I\nthink an an interesting quote that I\npulled from this release actually is\nthat the goal of Claude code was to\nbetter understand how developers use\nClaude for coding to inform future model\nimprovements. So essentially when we\nreleased Claude code the whole idea was\nfor it to be somewhat experimental to\ninform how we actually improve the base\nmodel itself. And you'll see this trend\nthat over time the models become better.\nUh the harness certain aspects of it\nmight become less necessary or it will\nevolve.\nUm just just in terms of uh these slides\nas well in the bottom left corner these\nare some of the things that are are sort\nof the focus of these releases whether\nit's uh context or planning uh or\nverification and then some some stats\nbut I'm not going to sort of read\neverything. Um so yeah next this was\naround May time of last year Opus 4 and\nSonnet 4 4 were released. And just in\ngeneral um these tools got much better\nat sort of managing their own contacts\nand getting to task completion uh\nwithout reward hacking or anything like\nthat. And then\nClaude Code became GA as well and we\nreleased the Claude Code SDK. So sort of\nthe the harness powering Claude Code.\nUm little interlude here from the\ntimeline. I think everybody now knows\nabout this Ralph Wiggum technique. Uh\nyou might not know that it was actually\nlast July that this was that this came\nout uh when when Jeffrey Huntley\ninitially released the paper because it\nreally sort of gained a lot of traction\naround say December or so of last year\nuh when for example people started\nplaying around with it themselves.\nClaude also released our own uh Ralph\nLoop within the the Claude Code uh\nharness itself. But essentially it's\nit's quite sort of a simple technique\nthat you're just taking a prompt and\nyou're feeding it into Claude Code CLI\nfor example and then you're just running\nthat on a loop until uh all the tasks\nare complete. It's a little bit deeper\nthan that. I I think people tend to\nsimplify it. There's actually a few\nphases where at first you know would\nhave some kind of planning where it\nbreaks down that prompt into a few\ndifferent features and then it would\npick sort of one task from that and\nstart a new session and then work with a\nfresh context window. So a lot of those\nconcepts were were applied in the Ralph\nLoop but I think um why it caught so\nmuch attention is because it sort of\nseems really simplistic and he put it uh\ndeterministically bad in an\nundeterministic world. So the idea being\nthat it's better to fail predictably\nthan it is to succeed unpredictably. Um\nwhen we actually created our own plugin\nfor this in Claude Code\nyou'll see well I don't know if people\ncan recognize what what the major\ndifference is. There's some people say\nyou know that's not a real Ralph Loop.\nUm the idea is that this is just running\nwithin a single Claude Code session. So\nit's not creating a fresh context\nwindow. It's just relying on compaction\nto happen over time. So you know maybe\nit's not considered sort of a real Ralph\nLoop but you'd set the max iterations.\nYou'd set\na safe word, and then essentially a stop\nhook would intercept when Claude would\ntypically stop, and if it's not\nfinished, it would just sort of continue\nuntil it hits one of those exit\ncriteria.\nOkay, so on to Sonnet 4.5. This was\nwhen the model just generally started\ngetting better again at\nhandling its own context. So, this is\nwhen it became more context aware,\ntracking how many tokens had been\nconsumed. So, as it got towards the end\nof the context window, it sort of\nunderstand that, and it could manage its\nown context. Um\nClaude Code 2.0 also shipped. This is\nwhere we introduced checkpoints. So,\nactually keeping track of of the code\nover time, being able to rewind to\nprevious parts of the session. And then\nwe released We just sort of renamed the\nClaude Code SDK to the Agent SDK. And\nthat's because we realized it's much\nmore general purpose than actually just\nfor coding. So, you'll see we're talking\nabout coding a lot right now, but I\nthink what's very interesting is\napplying these long-running harnesses to\nother domains as well.\nUh at this point, we could run for about\n30 hours or so uh\nwith Claude Sonnet 4.5. But then\ncompleting the family with Haiku 4.5 and\nOpus 4.5, this is where it got really\ninteresting because all of a sudden\nrunning many sub-agents became really\neconomical. And Opus 4.5 became really\ngood at planning. So, we could start\ndoing things like using Opus 4.5 for\nplanning,\nand then using Sonnet 4.5 as the\nworkhorse for really executing all of\nthat code. Um and then there's this big\ncouple months as well because this is\nwhen we released skills, which again\nvery good at making use effective use of\nthe context with this notion of\nprogressive disclosure. So, just the\nfront matter of the skill is loaded in\ninstead of sort of all of your tool\ndescriptions, you know, which can\nconsume quite a lot of the context\nwindow up front.\nUm, and then sort of the entire rest of\nthe body of the skill is loaded in if\nit's instantiated followed by say some\nreferences to even even code that could\nrun more deterministically. And then\nmore context improvements, things like\nprogrammatic tool calling. So instead of\nrunning a bunch of tools, pulling all of\nthat into to context and then trying to\nprocess it, actually just writing code\non the fly and being able to sort of run\na series of tool calls and then just get\nthe final result back. And again, this\nis all just to improve the the usage of\nof the context window.\nOkay, so a lot going on in this slide,\nbut at this point, um, this is around\nNovember time. We released our first\nblog post on long-running agents and and\nhow you go about building these. So a\nlot of the concepts I've already\ndescribed um, should make this fairly\neasy to understand actually. Where say a\nhuman would write something like, you\nknow, write me a browser or create a\nSlack clone or a Salesforce clone, just\nsomething like really, really vague. And\nuh, the first thing that would happen,\num,\nin this harness that we built is there's\nan initializer agent that would take\nthat simple prompt and it would break it\ndown into a series of of persistent\nartifacts. The first being a feature\nlist of say X number of features.\nFeaturelist.json because we actually\nfound the models might overwrite\nmarkdown files, whereas they're they're\nless likely to just overwrite JSON\nfiles, which is kind of interesting. Um,\nit would also write a progress file, um,\nof course, sort of start the Git repo,\nuh, build an init script, and then just\nhave a flag for, you know, whether the\nfeatures are complete or not, if they\nwould pass all the tests. Um, from there\nit would go into this harness loop where\nuh, there's multiple different steps\nhere. So the first one is, you know,\nagain, in fresh context window, just\ngetting the bearings, what's the present\nworking directory, um, what's the the\nprogress file say, okay. And then um,\ndoing a smoke test or running the init\nscript, so it didn't have to figure out\nhow to do that every time, get the\nserver up and running, etc.\nAnd then picking one feature, only one\nfeature that that hasn't passed all of\nthe tests,\nimplementing that feature, doing some\nactual tests, much so verification loop,\nmuch like a human would do using\nPuppeteer in this case.\nAnd then if if everything passes,\nactually writing the Git commit and\nchanging the the state of of this\nparticular feature to passes. And then\nif there are any features that are\nunfinished, just continuing that loop in\na fresh context window. So, we're\nstarting to layer in a lot of these\nconcepts here, fresh context windows,\nthese sort of persistent artifacts,\nverification loops,\nreally good planning up front. You'll\nsee like this this is sort of the first\niteration of of these long-running\nharnesses here.\nOkay, so\ncontinuing with the history tour, so\nthen Opus 4.6, Sonnet 4.6.\nThese models are really great because\nSonnet 4.6 was basically offering that\nOpus level intelligence more at the the\nSonnet price, and it became again like\nvery much a workhorse for a lot of\nClaude code.\nAnd Opus 4.6 just became really good at\nplanning. We called it\nvery much an agentic model. So, Opus 4.6\nwas great at at deciding like which\ntools to use and just being able to run\nfor much longer. If you recall that\nmeter chart, you'll see that this was a\njump from about 4 hours up to 12 hours\nwith sort of that very simple harness.\nSo, this model is like very very\nagentic. And then along with that with\nsome of the research we'd done, we\nreleased agent teams, which the idea\nbeing in Claude code is sort of more\ngeneral-purpose way for you to say\nscaffold out your own set set of of\ncustom agents. And the innovation with\nagent teams is that instead of\neverything reporting back into the main\nagent, Um, the actual sub-agents could\ncould communicate with each other, so\nthey sort of had their own way to\ncoordinate and then report back to the\nmain agent only when it was required.\nUm, we also introduced server-side\ncompaction, which basically meaning that\nthese models can now just run\nindefinitely and compaction could just\nsort of, you know, happen on the server\nside.\nAnd then um, this 1 million context GA.\nSo now we have like one big context\nwindow. You see like the models are\ngetting better. Maybe you can just run a\nlot, you know, within a single context\nwindow even instead of necessarily\nneeding new sessions all the time. You\nsee how things start shifting over time.\nSo, that's sort of the uh, the whole\noverview. You can see all of the\ndifferent uh, releases that I shared\nhere on this table and you can see how\nit's changed from, say, Sonic 3.7 at 1\nhour uh, to 12 hours with Opus 4.6. And\nthen we have our own anecdotes as well,\num,\nwhere tasks would take, say, you know,\nlike\n20 minutes when it was Opus 3.5 and now\nwe're building, say, fully fledged apps\nthat you don't have to run for, you\nknow, 30 hours. They can run typically\nwe're seeing like, say, 3 to 5 hours.\nYou can build like a really, really\nfully featured application that that\nruns out of the box.\nSo,\nwhat's really interesting is is the\nharness doesn't just disappear as the\nmodels get better. It's really evolving\nas the models change over time and it's\nreally fascinating to sort of find the\nthe gaps in the model and then fill that\nin with the harness and then you train\nthe model on um, the using that aspect\nof the harness and maybe at some point\nyou actually remove that entirely and\nsort of this iterative uh, loop just\nkeeps happening over time with with more\nand more of these sort of co-releases\nthat we have. So,\num, yeah, hopefully that was an\ninteresting\nlittle trip uh,\nback through the Claude evolution uh,\nand how it applies to the long-running\nagents. And so I'll I'll hand over to to\nAsh to continue with where we are today\nuh in terms of the state of the art.\nUh\nall right.\nUm\nQuick question. Any of you guys have any\nagents running at the moment in the\nbackground doing work while you have?\nJust one, two, three. Okay.\nProbably should be more of you. Um\nuh hopefully by the end of this you'll\nhave some ideas to like take away and\nactually like put into practice.\nUm\nSo, yeah, that's that's a history. Um\nand I quite like that quote that uh\nAndrei talked about where the frontier\ndoesn't really shrink, it just uh\nmoves. And so, what I wanted to talk\nabout a little bit is some very simple\nuh\nkind of a harness patterns that we've\nbeen playing around with internally that\nwe use to to build these like very fancy\none-shot demo apps. Um but also, you\nknow, we're experimenting with this\nstuff in post-training um in in RL. How\ndo we make our models and and just their\ngeneral behaviors more adapt at\nautonomous work?\nSo,\nif you've ever tried to get um\nan agent to try and review its own PR,\num you'll kind of understand uh where\nthis is going.\nSo,\nthis general uh idea is shamelessly kind\nof stolen from from GANs uh\ngenerative kind of adversarial networks.\nSo, you have this\nuh generator kind of model, and then you\nhave some sort of discriminator,\num and uh you have some sort of\nadversarial pressure between them.\nYou know, the generator builds, the\nevaluator grades,\num and the whole idea here is we're\nsplitting up, you know, the context\nwindows, uh uh, system prompts, uh uh\nthe jobs entirely, right? The evaluator\nhere isn't just reading diffs, but it's\nactually using playwright,\num, to open live pages, click around,\ntry things out,\num, and then it eventually hands back\nwhatever critique it's decided back to\nthe actual generator, and you, you know,\nyou kind of continue that loop. Contrast\nthat with what most people today are\ndoing, which is kind of using one cool\ncode session, telling it to check its\nown work, um,\nand kind of loop that way. So,\nthe obvious question for me at least is,\nyou know, if the evaluator is also just\nan LLM,\num, why doesn't it just rubber stamp it,\ntoo?\nAnd so,\nthe key idea that we're kind of\nexploiting here is,\num,\nyes, the evaluator is still, uh, a large\nlanguage model, and yes, it's still\ngoing to be biased towards, uh, liking\nlarge language model style outputs,\num,\nbut tuning a standalone critic, um, to\nbe harsh is actually very tractable, but\ntuning a builder to be somewhat\nself-critical,\num, is is not.\nI think a really good analogy for this,\nright, is the same as humans. Um, it's\nvery easy for,\nuh, me to, you know, critique, uh,\na lovely piece of artwork or, you know,\na fine meal, um, much harder for me to\nactually go ahead and like, you know,\npaint that,\nuh, or or cook that meal myself. So,\nwhat we're doing here is exploiting the\ngap between the ability of an LLM to be\nkind of a critic, uh, versus a a\ngenerator.\nSo,\nthe next thing I kind of want to talk\nabout is like,\nhow do you actually think about\ndesigning,\nuh, these critics? It's very similar to\nthe process of creating good evals, but\nin the context of full stack apps, there\nare a lot of fuzzy kind of areas which\ngo into what makes something good. It's\nnot just does it work, but does it look\ngood? Does it feel good?\nUm, is there an element of taste, um, uh\nthese kind of products as well.\nSo, this is where we've been doing a lot\nof experimental work um especially when\ntrying to, you know, imbue Claude with\ndesign taste and post-training, um but\nalso,\num\nyou know, create these kind of front-end\ndesign skills that we kind of put out\nthere and just generally improve the the\nfront-end design\nuh ability of of our models.\nSo, the way we think about this is\nmost people say you can't grade taste,\nbut, you know, we think\nyou can if you have a a strong enough\nopinion on it and you just kind of write\nit down. And so, the way we do this at\nleast is with kind of creating a rubric\nwith four criteria, uh design,\noriginality, craft, and functionality.\nUm we actually weight this towards uh\ndesign and originality.\nUm we've kind of shifted the weightings\nbetween these four things uh depending\non which model's in play, but at the\nmoment, you know, Opus 4 6 is pretty\ngood at at at functionality already, so\nthe problem that we're trying to\novercome is how do we prevent things\nlike, you know, purple gradients,\ngeneral kind of AI slop type aesthetics\nin general.\nAnd we kind of just go ahead and\ncalibrate this with a few-shot examples\num on reference sites, so the\nevaluator's kind of taste converges on\nour own.\nUm and\nlet me show you an example, I guess, of\nwhat this actually looks like uh kind of\nin in practice. So,\nthis is just an example\num\nof\na model uh going through this similar\nkind of loop, generator,\nuh evaluator launches Playwright,\nnavigates screenshots, scores on those\nkind of four criteria,\nwrites critique, and then hands back to\ngenerator. So, all of these examples are\njust HTML and CSS only um that I've gone\nthrough here for maybe 4 hours, 5 to 15\nrounds. Um\nI think the interesting thing here,\num which is quite unique and something\nwhich you wouldn't necessarily get um if\nyou're just using a single kind of agent\nloop is that\nthe thing pivots, right? So, imagine the\ngenerator gets stuck on one of the four\ncriteria. Let's say it's like really\nstruggling and constantly scoring low on\noriginality. Um, you know,\nuh\nthis kind of GAN style harness which\nwe're using will just throw the whole\nthing out and try again from scratch.\nUm, whereas uh in a single pass\ngeneration or a RALF loop,\num it gets it keeps trying to patch the\nsame thing. Uh and this kind of ability\nto kind of course correct over very long\nkind of time horizons is something which\nis quite unique uh to kind of breaking\ndown uh different roles uh that go into\nto building something.\nSo,\nthat was just an insight into, I guess,\nhow we think about the front-end\ncomponent. Um\nbut how to go from kind of like just\nnice pages to fully working apps, we\nadded\nuh one more role.\nUm, a planner. And so, again, sounds\nvery simple. Um,\nit's ultimately just taking kind of a\none-line prompt uh and then breaking it\ndown into uh a very deliberately\nhigh-level uh kind of spec.\nSo,\nwhat it does is actually just spec the\ngranular um\nuh sorry, it kind of specs\nuh the general workflow into a series of\nsprints. Um, what it doesn't do and and\nwhat most harnesses do today is\nnecessarily try and plan the granular\ntechnical details of of the product. The\nreason being is, you know,\none, it's very likely to still make an\nerror, but when it does make an error,\nit's going to cascade\num through every single one of these\nsprints uh and kind of magnify errors\nover a multi- multi-hour time horizon.\nIf you kind of squint uh at this, um\nthis is kind of just, you know, a very\nsimple kind of like PM, uh IC, and and\nQA kind of org structure, right? Like we\ndidn't invent this.\nWe just kind of gave each role its own\nkind of context window.\nUm\nand the bit which is kind of\ninteresting, I think to talk about, is\nthe glue between the generator and the\nevaluator in this kind of setup.\nSo,\nbefore the generator actually goes ahead\nand writes a single line,\nwe have the two agents basically\nnegotiate what done actually means. And\nso,\nlet's say the generator proposes,\n\"I'm going to build X feature,\nand you should verify it by testing Y.\"\nThe evaluator might push back and be\nlike, \"Actually,\nthe scope is too big and those tests\nthat you propose are a bit too weak, and\nyou've missed XYZ edge case.\" And you\nbasically have this back and forth\nvia files on disk. One writes the\nmarkdown,\nthe other reads it,\nand you iterate until both agree. And\nthen once you kind of reach that kind of\ncondition, um\nthen you actually start building.\nAnd then the evaluator kind of grades\nagainst the contract\nthat\nthose two agents have decided between\nthemselves, not the original spec which\nthe planner has kind of one-shotted at\nthe beginning.\nAnd why this matters,\nas it kind of bridges this kind of idea\nof kind of user stories, i.e. the spec,\nand kind of converts it into slightly\nmore tangible, testable kind of\nassertions, some sort of contract,\nwithout the planner having to\nover-specify kind of up front.\nAnd I think this is kind of the key\ninnovation that the Ralph loop never\nreally had. It had a kind of fixed\nplan.md\nstyle\nkind of thing, but nobody on the other\nside is necessarily arguing with kind of\nthe main loop.\nAnd again, it comes back to like having\nthese separate context windows and\nadversarial pressure.\nSo, let me show you an example of\na very simple prompt that we had\nin a solo kind of loop versus the\nharness that we just discussed. So, the\nprompt was basically\nbuild a retro game maker\nand and that was it.\nAnd I'm not, you know, going to going to\ntry and convince you that this is like\nnecessarily the most cost-effective or\nmost efficient way to try and build an\napp. Um as you can see, one, it takes\nat the moment an extremely long amount\nof time. Um two, it's very expensive.\nBut also, as we'll see in a second, a\nlot of the stuff actually starts working\nonly\nwith this harness when it didn't in a\nkind of a solo loop. So,\nthis is\nwhat it kind of looked like the opening\nscreen at least when we didn't have the\nharness.\nUm pretty simplistic, a little bit\nboring,\nbut it still looks nice, right? If this\nwas the whole app, uh you'd ship it, but\nit's kind of the bait, I guess, if you\nwill.\nUm this was kind of the sprite editor,\nif you will. Again, it still looks fine.\nThe canvas is there, the palette, the\nframe timeline, live preview. Maybe it's\na little bit cramped.\nUm\nuh and the color picker is just black\nswatches, but it it kind of works.\nClearly, the agent like actually did\nunderstand what it was trying to do.\nUm and then the one thing that you have\nactually has to do,\nwhich is play mode.\nEntities rendered,\nscore, health, all the other things\nwhich go into an actual game. Pressing\nthe arrow key does nothing. Pressing a\nspace key did nothing.\nUm\nthe agent really didn't have any idea\num\nhow to test itself,\nwhat it actually meant to play a game\nand actually succeed. Um\nand\nyeah, this is kind of the same prompt,\nsame model,\nand this is kind of the the breaking\npoint. It kind of looks done on the\nsurface, but when you try and actually\npush it to its limits, it just it just\nkind of failed. And then, if we ran the\nsame prompt with the same model, this is\nkind of what it looked like when we ran\nthe harness.\nSo, this was about, yeah, 200 bucks, 6\nhours.\nUm,\nfirst up, it decided to name itself\nRetro Forge. Um, it decided to like\ncreate a a new project dialogue, um,\nhave a very nice canvas.\nUm, none of that was in our prompt. So,\nthis is all the planner um, deciding\nlike, okay,\nuh, here's what the product decisions\nshould look like. And then, you know,\nthe two other agents deciding, right,\nhow am I going to test this?\nUm, if we look at the sprite editor,\num,\nwe have kind of a full 54-color palette,\num, the kind of eight-bit preset from\nthe project dialogue flowing through.\nUm,\nyou see the sprite at actual game scale.\nUm,\nit's a lot more complete, uh,\nas a product in general.\nUm,\nwe had a whole new kind of\nAI-level assistant. This is where it's\nkind of started to get recursive.\nThe planner had decided like, right, we\nshould have some AI features,\num,\nuh, which is just a very vague line in\nthe spec.\nAnd the harness turned that into a full\nAI-level assistant inside the app that\nit it was building. So, you know,\nsomeone could come in and say, \"Hey,\ncreate a castle uh, with sprites guide\nguarding it, let's say.\" Um,\nthis is something which a solo run would\nnever even attempted to look at. Um,\nwithout a planner, that phrase just\nbecomes never even comes becomes like a\nwork item to look at.\nThen, finally, I guess,\num,\nuh, the actual results kind of applied.\nSo,\nplay mode, um,\nyou know, you have this whole debug HUD\nin the top left, um, which you can\nclearly tell is to make life easier for\nthe for the for the evaluator, for\nexample. Those numbers are live. The\nphysics loop is actually running. Um\narrow keys work, the player moves, um\ncollides with castle walls, um because\nthe evaluator actually launched the\ngame, tried to play it,\nknew necessarily like what\nwhat features need to be tested to make\nthis game kind of real and successful.\nUm and the difference between this\noutput and you know, the previous output\nis entirely just scaffolding. It's a\nvery simple loop ultimately, but the\nresults are quite startlingly different\nat least.\nAnd so,\nin case you're curious, the kind of\nthings which the evaluator\ndid catch,\num\nare pretty basic kind of stuff. It's\nthings like um\nyou know, fast API route ordering,\num\npasses every unit test but might\nactually break in prod, the evaluator,\num catching things like the delete key,\num having some kind of Boolean logic\nbug. Um\nAgain, these are things which were only\ncaught because the evaluator's actually\nusing the app. Um it's things which\nmight get through CI in a rough loop,\num\num but\nthis isn't, you know, that level of\nspecificity isn't something which\nhappened by accident.\nAnd so,\nthis is the kind of level of detail\nwhich these models are kind of going to\nat this point in time. So, we talked\nabout the kind of contracts that the\ngenerator and the evaluator would write\nbetween themselves. For this app, um it\nwas decided that there were 27 contract\ncriteria. That's the level of\ngranularity which we found, you know,\nthat you really need to make findings\nkind of actionable.\nIf you have vague criteria, you have\nvague critiques, the generator just kind\nof shrugs and does things,\nwhereas if you have granular criteria,\num the agent knows, okay, I need to fix\nthis exact line.\nWhat's kind of interesting I thought,\nyou know, uh and I want to be honest\nabout this part, is that out the box,\nClaude is a really, really bad just\ngeneral QA agent. Um\nAndrew talked about this uh a little bit\nuh in his bit, right? But\nthe same kind of six-fency and\ngenerosity bias that everyone hits with\nuh general elements of judge systems\nalso applies here.\nUm\nMost of the time in early runs, it\nwould, you know, the QA agent would kind\nof find a bug uh and be like, uh\nfix it later, might take 2 weeks. Um\nuh and then just kind of like be done\nwith it. Um\nSo, we actually have to spend an\nexorbitant amount of time like going\nthrough\num\ntrying to tune, you know, small layout\nbugs, edge cases, and and kind of\nfeeding that into the prompts.\nI wish there was some kind of secret to\nto actually doing this, but\nrealistically, the whole\nuh kind of art to building this system\nand making it good uh was kind of\nreading the traces.\nUm\nThe primary debugging loop was this, and\nnot necessarily running more\nexperiments. It was reading what the\nagent actually did,\num finding where its judgment diverged\nfrom um ours as humans, and then tuning\nthe prompt for that.\nIt was the same kind of muscle as\nreading kind of a stack trace.\nUm\nOne kind of tooling tip that we had was\nkind of piping agent transcripts uh into\nfiles, uh kind of grapping them uh with\nanother agent, or having another agent\nkind of play through them, um and then\nkind of update the prompts itself. So,\nyou have some sort of like closing the\nloop even on just like building\nuh this harness out.\nSo,\nthe last thing I kind of want to talk\nabout was um\nhow to think about adjusting your\nharness as these models kind of get\nbetter in time. I think there's a lot of\nlike discussion around whether harness\ndesign is kind of dead or null,\nespecially where, you know, models that\nI mean, when I wrote this, it was just\nOpus 4.6, but even like Mythos, you\nknow, level level models.\nAnd\nI think the key thing that we note noted\nis it's really important to get a feel\nfor what the kind of spiky behaviors of\nany individual model are, and then try\nto adapt your harness to kind of fill\nfill the gaps. So,\num\nAndrew talked about this a bit, but you\nknow, context resetting between\nsessions. We kind of dropped that\nentirely. Opus 4.5 he started really bad\nkind of context anxiety. Um\nwhereas Opus 4.6 just, you know,\ndoesn't. Um uh as part of the part of\npost training that. And so, one\ncontinuous session and compaction was\nwas more than more than enough to handle\nvery long sessions.\nSprint decomposition. Um\nwe don't have a very strong opinion on\nthis, but it was something which was\nreally really critical to getting Opus\n4.5 to work. Um\nbut\nuh Opus 4.6 was able to kind of hold a a\n2-hour continuous build coherently uh in\na way without Nassy having to be\nforce-fed one feature at a time. Um\nthe cadence at which the evaluator\nshould run. Previously, we were running\nat every single sprint per se, whereas\nnow we were just running at at the end\nof a one-shot generation from the model\nand then passing back. So, the harness\nis still the same, we're just kind of\nsimplifying the specific uh kind of\nloops um and the kind of recipe that\nkind of goes into it.\nThe lesson isn't necessarily our harness\nwas wrong,\nbut rather\nit was right for 4.5, the frontier\nmoved, um and we ran a simplified\nversion uh to see how it worked.\nSo, this is kind of what the final kind\nof setup kind of looks like today. Um\nHaving that planner generator evaluator\nloop is still the kind of core of our\nsystem, but you can see we kind of\nditched a bunch of the other kind of um\nkind of components uh that made this\nslightly more complicate complicated\nthan it had to be.\nUm we also, as kind of mentioned, big\nfan of just using a file system for\nshared state um\ninstead of kind of leaning on context\nwindows for very long-running agents in\ngeneral.\nAnd this is an example\nof the simplified harness running\nwith one of our latest models. Um\nagain, very very expensive, but you can\nsee it's actually roughly like half the\ncost of the previous runs. Um\njust because\nwe're kind of doing things in a slightly\nmore simplified manner, but it's still\nrunning over a very extended period of\ntime.\nAnd so, this is an example of a DAW,\nwhich is basically just like a a music\ncreating app, if you will.\nUm\nthe agent sets the tempo, a key, it lays\ndown the melody, it builds the drum\ntracks. This is the evaluator\num actually going and\ntesting the app itself. Um\nwe did actually listen to like the music\nin this.\nUm obviously, Claude\ncan't hear at the moment, and so the\nmusic was pretty trash, but\nthe app was really good in general and\npretty pretty fleshed out, which, you\nknow,\na model ago, this is something which\nwould never have worked.\nUm\nbut this is something which is possible\nwith just a couple rounds.\nUm and this is kind of that that meter\ncurve which Andrew was talking about\nkind of really in in action.\nAnd so, I kind of wanted to close just\nby saying\nyou don't actually need, you know, our\ninternal harness to to go away and start\nthinking about this. We are constantly\ntrying to ship bits of, you know, these\nprimitives into Claude code directly,\nbut also there's nothing stopping you\nfrom just going ahead and building\nsomething similar to this\nkind of on your own. So, we just\nshipped, you know, auto mode is probably\nmy favorite thing\nfor slightly more, you know, safe safe\nyellow, if you will,\ninstead running dangerously escape\npermissions all the time.\nUm,\nwe already have custom sub-agents as a\nprimitive, right? Your evaluator, your\nQA role, um, give it a harsh system\nprompt and a very detailed re-break. Um,\nPlaywright MTP or Claude for Chrome MTP,\nalready extremely extremely good, uh, at\num, web app stuff or just use computer\nuse if you're building kind of native\napps. Um, and skills, again, a very nice\nway to package your kind of grading\nrubrics into your kind of general\ndevelopment flow. Um,\nso, yeah. Five things, if you're kind of\ntaking a photo, this is the the slide I\nwould say to to to kind of remember.\nUm, self-evaluation, very much a trap.\nJust use an adversarial evaluator. Um,\ncompaction doesn't necessarily, uh, does\nnot equal kind of coherence, right?\nLossy summaries really drift. Um,\nstructured hand-offs, uh, and clean\ncontexts, uh, are very good patterns\nthat I've seen. Um, don't think that's a\nsubjective quality isn't gradable. If\nyou have a strong view on what something\nshould look like, um, then kind of force\nyourself to write it down.\nUm, we found this made kind of a really\nmassive difference, uh, to the quality\nof kind of apps, um, that a model was\nable to generate. And then kind of\nfinally was really just, you know,\nsit with the model, read the traces,\num,\nuh, only then can you kind of really\nknow what bits, uh, of a scaffold to\ndelete,\num, what bits to keep, especially as the\nkind of frontier, uh, moves. But, yeah,\nthat's it from me.\nUm, thank you very much for listening.\nUm,\nand yeah, check out our blog post. Um,\nbut, want to just open up for Q A in\ngeneral, cuz we've been yapping for\nlike, you know, close to an hour now.\nSo, um, if you have any questions for me\nand Andrew, just fire away. We'll do our\nbest to to try and answer them.\nYeah.\nThank you. Uh Joan from Paul side.\nUh one question for you, when you uh\nimprove the evaluator by like reading\nthe logs and improving it, is that uh\nsort of like on a per project basis or\nmore of a secret sauce that you you\nreuse across project? The goal the goal\nis very much trying to do this in a way\nwhich was\nreusable, right? Like I think anyone can\ntune this in a way that's that's\ncreating, you know, a very specific type\nof app, that's fine. At that point it's\nnot that different from, you know, going\nahead and just prompting tool code\nyourself and and doing it, right? I\nthink there were just the the key was\nlike what are the common patterns uh\nthat you can kind of draw across\nthe model weak points, right? So,\ntalking to that kind of front-end design\npiece, we knew like\nwhat we thought good design would be,\nright? You could give examples like\nthis is what, you know, um uh a read\nbefore prompt looks like. This is what\nAI slop looks like, right? Um and that\ngeneralizes quite well, so\nyeah. This was all around web apps, but\nit could quite easily apply to to other\nkind of things as well.\nThanks for\nOh. Test. Oh, yeah. Uh\nThank you for presentation, very\ninteresting. Uh I was just wondering\nwhat is your view on um\nuh\nconcept of dump zone and smart zone of a\nmodel. So, I understand like\nbefore it was around 40% and now with 1\nmillion context it's about 100K is what\nI understand and the way how I\nunderstood Ralph loop\nRalph loop was designed is to kind of\nnegotiate this problem. So, basically we\nkeeping the\nmodel always in a smart zone, so\nbasically trying to\nslice the task below 100, so it\nexecute the task within the 100 context\nzone. And what I understand from your\npresentation, you can like advocating\nnot to use it anymore so because we can\nnow rely on a compaction\nand so on and so on. Is it like\nsomething you use\nsuggesting to do or we still with like a\nRalph\nloop model still has its own place given\nthe smart and dumb zone\nconcept.\nYeah.\nYeah, well, I suppose from Ashish's\npresentation and mine, you see that the\n1 million contacts window is now GA, and\nso you have sort of a bigger context to\nuse.\nThe models are more agentic, so they can\nsort of maintain coherence for a longer\nperiod of time within that context\nwindow. And then actually, with the\nrelease of 4.6, we decided to move from\nnew context windows to just a single\nlong-running continuous session with\ncompaction.\nSo, I think I mean, the\nwhether or not you use multiple fresh\nsessions or just one long-running one is\nprobably still up to your use case and\nyour evals,\ndepending on what you're seeing as\nworking best. But at least for sort of\nthis general\ngenerator-evaluator\npattern with Opus 4.6, we saw that it\nwas possible to use a single session. I\ndon't know if you want to add to that. I\nthink I think it's also just like a\ntemporary problem, right? Like\ncontext rot is\nyou know, a failing of like today's\nmodels to some extent, and much less so\nthan, you know, even just one model\ngeneration ago. So,\nis there a place for for\nyou know, the type of thing which you're\ndiscussing? I think\nyes, depending on your use case, but you\nknow, it's not like a\nit's one of those pieces which I'd look\nat as like, okay, as soon, you know, I'd\nbe I'd be kind of hunting for the model\nrelease right and kind of strip it out,\nlet's put it that way.\nI always have a lot of FOMO around\nPlaywright. I mean, you said Playwright\nMCP, is Playwright skills.\nDo you Can you speak to how to improve\nthe Playwright Cuz like, I imagine I\nwould like to like have my browser open\nand then I can see the model working\nthrough and then maybe I could steer it,\nyou know, a few tabs open.\nBut like, yeah, what\nis there some innovation that I'm\nmissing out on or is is Playwright MCP\nreally\nwhat you recommend people use?\nPlaywright MCP or just use the code for\nChrome MCP, which is like a a slightly\nmore\nrobust\nthing, I guess, around browser control.\nI mean,\nI don't know why you want to watch it do\nthings. I mean, you can, but I think\nthat's like a a trust gap, right, today?\nLike, you know, the whole point of what\nwe try to get to do here is is like\nyou set something off, uh you trust it\nto do it do the work and test it and you\nhave the confidence that it's doing it\ncorrectly and you come back to it. Um\nand that's where, you know, yes, there's\ngoing to be some iteration at the\nbeginning where you're like watching,\nreading the traces until you get to a\npoint you can trust it. But\num at least internally, right? Like,\nwhen I'm when I'm doing full stack out\nthere,\num\nI have got to a point now where I'm\nlike, \"Okay, with Opus 4.x, I can like\nreliably trust the model to go ahead,\nread\num network errors, um\nuh\nuh console errors, actually navigate an\napp, zoom in where it needs to,\num the vision is now good enough on\nthese models that it can like identify\noverlapping text on elements and things\nlike that, whereas that just wasn't the\ncase uh\nuh until realistically the last, you\nknow, generation of models. So,\nyeah, I would I would recommend um\nI'm curious, like, with the generator\nevaluator pattern, what happens?\nDo you can you throw unlimited tokens at\nit or will it stop because the evaluator\nis not good enough? Like,\ncan you tell me more about that?\nSorry, do you mind clarifying? I kind of\nmissed that part.\nokay, let's say uh um\nI say, \"Okay, create like a very cool\ngame and\nwith\nsome features.\"\nYou have the generator-evaluator pattern\nthat um creates like the contracts,\nbuilds the apps.\nIf I\num then it will give me back something,\nright? Um\ncan I restart it again and say like,\n\"Okay,\nmake it better. I'm not happy about it.\"\nAnd generator-evaluator will pick the\npattern will pick it up and make it\nbetter. Yeah. Or will the evaluator be\nnot good at the one point and just say\nlike, \"This is it.\"\nUm\nI think that's I mean, one festival like\nif you want to like a some some level of\nhuman loop in this process, that's\nthat's like, you know, just implement\nhooks at some point in some point in\nthis in this loop. Um\nI think the bit which was kind of\nsurprising to us\num\nuh was\nwith this\nthis general pattern and especially with\nthe kind of 4.6 atom models, both Sauna\nand Opus, it was extremely willing to\nlike throw away\neverything, you know, even if it'd done\nkind of 10 passes at something.\nIt was kind of very happy to just like\nthrow it all away and start from scratch\nif for some reason it wasn't able to\nlike hill climb against the rubric of\nthe evaluator in a kind of effective\nway.\nUm and so\nthat's why kind of when we're kind of\nwhen we were playing with this kind of\nthing, we didn't naturally like lean\ntowards having\nsome kind of resume or human in the loop\nuh type intervention system, I guess.\nAnd we didn't really observe We expected\nto, but we didn't really observe um\nthat kind of behavior which you were\ntalking about, where it kind of just\nlike evaluator's like, \"Ah, just give\nup. Let's just like pass it on, shall we\nsay?\" Um\nYeah, it was just much more willing to\nlike throw away everything and and\nrestart. And that was just a behavior\nwhich we never saw when it was the\ngenerator itself or was kind of being\nproud of its own work and being like,\n\"I'm not going to restart this whole\nthing.\"\nUm\nso\nyeah, I mean there there's been example\nwhich I've seen where the evaluator is\nlike\nit kind of gets fed up and is like,\n\"Right, this approach you're taking just\nobviously isn't working.\nCan you just like delete everything and\nrestart?\" Um which I don't know about\nyou guys, but while coding regularly, I\noften I often do uh\nas a human to like, you know, just\nbenefit from fresh context windows, not\nhave to deal with an an already messy\ncode base, etc. So it's quite neat\nseeing models now also kind of get to\nthat point.\nI'd also just briefly add obviously you\ncan then open that code base in Cloud\nCode and continue where you left off. Um\nsort of goes without saying. And I think\nwe're generally thinking about what the\nworkflow looks like if it's sort of more\nback and forth um cuz there's sort of\nthe extreme of build me a really complex\ngaming application that you don't know\nis it going to take 3 hours, is it going\nto take 20 hours? Um it's a bit unclear,\nso maybe there's sort of something in\nthe middle that's like more of a yeah,\nfeedback loop.\nI um I really like the idea that you\nhave of like\nyou know, there's a\nthere's a human element here where it's\nlike, you know, PM, engineer, evaluator\num\nPM role is a lot of the time is like\nscope creep and\nkeeping the time going and stuff like\nthat.\nBut you're just like letting this off.\nYou're letting\nengineers go play in the sandbox for\nages.\nUm\nis there a harness loop that needs to go\nback to the planner eventually? Does it\nneed to move again? Um well maybe\nbecause we're engineers, we just decided\nlike, \"Ah, screw the PM. We'll just\nshove it to the side.\" Um\nyeah.\nWe actually Well, this is where the kind\nof like that kind of contracting piece\nbetween the the\nthe kind of\nuh evaluator and the builder work quite\nwell. For context in that, we typically\nlike insert the main spec that was\ngenerated by like the PM per se uh into\nthese sessions regularly. So that, you\nknow, it's always a reference point um\nfor like, \"Okay, this is what we're\nstill actually trying to build.\" And the\nmain function of them, the builder and\nthe generator, uh sorry, the builder and\nthe evaluator is just to like figure out\nthe exact feature set and tests and\ncontracts that say that actually satisfy\nthat spec. Um\nbut\nthe reason we don't is because\nwe don't want the planner to be like a\ncore part of this loop. It should be\nvery high level. It should Its purpose\nreally is just kind of\nset out like kind of the hard outer\nlines of what this product could could\nbe. Uh but its job is not necessarily to\ncome in and intervene and be like,\n\"Actually,\nthis is like an impossible feature. We\nshould not do this.\" And and and edit\nitself. Um\nwe kind of want to\nkeep that context relationship between\njust uh just the builder and the\ngenerator. That being said, like this\nloop I've applied it in lots of\ndifferent ways. It doesn't have to just\nbe, you know, one generator and one\nbuilder, right? Like that adversarial\nkind of trade-off can be applied to like\na workflow consisting of multiple\nseparate agents, right? Um\nI don't know. It could\nuh if you're trying to do uh I don't\nknow, generate evals, let's say. You\ncould use a similar harness to be like,\n\"Hey, generate a\nuh it could be like planner\ngenerate a synthetic a generator for\nsynthetic data set, right? Uh with a QA\nagent. Then hand off to\nuh like an integrator which like\nactually wires up something. Also has a\nQA agent. Then has like a final kind of\na\nYou can basically add this kind of\ngenerator evaluator thing into a\nmulti-step workflow uh where each like\nbuilder that maybe has like a slightly\ndifferent function per se as part of a a\nlonger workflow. So\nthere are different ways in which you\nkind of keep things on track depending\non the task and break down uh\nthis this general pattern into slightly\nmore specified\nyou know, uh tasks or workflows if that\nmakes sense.\nCan you uh you mentioned that uh some of\nthe later tasks could not possibly be\ndone by an earlier model? Can you talk a\nlittle bit about your process comparing\nthe tasks on the different models? Like\ndo you fire off the same task on Opus\n4.6, Opus 4.5, Sonnet\nor is this sort of artisanal\nuh co-evolving uh harness model\nuh setup uh obviating that?\nYeah, I mean I suppose we walk through\nthe history a little bit. And if you\nlook at say the first blog post on long\nrun running agents versus the more\nrecent one, um there are some pretty\nsignificant differences there. They are\none being what we were just discussing\nthat the initializer agent would build\nthis super comprehensive spec of say 200\ndifferent features um and then um\nin\nthen the loop would have to actually go\nand execute against every single one of\nthose features, which may lead to say\nincorrect design decisions, but it's\nsort of forced into that behavior.\nWhereas I think now\nyou're able to have sort of a more\ngeneric creative direction set with say\nOpus 4.6 and then just having this this\nloop of the generator evaluator. But it\nit it does Yeah, your model selection\ndoes inform your harness design very\nmuch so. Um of course, in a perfect\nworld, you could just sort of throw\neverything at say Opus 4.6, but if you\nhave cost concerns, for example, maybe\nyou do use Opus 4.6 for planning and\nthen Sonnet 4.6 for for the coding or\nthe execution, that's something that we\nwe tend to see quite frequently. Um but\nagain, if you're building specific sub\nagents for each of these, you probably\nwant to have some evaluations to be able\nto understand for that model and that\nprompt how it's performing against that\ntask and then just optimize.\nDo you have any advice on moving beyond\nsort of these one-shot applications to\nlong-lived products where\nyou're looking to make changes days,\nweeks later?\nAnd what sort of artifacts you need to\npersist to future instances to be able\nto know what has come before, what can I\nchange, what should I change? Yeah, it's\nsomething which we're working on.\nUm\nlike right now, like we use like similar\npatterns for just a bunch of random\nstuff internally, shall we say? And so\nat the moment it's like set this thing\noff, it's running, you know, um on a\nremote server somewhere. Um I'll just\ncome back and check it like after this\ntalk, let's say. And\nthen I kind of iterate on it kind of\nmanually in code directly, like polish\nany rough edges, that kind of thing.\nI think\nin terms of the way that you're actually\nlike setting up this harness, just\nhaving\num this is why we kind of default kind\nof using a file system of state\nfor this kind of loop. One, because it's\njust very easy for another model to come\nup and and grep through and and pick up\nwhat what's been going. But\none thing which I like to do is kind of\nembed a little bit of prompting\nthroughout this kind of loop, which\nbasically tells it to write kind of\nlearnings and state to\nsome kind of\nJSON file because the model doesn't kind\nof overwrite that too much. Um and so\nthe nice thing about that is you're\nbasically just leaving like breadcrumbs\nfor another model to come and pick up.\nSo\nhonestly, the the key thing for me is\nlike how do I\ninstruct this this harness to leave\ncrumbs for a human to come in and then\nuse code code on top of. So\ngenerally, it's like hey uh\nthe shape of that file might be like uh\ntried this, evaluated, found this bug,\nuh implemented this fix, this fix\nworked, yes, tick. And then continue.\nAnd you have kind of like a time stamped\nuh\nkind of\ntime log, if you will, of like\neverything the model has tried, the fix\nit's made, and the final state. Um and\nthen also uh having some sort of\nlive updating kind of set of docs, if\nyou will. Just very high level, here's\nthe file structure. And then those two\nfiles, to be honest, are more than\nenough for Claude code and a human to\ncome in and start iterating on the app\nwith. But\nthat's what we're doing at the moment.\nMhm.\nPerfect.\nUm\nSo, it's very interesting to hear the\nOh, yeah. First of all, congrats on the\npresentation.\nThank you.\nUm and then I was wondering, there are\nkind of two approaches. Like, you have\nthe agent team\nwhere multiple agents interact with each\nother.\nAnd then the explicit generator critic\nuh setup.\nBut what are the uh\nbecause in sense, like, the agent team\nhas the same setup where the main agent\ninstructs someone and then can act as a\ncritic for the sub agent.\nBut what are the current failure modes\nthat causes us to still need the\nspecific generator critic harness\ninstead of just\nthe agent uh team\nitself?\nAnd then what's your estimate of how\nmany model generations we would need to\njust\ncompletely rely on the agent team\nMhm.\nMaybe Clearly.\nI mean, I can I can sort of address the\nfirst aspect of that. So,\nI mean, one of the limitation of\nFirstly, Claude code is is using the\nsame harness that that is the agent SDK.\nSo, you can Technically, you should be\nable to build this type of pattern into\nClaude code. Um agent teams is a useful\nframework for potentially doing that\nbecause you could say have the the\ngenerator and the evaluator sort of\nintercommunicating or it maybe it's the\ngenerator is sort of the main agent and\nand the evaluator is say a member of of\nthe agent team. Um\nbut I think it's it's sort of evolved\nmore so from that first blog post that I\nshared. I think that was like the the\nresult of that to some extent to try and\nmake that more generally available. Um\nbut one of the things you're limited by\nobviously is like cloud\ncode would just have to run on your\nmachine. I think with the agent SDK you\ncan also just run it in more of a cloud\nenvironment and a sandbox environment\nfor long periods of time and um without\nit it failing um or you having to run\nthe caffeinate on your machine.\nUm but I think yeah cloud code is a good\ntesting ground for building out any of\nthese types of harnesses to experiment\nand explore and see what works before\nmaybe you build it into the agent SDK\nand then actually deploy it as its own\napplication.\nUm and yeah I I mean again I would just\nexperiment see if agent teams is\nsomething that makes sense for you or if\nmaybe just using regular sub agents or\nsome other\nframing of it it works better. But yeah\nI think people are using agent teams\nlike\na a ton. I I don't I don't know if you\nYeah well this is the thing is like I\ndon't think we have like a super\nstrongly opinioned viewpoint on like um\nwhat is the best at any you know set up\nat any given moment in time. And so\nBoris always like updates his tweets\nlike this is what I'm doing now. Um\nlike agent teams is something which\na bunch of people loved internally um\nand so we were like okay let's ship it.\nLet's see what people think about it in\nthe field.\nUm\nI'm not saying we will but it you know\nwe regularly unship things as well. Um\nand I do see the generator evaluator\nkind of pattern is like a a subset of\nthat like teams approach to thinking\nabout sub agent design. Um not\nnecessarily like contradictory to that\nper se. You know you can imagine like\nyou know classic language teams breaks\ndown is like, you know, front end, back\nend,\num some sort of integrated between them\nlike sub agents. Each of those probably\ndeserve their own kind of critic,\num\na kind of agent pairing with them, for\nexample. Um so, you can kind of see how\nthe two concepts like overlap. Um just\nthe general idea behind this is\nyou know, most people when they're\nrunning cloud code, at the moment, their\ngoal isn't to like one-shot an app over\nlike 6 hours. And so, that isn't\nnecessarily primitive, which we like by\ndefault like ship\nuh there.\nUm so, yeah.\nOne thing I was wondering, have you also\ntried like a critic that gets the\ncontext of the generator?\nThen you\nI feel like if it has some clue about\nthe traces of the agent or like the\nexecutor. Yeah. Is that currently the\ncase in the the critic? Uh we do we use\nlike a handoff pattern. I\nI would be very hesitant of that. We did\ntry this, but this is the whole like\nmuddying of\nof like thoughts between the two two\nmodel streams. I think it's actually\nmuch more effective to just let it judge\nthe output. Um and just provide, instead\nof being like, \"Hey, you made a misstep\nwhen building this by doing X, and\nthat's what's resulted in this issue.\"\nIt's much more effective to just have\nthe value to be like, \"This is an\nissue.\" and then let the generator\npurely reflect on its own work and then\ntry and figure out how to fix that\nissue. Um otherwise, you kind of just\nsee\nwe found that\nit's very easy for the model to like kid\nitself that something is working or not,\nand that feed into the evaluator as\nwell.\nAnd last note on that,\nI think it would then be interesting if\nyou\nif for the training team, if you could\ntrain like the the generator to predict\nwhat a critic currently said. Yeah. Like\ndo you do you have it be more honest\nabout what it did and stuff?\nMaybe we'll work on that.\nUm I want I wanted to ask more about\ntraceability. Like I use um superpowers\nor like my own prompts to\ngenerate like multiple sub-agents to\nimplement my let's say my software or\napp.\nBut what happens is like\nI don't really know. I want to go back\nand see where it actually went wrong.\nEven but then I'm not able to figure out\nhow to find those traces. How do What do\nyou use for traceability is my question.\nWhen you have so many like five, six\nagents running in background\nlike um yeah. Um\nto be honest, a lot of it is just\nreading through traces by hand. Um\nwe do a lot of that. I'd say Anthropic\nin general just like reading through\ntraces by hand.\nUm\nwe also just like have, you know, hacked\ntogether various things where we\nyou know, point Claude at uh\na bunch of traces um uh with some custom\nprompts uh to try and identify like\nissues with the loop like this is where\nit veered off and whatnot.\nUm\nwe kind of use that as like a first pass\nI would say maybe to just kind of like\nsee where something where where like\nwhere something might have gone wrong.\nBut to be honest, by far and away the\nthe the best approach at least that we\nuse internally is just just reading\nreading the traces by hand. Um\nonly then do you kind of like truly get\nto kind of relate to what the model is\ntrying to actually do.\nUm\nyeah.\nThanks for the talk. Um I have a few\nquestions. Uh first of all, how do you\nmeasure the quality of a Harness agent\npair?\nIs it It feels like a vibe check. Like\nit's a green field. Uh let's build an\napp. Mhm. Um but let's say you're\nyou're going into a new project, maybe\nbrown field. Um\nit feels like a vibe check or some kind\nof art. Can you make it more scientific\nor is that just not feasible?\nI I mean the way that we thought about\nit at least, right? Is like we specified\nthe rubrics in kind of extreme detail at\nthe kind of generator and and evaluator\nlevel, right? So we talked about for\nexample those four kind of criteria,\nthat's very high level, the rubric which\nwe use for like design taste, let's say.\nAnd so\nwe set those up for various bits of this\napp, right? So that can be just for the\ndesign element, maybe another piece for\nlike how we think about um uh kind of\nAPI design, let's say, um\ncode quality, whatever. And we kind of\nuse those as the uh\nkind of\nvarious set of rubrics which we're hill\nclimbing against, right? And then the\nevaluator's job is to, you know,\nencourage the builder to hill climb\nagainst those. And so for any given app\nor output, we have like a signal of this\nis where the model started on those\nthose kind of\ncriteria and this is where we kind of\nended up. Now,\nthat's less useful for like kind of like\nyou said, working on\num\nkind of newer codebases, but it still\napplies, right? Like you could point\nyou just have to start the loop in a\ndifferent way. Um you just point the\nevaluator at a given codebase.\nUm\nand be like, this is where we are now,\nand then uh give it the the spec of what\nyou're trying to achieve, and then let\nthe loop kind of iterate against those\nkind of criteria. So it's not like a\nnecessarily a one set of evals at the\nvery end, it's kind of like\nhere are the criteria for what we think\ngood looks like, then letting the uh\nevaluator and the generator come up with\na set of kind of tests\nuh or contracts that need to satisfy,\nand then letting it just as the harness\nhill climb against those. Um\nthat's not super comparable across\ndifferent products and runs,\num but it's it's very useful for for\nyeah, within a product or run. Also,\nthis um\nthis particular pattern is it's great\nfor greenfield, like you said, but it's\nquite opinionated. You know, I might be\nusing React, like Postgres as a\ndatabase, and Node on the back end, but\nyour brownfield app might be using\nsomething totally different. Or the\nrubric that we've created for what we\nthink, you know, good sort of design\npatterns are would might be totally\ndifferent in your project. So, I think\nthat's why we're proposing this as more\nof a pattern that you would then um\ntailor towards, you know, your\napplication.\nThanks. Um one follow-up question.\nDoes it work?\nYeah. Um\ndo you use uh\ndo you like direct the harnesses\nindividually and how do you cooperate as\na team on that? I find it's very hard to\nlike uh\nwhen I share my screen and I'm working\nconversationally, it's very hard for\npeople to keep up. And the other way\naround, I find it cumbersome to dictate\nwhat to prompt.\nUh how do you cooperate as a team?\nDo you have like team-owned harnesses?\nUm\nIs it maybe a good feature for Cloud\nCode?\nUm\nYeah, maybe. Maybe I I think we we\nprobably do have a lot to do on that,\nright? Like I think like\num\nquite often what happens internally is\nthat, you know, people come up with\nthese ideas and then they're generally\nquite bottoms-up adopted by different\nteams and\num it's then the job of the kind of\noriginal, you know, uh\nidea holder, should we say, which was\nPrithvi in this case, to kind of\nmaintain it and make it kind of\ncomposable and generalizable for\ndifferent teams, and different teams\nwill\nadopt it and and you know, uh make it\nuseful for like their section of a code\nbase, let's say. Um but we don't have\nany like\ngood things in that sense. I think like,\nyou know, even just observability, like\nsome of the people talked about, right,\nis like a um\ngenerally speaking, a thing which is not\nfully solved yet for these like\nultra-long-running uh agents. And yeah,\ninteresting area of kind of greenfield\nsoftware to explore.\nYeah, that is that is an interesting\none, whether it should be sort of a\ncollaborative experience in in Claude\ncode or even Claude.ai. I think in\njust leveraging software engineering\nbest practices with version control and\nmaking your commits and pull requests or\nif you're working on your own using\nsomething like Git work trees so that\nyou're not overriding the the file\nsystem on multiple different features\nall make sense, but yeah, I think when\nit comes to collaboration, maybe it's\nsomething that you know, doesn't happen\nquite as much because people just build\nthese projects as Ash said from the\nground up and then sort of\npresent them to the rest of the company.\nUm I\nUh Jose from Mercedes-Benz research and\ndevelopment here. Hi, thanks for the\ntalk. Um while looking at it, I thought,\nokay, it looks a lot like a scrum team,\na feature team working\nfor longer times uh on a on a product.\nAnd I was thinking\num\nhow does human in the loop look like in\nthat scenario? Um because\nyou have\nyou have this kind of sprint. Uh have\nyou thought about a sprint review\nkind of moment where you where you as a\nhuman get asked, \"Oh, hey, here's what\nwe built the last 2 hours. Yeah. How's\nit looking like for you?\"\nYeah. Should we subject our agents to\nsaving trauma that like engineers go\nthrough\nof of of of scrum review?\nI mean, like the whole point of this\ngeneral the general idea which behind\nthis talk and also\nwhat we're trying to do is like trying\nto be as like agile as possible, right?\nLike how do we build harnesses where we\ndon't need a human in the loop, right?\nLike what does that look like? Are we\nusing this today for everything?\nObviously not, right? Um\nuh but the goal is\nyou know, this is this is a technique or\na pattern which should extend very\nnicely such that you you don't have a\nhuman in the loop for most things.\nIf you did, right? It's like uh you\nknow, hooks is probably the main\nprimitive uh to just basically inject uh\nI some sort of specific type of stop\ncondition that's say with an evaluator\nto basically like hand back to human\nallow some kind of develop a message\ninput and then continue the loop would\nbe like kind of a simple way to\nimplement it. But\nyeah, to be honest\nwe're kind of exploring this from a\nwhat can we do fully autonomously kind\nof approach as opposed to thinking this\nas like\nhere's gold code and and like\nhow do we make this like you know more\npowerful per se. It's very much like a\nkind of more green field exploration of\nagent design.\nNo of course it's just like\nif if if I would get the chance to\nreview it maybe a few hours in\nand I might be able to steer it in a way\nbetter way. Yeah. So that 8 hours later\nit's more like the one kind of project I\nwould like to have. Yeah, I mean\nI get what you're saying. I think the\nquestion then is is like should that be\nlike a permanent feature of the harness\nor is that just like a\na thing which you should have like kind\nof basically prompted around when\nbuilding the harness, right? So we would\nhave that, right? We would run this this\nthis harness on loop\nand we would have you know we might spin\nup like 10 generations of different\nthings and like three of them succeed\nand seven of them fail in like random\nways.\nAnd then we would just sit down with\nthose seven, read through them,\nadjust the prompting of the main harness\nand then and then try again and then\nuntil we get to a point where we're like\nquite happy leaving it run leaving it to\nrun fully autonomously. So ultimately\nthat's still the end goal for us as\nopposed to being like\nbasically giving up on the harness and\nbeing like okay we'll just insert a\nhuman here to to like cover for any kind\nof stability issues instead but rather\nembed that and bake that into the\nharness itself\nin the first place.\nHave you used this to build anything\nlike sort of like non green field or I\nguess like production like anything in\ngood code itself or have you used it for\nactual features and and seeing it to the\nend?\nUm I think I mean this does mostly\nextend to greenfield projects. I think\nfor brownfield maybe you do need a\nlittle bit more control\num as you're starting to build out your\nown rubrics and and patterns. Um\nI mean what we're seeing\nin brownfield is that if you look at the\nwhole software development life life\ncycle it's not just the coding aspects\nthat people are starting to use\nsomething like Cloud Code for it might\nbe saying there's like autonomous\nmonitoring happen happening um and then\nthat could feed into say generating some\nkind of like issue or or feature request\nthat could then just feed into um an\nagent that would then go through to make\nthe pull request and then there's sort\nof a pull review um\nalready happening and then maybe you're\njust reviewing that before you actually\nmerge. So I think there there are other\nways to automate the whole software\ndevelopment life cycle um uh in a\nbrownfield project but I think this\nparticular pattern maybe without a lot\nof testing within your project and\nbuilding like customizing it for your\nproject it's probably more suited\ntowards brand new applications.\nHave you built any greenfield apps that\nlike I don't know like an internal\ntooling or anything like that that\nyou've been using like not just a demo\nso like\nYeah um to be frank I can't really like\ntalk to like internal tooling too much\nbut\na good anecdote to this was\num\nlike a lot of the uh the new and fun\nstuff that you see in Cloud Code will\nwill like um\nuh when I'm speaking to the team and\nworking with them on on stuff\nuse a lot of the lessons from this per\nse like in in like even just general\nhands-on\nuh\nCloud Code usage the way that they\nprompt you know the main\nuh model to to spin up a sub agent let's\nsay and and and go after something or as\nkind of Andrew said right in kind of\nmonitoring and bug fixing loops like,\nyou know, when generating effects, like,\nshould you have a separate evaluator and\na generator go after the same thing? So,\na lot of these principles apply. Um, is\nit like, you know, one for one this?\nMaybe not, but it's like taking the good\nbits of this or whatever you think is is\nkind of applicable to a certain space\nand field and then kind of running with\nit in your own way.\nHi.\nWhen you say you're needing the traces,\nis that literally just like the raw\noutput or is there something more\nspecific you've prompted it to like\nwrite this to file, these are the sorts\nof things I care about and I want to\nsee? No, you got to read the whole\nthing. Read the whole thing. I do think\nit's like a a really important skill\nwhen building agents in general is to\nlike\nempathize as much with the model. Um,\nthis was like there's an interesting\nuh\nanecdote which we used when we're\nbuilding, for example, the agent harness\nfor Claude for Chrome, um, which is our\nkind of browser use thing.\nUm,\nand\nwe would run this like experiment where\nlike imagine if, you know, you were\ntrying to navigate a web page and click\naround where like, you know, you're\neffectively doing with your eyes closed\nand like every 10 seconds you just\nopened it to see like a static page and\nthen close it again and then have to\nlike do things. Um, and like really\nputting yourself in the shoes of the\nmodel um,\nis kind of like there's kind of\nempathetic skill set which you need to\ndevelop um, and the only way to really\ndo that is to like spend as much time\nwith these models, but also\nyeah, reading through line by line being\nlike, oh, why did it think this? Oh, I\ncan kind of see why it did that. And\nthen kind of adjusting the way you\ninstruct it next time to to do better.\nUm,\nbut that's why I think Claude for Chrome\nis very good was just really just like\nspending a lot of time as a team uh,\nclosing our eyes and trying to navigate\nweb pages, for example. So,\num,\nyeah.\nMhm.\nYeah, and I think\nthen actually taking those learnings and\nputting them into say your prompt\ntemplates or your Claude.ai MD or\nbuilding a skill or\ngenerally understanding how to sort of\navoid that type of behavior in the\nfuture. I know Claude code now has auto\nmemory for sessions as well, so it's\nsort of constantly memorizing little\nthings as it goes. Um but yeah, you can\nlearn quite quickly from reading some\ntraces like where things might be going\nwrong.\nCool.\nShould we wrap up there? Um I think we\nhave a few minutes left, but we'll be\naround in general in case you guys want\nto ask any questions or just chat. But\notherwise, thanks for coming down.\nThat's the session for today.\nThank you.",
  "transcript_chars": 72640,
  "ingested_at": "2026-05-21T19:20:11.224802+00:00",
  "source": "retry-no-transcript",
  "yt_meta": {
    "view_count": 38458,
    "like_count": 746,
    "channel_id": "UCLKPca3kwwd-B59HNr-_lvA"
  }
}