{
  "video_id": "vtugjs2chdA",
  "channel_slug": "sequoiacapital",
  "channel_handle": "sequoiacapital",
  "title": "Context Engineering Our Way to Long-Horizon Agents: LangChain’s Harrison Chase",
  "duration_seconds": 2388.0,
  "url": "https://www.youtube.com/watch?v=vtugjs2chdA",
  "upload_date": "",
  "transcript": "People use traces from the start to just\ntell what's going on under the hood. And\nit's way more impactful in agents than\nin single LLM applications because in\nsingle LM applications, you get some bad\nresponse from the LLM. You know exactly\nwhat your prompt is. You know exactly\nwhat the context that goes in is because\nthat's determined by code and then you\nget something out. In agents, they're\nrunning and and and repeating and so you\ndon't actually know what the context at\nstep 14 will be because there's 13 steps\nbefore that that could pull arbitrary\nthings in. So like what exactly is\neverything's context engineering?\nContext engineering is such a good term.\nI wish I came up with that term. Like it\nactually really describes like\neverything we've done at Langchain um\nwithout knowing that that term existed.\nBut like traces just like tell you\nwhat's in your context and that's so\nimportant.\nWelcome to training data. Harrison, you\nare our very first guest on training\ndata and the AI space has moved so\nquickly uh in the 18 months or so since\nwe originally interviewed you and so I'm\ndelighted to get you on the show today.\nUm topics of the moment. I think there's\nnobody better than you to talk about\nsome of these topics. We're going to\ntalk first about long horizon agents and\nagent harnesses. Pat and I had this blog\npost on this yesterday. I know this is\nsomething that you are deeply um fluent\nin. And then we're going to talk about\nwhat's the difference between building\nlong horizon agents versus building\nsoftware um and the role that you see\nLang Chain playing in that ecosystem.\nAnd then finally, I just want to chat\nwith you about the future. I think you\nsingle-handedly, you know, kind of saw\nthe agent opportunity I think before\nanybody. You know, we were back in the\nGP3 days and um I think you see the\nfuture for what's happening with agents\nand so I'm just excited to chat with you\nopen-endedly about the future as well.\nI am really excited as well. Thank you\nguys for having me back. It's quite an\nhonor. I'll tell my mom again and that\nI'm back on the line.\nWonderful. Okay, let's start with long\nhorizon agents.\nYes, that was a great term. You guys\nwrote a great article.\nSony's good at naming things.\nWe're not going to get into the the\nbackstory there. Um, what do you think?\nWhat do you agree with? What do you\ndisagree with?\nI mean, uh, I agree that they're\nstarting to finally work. I think like\nthe idea of running an LLM in a loop and\njust having it go was, uh, was always\nthe idea of agents from the start. uh\nautog was basically this then this is\nwhy it took off and and captured so many\npeople's imagination because it was just\nan LLM running a loop completely\ndeciding what to do. The issue is the\nmodels weren't really good enough and\nthe scaffolding and harnesses around\nthem weren't really good enough and I\nthink the models got better. We learned\nmore about what makes a good harness\nover the past few years and now they\nstart to like really really work and you\nsee this in coding first and I think\nthat's the domain where they're taking\noff the most and that's spreading to\nother domains but you can give a task to\nan agent and you still need to\ncommunicate to it what you want it to do\nand it needs to have the right tools and\nall of that but it can actually operate\nfor longer and longer periods of time\nand so that yeah the long horizons kind\nof like uh uh framing of it I think is\nreally really apt and really really\ngood.\nAwesome. Um, what are your favorite\nexamples of long horizon agents and I\nguess what shapes you see them taking?\nSo, coding is the place where there's\nthe most. I think that's the one that I\nprobably use. Yeah, that's the one that\nI use the most. Um, adjacent to that, I\nthink like really good ones are AI\nSRRES. So, Traversal I think is a\nSequoia company and and they have an AI\nSR that that that operates over longer\ntime horizons. research in general and\nI'd call like AISR kind of research like\nthey're taking a incident and they're\ngoing and digging through logs like\nresearch in general is a really really\ngood task because it ends up producing\nlike a first draft of something and the\nissue with agents is they aren't like\nreliable to 99 of reliability but they\ncan do a ton of work and and more and\nmore work over longer time horizons. So\nif you can find these framings where\nthey run for a long period of time but\nproduce like a first draft of something\nthose to me are like the killer\napplications of long horizon agents\nright now. So like I coding coding is an\nexample of that like coding you usually\nput up a PR you don't directly push to\nprod unless you're vibe coding which is\nalso starting to get better and better.\nUh AISR usually surface it to a human\nwho comes in and then reviews it. Report\ngeneration you don't send it out to all\nof your followers right away. You look\nat it you edit it. It creates like a\nfirst draft of something. So we see this\nin uh we see this in finance a bunch.\nThis is this is a huge research\nopportunity. Customer support. We see a\nlot of things pivoting from kind of like\nthe initial the initial customer support\nwas like first-line response like\nsomeone messages you just respond really\nquickly and there's still that and and\nthat's going great but now there's\nexamples K is a great example of this\nwhere it's like humans and AI working\ntogether when the first line fails you\nescalate to a human you don't just have\nthe human handle it you have this long\nhorizon agent run in the background\nproduce a report of everything that\nhappened and then hand it off to the to\nthe to the agent there um to the human\nagent there agent starts to get\nconfusing in customer support.\nUm, so so I think the killer use case of\nall of these is places where you have\nlike this first draft type of concept.\nAnd then how much of the uh the why now\ndo you think is the models themselves\nare just so good versus people are doing\nreally smart things on the harness side?\nAnd maybe even before we get to that,\ncan you say a word for for our listeners\non like you know how you frame the the\nharness versus the model in terms of the\nactual composition of an agent?\nYeah. So and and I'll maybe I'll maybe\nbring in like framework as well because\nI think early on I mean that's how we\ndescribe lang chain as that's what lang\nchain is it's an agent framework and now\nand now we have deep agents which I'd\ncall an agent harness and we get asked\nabout what's the difference so a model\nis obviously the LLM tokens messages in\nmessages out um the framework would be\nabstractions around that so making it\neasy to switch between models uh adding\nabstractions for other things like tools\nand vector stores and memory and things\nlike that but pretty like unopinion\nabout what goes in there more values\nare more like batteries included. So\nwhen we talk about deep agents we're\ntalking about we we actually give it a\nplanning tool by default. So it has a\ntool that that that comes built into the\nharness. That's pretty opinionated that\nlike this is the right way to do things.\nUm we do compaction. So you have these\nlong horizon agents. They're running for\nlong periods of time. Context windows\nare larger but they're still not\ninfinite. And so at some point you need\nto compact that. How do you do that?\nThere's a lot of research going on there\nright now. Uh, one of the other sets of\ntools that we and a lot of people are\ngiving to these agents are tools for\ninteracting with the file system whether\ndirectly or via bash and and\nthis is it's kind of tough to separate\nfrom the models because the models are\nbeing trained on a lot of this data as\nwell. And so there's this kind of\nevolution between like I don't know if\nwe could have known that like these file\nsystem based harnesses are the best\nthing at like if if we fast if we go\nback two years ago I don't think we\ncould have known that because models\nweren't really being trained on that as\nmuch as they are now and so they're kind\nof evolving together. So so I think it's\nlike a combination of things. It's the\nmodels absolutely are getting better.\nReasoning models are helping helping a\nlot. Um, but it's also the fact that\nwe're figuring out all these primitives\naround compaction and uh and planning\nand and these file system tools being\nreally useful. And so it's I I do think\nit's a combination of both.\nUm I remember in in that very first\nepisode we did together, you know, you\ndescribed\nlaying graph I think as the almost the\ncognitive framework.\nYeah.\nUh of of the of the agent. Is that the\nright way to think about what the the\nharness is? Uh yeah, I think I think\nthat's right. Yeah, so we we build we\nbuild deep agents on top of Langraph.\nIt's one particular kind of like\nlangraph instance. It's very\nopinionated. It's more general purpose.\nUm and so I think early on we talked\nabout general purpose architectures and\nmore specific architectures. And what\nwe've seen is that a lot of the\nspecificity for tasks previously that\nmight have been in langraph because you\nneeded to put more structure on the\nmodels. Now that specificity is moving\ninto the tools and the instructions. So\nthere's still the same level of\ncomplexity. It's just in natural\nlanguage. And so prompting and editing\nthose prompts and and and automatic\nmaybe automatically updating those is\nbecoming a part, but the harness is\nremaining a little bit more fixed.\nUm\nwhat's the hardest thing to get right on\nthe harness side? And do you think, you\nknow, do you think individual companies\ncan actually excel at the harness\nengineering side of things? Who do you\nadmire there? Uh I think a lot of the\ncompanies that are doing the best\nharness engineering are coding\ncompanies. Honestly, I think that's the\nplace where it's taken off a bunch. I\nmean you look at cloud code. I would\nargue a big reason for the popularity of\ncloud code is the harness itself. Um\ndoes that by the way imply that\nharnesses are better built by foundation\nmodel companies than by third party\nstartups?\nI I I don't know. Um I so the next\ncompany I was going to mention is\nFactory which is another coding company\nand and I think you look at at the\nharness they've done there. AMP um is\nanother coding company has a has a has a\nreally good harness. Um I think there's\npros and cons. They there definitely is\nsome aspect of the harness being tied to\na model. Um and and maybe not just not a\nspecific model but a family of models.\nSo like all cloud models like anthropic\nfine-tunes on some specific tools.\nOpenAI fine-tunes on different ones. So\nlike I think probably probably probably\nprobably probably when we were doing\nthis last time we maybe talked about how\nprompts need to be different for one\nmodel versus another. Hardnesses also\nneed to be slightly different for one\nfamily of of things versus the other\nbut there are similarities. Uh all of\nthem use the file system in in in some\nsense.\nUm so I think this is I I I I actually\ndon't know the answer to that. It's a\nreally interesting uh thing. Um we see\nthat a lot of the coding com a lot\neveryone who's building a coding company\nis basically building their own harness\nright now. Yeah.\nUm,\nand there's all these leaderboards and\nyou can see it's actually kind of\ninteresting if you go to terminal bench\n2, which I think is probably the one of\nthe more kind of like popular coding\nbenchmarks right now. You can actually\nsee they they have like the the agent\nharness and then the model and so you\ncan see the variation in performance and\nclaude code is not at the top of that.\nUm, so there's there's differences but I\nthink it\nit doesn't necessarily mean that the\nmodel labs are better at it. It just\nmeans that you have to understand how\nthe models work and people who look at\nthe at what makes a harness tick around\nthe model can can get some performance\ngains there.\nYeah.\nWhat do you think goes into, you know,\nmaking the harness tick? Like what do\nyou think that the guys at the top of\nthe leaderboard are doing exceptionally\nwell?\nI think part of it is definitely\nunderstanding like what tools the models\nuh trained on. So uh I think OpenAI\ntrains really heavily on Bash. I think\nAnthropic has an explicit kind of like\nfile editing tools and so I think\nleaning into that is part of it. Um\ncompaction is becoming more and more of\na thing. Uh so especially as you start\ndoing longer horizon tasks like you\nstart to fill up the context window and\nso what do you do there is a really big\nquestion and there's a bunch of\nstrategies for for kind of like\napproaching that. I'd argue that's part\nof a harness. I mean, so all of these\nharnesses also this is where like skills\nand MCPs and sub aents start to come\ninto play as well and you can use those\nin like different ways and and I don't\nknow how I I don't think a ton of like\nskills or sub agents are trained into\nthe models yet like those are still\npretty new.\nYeah.\nUm and and so like one one of the things\nthat we see in our harness is like when\nyou have a sub agent the the the the\nmain model needs to communicate with it\nlike well it needs to it needs to give\nit all the appropriate information. it\nneeds to let the sub aent know that it\nneeds to like give it its final response\nout. So like we would see some failure\nmodes where the sub aent because\nbasically what happens is you kick off\nthe sub agent and then only the final\nresponse is passed back to the main\nagent. And so we'd see some failure\nmodes where the sub agent would do a\nbunch of work and then it would be\nbasically like look at my work above and\nthen you know pass that back to the main\nagent and it can't see it and it's like\nwhat are you talking about? And so like\nthat type of like prompting to get these\npieces to work together is a big part of\nit. So like skills, sub agents, MCP,\nthere there are like prompts in all of\nthese harnesses that make them work well\nor don't make work them work well. And\nthey're they're hundreds of of lines\nlong if you look at some of the the the\nones that are out there.\nCan I ask you a question on how this has\nevolved and um\nsince you've always been\nreally kind of on the bleeding edge of\nwhat are people doing around the models\nto make them work in the real world,\nright? If we think about in our\nsimplistic view on like what the big\ninflection points over the last five\nyears have been feels like there was a\nbig inflection point around pre-training\nwhen chatbd came out. Feels like there\nwas a big inflection point around\nreasoning when 01 came out. Feels like\njust recently there's been a third big\ninflection point around these long\nhorizon agents with cloud code and opus\n45. Um in your world the world of all\nthe stuff around the models that that\nmakes them work in the real world. Would\nyou have a different set of inflection\npoints? Like what have the major changes\nbeen? I remember we talked about\ncognitive architectures a couple years\nago and now we're talking about\nframeworks and agent harnesses. Like\nwhat have the major leaps in\num sort of the design around the model?\nYeah. What have they been?\nSo I think there's maybe like three eras\nI would say. I'd say like early on and\nthis is when Langchain was just started\nlike these were still the raw like text\nin text out like not even chatbased\nmodels. And so they didn't have any of\nthe tool calling. They didn't have\nany content blocks, any reasoning at\nall. They were really just like\nreally really\nbasic. Uh and so the the things that\npeople were doing were mostly like\nsingle prompts or like chains. Um and\nand it wasn't even possible to do\nanything like that complicated. Then a\nlot of the model labs started training\nin a lot of like the tool calling into\nthe models and they got really good at\nkind or they tried to make them good at\nlike thinking and planning and they\nstill weren't they still weren't good\nyet. they sort of weren't good as they\nas they are today. Um, but they were\ngood enough to like\ndecide what to do. And this is where\nlike the custom cognitive architectures\nwould come in more into play because\nyou'd ask it explicitly like what do I\ndo here? But it was like a very like\npoint in time and then you go down this\nbranch and then like what do I do here?\nAnd maybe there's a loop and and there\nstarted to be some loops but it's still\na little bit more kind of like\nscaffolding around it.\nAnd then there was an inflection point\nand I don't know where exactly that was.\nI would say I think we noticed it\nprobably in like June July of this year\nwhere we saw Cloud Code taking off, deep\nresearch taking off, Manis taking off\nand these all use the same architecture\nunder the hood of just the the LLM\nrunning in the loop but like cleverly\nlike a lot of a lot of hardness is\ncontext engineering like everything\naround contraction context engineering\nsub agents context skills context\nengineering. So we basically saw them\nusing the same core algorithm but making\njust like improvements on context\nengineering and we're like oh that's\ninteresting that's pretty different than\nbefore. And so that's when we started\nworking on deep agents.\nI think for a lot of people in the\ncoding community, I think probably like\nOpus 4 5 was when they started to like\nreally feel this. It might have also\njust coincided with winter break when\neveryone went home and started using\nclaude code and realiz how good it was.\nUm but I think like around like\nNovember, December like I think there's\nhas been this like\nat least I I sense a pretty big like\nvibe shift in like people just like yeah\nyou throw hard problems at these things\nand you get long horizon agents. And so\nI don't know whether it was early 2025\nor late 2025, but at some point the\nmodels got good enough and and that's\nwhen we moved from like scaffolds to to\nharnesses.\nAnd what's next on this arc?\nI I wish I could tell you. Um, I mean I\ndo think that like this algorithm of\njust running the LM in a loop and\nletting it orchestrate its own uh let\nletting it really choose what to pull\ninto context and and and doing stuff\nthere that is like it's so simple and so\ngeneral purpose like I mean that was the\ncore idea of agents and all along and\nand and we're finally there.\nUm if you look at some of the manual\nscaffolding like maybe some of that goes\naway. So like compaction is still very\nmanual like the the harness author\ndecides what to do with it. Anthropic\nhas some interesting things where they\nlet the model decide like when to\ncompact things. We don't really see a\nton of people using that. Maybe that'll\nbe a part that's next. Uh part of what\nwe're really interested in is memory as\nwell. If you think about memory in the\ncontext of this, that's also context\nengineering, right? It's context\nengineering over longer time horizons\nand it's a slightly different set of\ncontexts, but it's still giving that to\nthe to the LLM. And I think uh I think\nlike the the core algorithm is is is\npretty like it's pretty simple. It's run\nthe LLM in a loop and we're finally\nthere and it kind of works. And so I\nthink there'll be a bunch of like\ncontext engineering tricks around it.\nAnd maybe some of that is giving the\ncontext engineering actually to the LLM\nlike letting like the anthropic thing.\nMaybe some of that is just pulling in\nnew types of context. The models will\nwill probably get better. They'll\nprobably I mean they'll probably get\nbetter and better at these types of\nlonger horizon tasks. that'll be great\nas well. Um, one of the big questions on\nmy mind it so a lot of these harnesses\nthat we see are very coding specific.\nYeah. Um, and that's where we first\nstarted to really see these long horizon\nuh, agents take off and even for\nnon-coding tasks I think you can make an\nargument that like writing code is\nreally useful and uh, can be general\npurpose.\nI was going to ask you are coding agents\nis that a subcategory or are coding\nagents just agents? Meaning the job of\nan agent is to figure out how to get a\ncomputer to do useful stuff and code is\na pretty good way to get a computer to\ndo useful stuff.\nI don't know. Um this is one of the big\nso I I I very very strongly believe that\nlike right now if you're building a long\nhorizon agent, you need to give it\naccess to a file system. Like there's so\nmany things you can do with a file\nsystem in terms of context management.\nLike when we talk about compaction, one\nstrategy is to summarize but put all the\nmessages in the file system so that if\nit needs to look it up, it can. Uh,\nanother strategy is when you have like\nbig tool call results, don't pass it all\nto the model. Put it in the file system\nand let it looking let it look it up.\nNow you can do all of that without like\na real file system actually without\nletting it write code. So we have a\nconcept of like a virtual file system\nwhere it's just backed by Postgress or\nsomething like that and it's more\nscalable, but there are obviously things\nyou can do with code that you can't do\nwith a virtual file system. You can't\nrun code in a virtual file system. Um,\nso like writing scripts is like really\nuseful for that.\nYeah. Um, and I think a coding agent can\nbe general purpose, but I don't know if\nthat means that today's coding agents\nare, if that makes sense. Because I\nthink a lot of the coding agents today\nare pretty optimized for coding tasks.\nAnd so I think it's possible that a\ngeneral purpose agent is a coding agent,\nbut I don't know if like the reverse is\ntrue if if if that kind of like makes\nsense.\nYeah. Yeah. Yeah.\nWe're thinking about that a lot as well.\nAre all agents coding agents?\nYeah. That's that's one of the biggest\nthings that we're thinking about right\nnow.\nYeah. Maybe can we transition into\ntalking about what goes into building a\nlong horizon agent versus building\nsoftware? Um can you maybe describe the\nsoftware development stack for for 1.0\ncode development and um and what's\ndifferent now? And I thought you had a\nreally good X article on this. Maybe\nmaybe just summarize the the punch line.\nI've been I to think about this a bunch\nbecause we like to say that build and I\nthink a lot of people would agree that\nlike building agents is different than\nbuilding software. But like what exactly\nis different? Cuz I think it's it's easy\nand lazy to say that it's different, but\nwhat actually is different? These might\nsound obvious, but hopefully that's good\nand they're not controversial. But like\num when you're building software, all of\nthe logic is in the code in the software\nand you can see it there. When you're\nbuilding an agent, the logic for how\nyour applications works is not all in\nthe code. A large part of it comes from\nthe model. And so what this means is\nthat you can't just look at the code and\ntell exactly what the agent would do in\na specific scenario. you actually have\nto run it. And so and so and so what\ndoes that mean? And and I think that's\nthe biggest difference by the way like\nwe're introducing like these\nnondeterministic systems into it and\nit's a black box and it lives outside\nand I think all that that's that's the\nbiggest difference. So what exactly does\nthat mean? I think like one thing that\nthat means is that in order to tell what\nthe application is actually doing, you\ncan't look at the code. You have to look\nat actually what it does in in real\nlife. Um, and so I think one of the the\none of the things that one of the things\nthat we do that is most popular is\nLangmith. One of the core parts of that\nis tracing. Why are traces so popular?\nBecause they tell you exactly what goes\non inside your agent at every step. Um,\nand it's different than software traces\nwhere in software you kind of have your\nsystem over here and it emits a bunch of\nlike uh stuff and you you know you look\nat it when maybe there's some errors but\nyou don't need like everything and and\nyou usually only turn that on when you\nput it in production because if it's\nlocal you just put a breakpoint or\nsomething like that. in agents like\npeople use traces from the start to just\ntell what's going on under the hood. And\nand it's way more impactful in agents\nthan in single LLM applications because\nin single LM applications you get some\nbad response from the LLM. You know\nexactly what your prompt is. You know\nexactly what the context that goes in is\nbecause that's determined by code and\nthen you get something out. In agents\nthey're running and and and repeating\nand so you don't actually know what the\ncontext at step 14 will be because\nthere's 13 steps before that that could\npull arbitrary things in. So, like what\nexactly is everything's context\nengineering. Context engineering is such\na good term. I wish I came up with that\nterm. Like it actually really describes\nlike everything we've done at Langchain\num without knowing that that term\nexisted. But like traces just like tell\nyou what's in your context. And that's\nso important. And so then and so like\nwhat does that mean? That means that the\nthe source of truth for software is in\ncode. In for agents it's a combination\nnow of code and and and traces are where\nyou can see the source of truth.\ntechnically in you know all those\nmillions of parameters but like you\ncan't really do anything with that. So\nnow so now that means that traces become\na place where you start to think about\ntesting because now you can't you can\ntest you can test some parts still of of\nthe harness and and and you can do some\nunit testing offline but like in order\nto get the the what the test cases are\nyou probably want to use the traces to\nconstruct that. You probably want to be\ntesting online. That's probably more\nimportant in agents than it is in\nsoftware is online testing because\nbehavior doesn't emerge until until it's\nactually being used with with real world\ninputs. Um, we see traces becoming a\npoint of collaboration for teams because\nif something goes wrong, it's not, oh,\nlet's go look at the code in GitHub.\nIt's let's go look at the trace. We see\nthis in our open source as well when\npeople are being like, hey, deep agents\nlike went off the rails here. What\nhappened? Our response is like send us a\nlang trace. Like we can't really help\nyou debug if it's if it's not that.\nPreviously would be like, show me the\ncode, right? So there's think there's a\ntransition there. And then I think the\nother thing that's and and so that was\nthe blog post that I wrote on next which\ngot a lot of good feedback on. I'm still\nkind of figuring out how to like phrase\nit but I think that's that's a big part\nof it. Um the other thing which I'm\nstill trying to think through as well is\nI think building agents is more\niterative. And we used to say that and I\nwould kind of roll my eyes because\nbuilding software is iterative as well,\nright? You you ship it, you get feedback\nand it's it's constant iteration. That's\nlike what it is. I think the difference\nis that in software you're you're you're\nkind of like iterating based on what you\nwant the software to do. Like you have\nsome idea, you ship it, you get\nfeedback. Oh, maybe this, you know,\nbutton is confusing. Maybe this uh maybe\nusers actually want to do X instead of\nY, but you know what the software does\nbefore you ship it. With agents, you\ndon't know what the agent does before\nyou ship it. You have an idea, but you\ndon't really know what it does before\nyou ship it. And so I think there's way\nmore iteration involved in order to get\nit like accurate, get it like right and\npassing like conceptual unit tests\nbasically.\nUm, and\nbuilding upon that like this is actually\nwhy I think memory is really important\nas well. Um, because memor is like\nlearning from those interactions. And so\nif now you have a process that's like\nway more iterative. And so now you have\nto like it's it's way harder to build as\na developer because I have to like\nchange the system prompt like way more\nthan I would have to change code in\norder to get it just perform like\ncorrectly.\nYeah.\nSo that's where memory comes in because\nif there's a way where the system can\nkind of like learn by itself that cuts\ndown the iteration that you have to do\nas a developer and makes it easier to\nbuild these types of agents. So that's\nanother kind of like angle that I I like\nI absolutely think agents are different\nthan building software. I think it's\nalso a little cliche to say that and so\nI've tried to think about what exactly\nis different and those are like the two\nthings that I've kind of come up with.\nWell, and I'm curious on that too. Um,\none of the questions there's a big\npublic market debate right now is are\nthe existing software companies going to\nmake it. And if you analogize to when\non-prem software went to cloud, very few\nactually did make it because it turned\nout that building cloud software was\nactually quite different than building\non-prim. And since you're in the middle\nof kind of how people are building with\nAI, um what's your take on not\nnecessarily the public market question,\nbut how different is it? Like do you see\nhave you seen a lot of people who kind\nof like were good at building software\nthe old way and now they're good at\nbuilding software the new way? Or is it\nmore just you either grow up building it\nthe new way or you never get it? Like do\nyou think people can make the leap? a\nlot of young founders out there right\nnow which which makes me think that\ncertainly it seems like the younger\npeople without a lot of preconceived\nnotions on how to build software you\nknow have the blank slate that has\nallowed them to like pick up on a lot of\nthis stuff. I do think we we have\nconsistently heard that a lot of the\npeople who are on these agent\nengineering teams are more junior\ndevelopers uh more junior builders even\num\nwho yeah don't have those preconceived\nnotions. Our applied AI team internally\ndefinitely skews on the on the younger\nside.\nUm I do think I mean in terms of kind of\nlike I think there's like a there's like\na person aspect to this. There's also\nlike a company aspect to this. Like I do\nthink that like data is still really\nreally really valuable. Um\nI think when you think about this\nharness basically there's like if if\nharnesses become I by the way I I don't\nthink that most people will build their\nown harness in the long run because it's\nactually way harder than than building a\nframework. And so I think they'll uh use\nuh a harness from us or from someone\nelse. And so if you think about what\ngoes into that it's like the prompt and\nthe instructions and then the tools that\nit's connected to. And I think one thing\nthat this is more at the company level\nnow, but like one thing that existing\ncompanies have is all the data and all\nthe APIs if if you've done a good job at\nthat then I think it will actually be\npretty easy to plug those in and get\nreal value out of things. Um we were\ntalking to someone in the finance space\nand and they are saying yeah like the\nvalue of data is just going up and up\nand up and up. So if you're a previous\nuh software vendor and you have this\ndata that is valuable like you should be\nable to expose it to agents and get a\nlot of value out of that.\nYeah. The other part of it though is the\ninstructions on what to do with that\ndata and that's probably like more net\nnew um in terms of like how to use that\ndata. That's probably you probably had\nsome ideas about that as a software\nvendor but you didn't kind of like\nconsolidate it. You didn't have it\nbecause that was something that humans\nwould still do. Like a lot of what\nagents are doing what humans would still\ndo. So you'd give them the tools to do\nit but you wouldn't have tried to like\nautomate that or you wouldn't have\nsuccessfully automated it before kind of\nlike agents. And so that part I think is\nis is newer and we're also seeing a lot\nof demand like I think a lot of the a\nlot of the vertical startups um rogo is\na great example of someone who has\nexperience in finance and is bringing\nthat knowledge to agents and the reason\nthat's kind of like effective is because\na lot of the agents are driven by by\nknowledge and and and and and not like\nworld knowledge but like knowledge on\nhow to do specific patterns. Um so kind\nof yeah I think there's like are the\npeople who are building software the\nright people to build agents. Um I think\nwe saw a lot of really senior developers\nadopt agentic coding and so I think it\nit it's a mindset thing but like yeah\nthere there is maybe a younger skew\nthere. Um and then and then for\ncompanies depends on the data.\nYeah.\nEven Pat's on on cloud code. So\neven old guys can get it.\nSonia got me on there.\nUm okay. Okay. So, it seems like the\ntrace is a core artifact you think in\nkind of this new world of of agent\ndevelopment and it's something that\nlinksmith helps a lot with. What other\ncore artifacts do you think are there?\nAnd specifically, I'm wondering about\neval.\nYeah, I think um\nmaybe artifact is the wrong word.\nComponents.\nComponent. Yeah, I mean I I I think one\nother thing that is different between\nbuilding software and and building\nagents is that to evaluate software you\ncould pretty reliably you you could rely\non tests and assertions of of things\nprogrammatically.\nUm with agents a lot of what they're\ndoing is things that humans would do. So\nin order to judge them you need to bring\nhuman judgment into that. And that's\nanother thing that we try to do in\nLingsmith is how can you bring you've\ngot these traces how can you bring human\njudgment to them? And so that like one\nobvious way to do that is to bring\nhumans into the equation. Um and so we\nsee data labeling startups doing really\nwell. Uh we have a concept of annotation\ncues in Langmith to bring people in\nthere. And so that actual like actual\nhuman judgment is is a big part of it.\nAnd this is humans annotating the actual\ntrace. So like ah the agent did this and\nthis and this and that was good or bad.\nYeah. Yeah. Um saying and and sometimes\ngiving like natural language feedback on\nit like this is good, this is bad,\nshould have done this. sometimes just\nlike correcting it like actually like\nlaying out what the what the uh correct\nsteps were kind of depends on the use\ncase and it's probably different for\nmodel companies doing RL than it is for\nfor agent companies building building\nagents. Yeah.\nUm but it's bringing that human judgment\nto it. Um but then another thing we see\nis trying to build proxies for this\nhuman judgment. And this is where LLM as\na judge type things come in where you\ncan run an LLM or something else that\nyou know has some semblance of human\njudgment in it to grade the the thing\nthat requires human judgment. Um, and so\none of the things that we think a lot\nabout is how to make building these\nelements as judges easy because a big\npart of them is making sure that they're\naligned with your human judgment and\nhuman preferences. And so and because if\nthey're not, you know, then your then\nyour greater is just bad. And so we have\na we have a concept in Langmith called\nalign evals. Um, where a human goes in\nlabels some traces and then that that\nbuilds an LLM as a judge that that kind\nof like is calibrated against those\ntraces. Um because a bit yeah a big part\nof it is bringing this human judgment\nand you just want to make sure that if\nyou're bringing a proxy of it it's it's\nwell calibrated.\nInteresting. I remember when we first\ngot into business with you we were\nemailing about LLM as judge. Is it a\nviable idea or not? So it seems like\nit's come a long way.\nOkay. So there's there's a few different\naspects of LM as a judge, right? There's\nlike the immediate like so what most\npeople use them for in eval is like\ntaking this trace and give it a score of\nlike one to to zero or or 0 to 10 or\nsomething like that. And yeah, I think\nthat's valu uh viable and people are\ndoing that. They're doing it offline.\nThey're also doing it online because\nsome of these judgments you don't need\nground truth for. But I think the other\narea where this comes into is uh I mean\nyou you kind of see this in the coding\nagents themselves like the coding agents\nwill they'll work up until something\nthen they hit an error and they get an\nerror and then they have to correct\nthere and so they're kind of judging\ntheir previous work. And so and and we\nalso see this in memory like a big part\nof memory is like reflecting on traces\nand then updating something. And so like\ncan LLM reflect on traces that are\neither like their own or their own from\na previous session or someone? Yeah,\nabsolutely. I think they can. We see\nthis all across evals and just like\nerror correcting and and memory. It's\nall kind of the same thing.\nI see. And then maybe Okay, so you have\nall this you you have all the traces.\nYep.\nYou have the evals.\nYep. Um I think the natural question\nthat comes to mind for me is is the EVEL\nlike a reward signal for reinforcement\nlearning or is it a feedback mechanism\nfor you know a human engineer to improve\nthe harness\nor for agent engineers to improve the\nharness because everyone's no one's no\none's coding manual anymore. They're all\nusing these. So yeah, one big thing that\nwe've seen is like um we we have a\nLangmith MCP and we have Langmith fetch\nwhich is a CLI because coding agents are\nactually great at using CLIs. um you\ngive that to an agent and it can pull\ndown traces and diagnose what went wrong\nand then and then it brings those traces\ninto the codebase where it can then fix\nit. That's absolutely a pattern that we\nare seeing and and we really really\nreally want to support that pattern.\nCrazy. Yeah, I know.\nAnd it's good.\nYeah. Yeah. Yeah. Yeah. It's good. Like\nit it Yeah, it um and and so we see I'm\nprobably more bullish on that than on\nkind of like reinforcement learning at\nleast for like the agent app kind of\nlike companies right now. That seems\nlike real recursive self-improvement\nthough.\nYeah, I think I think uh again there's\nstill a human in the loop. So like uh\nback to the point around around like\nthings are good when you can do\nsomething as a first draft like it it\nchanges the prompt and then the human\nreviews it and like it it keeps it on\nthe rails. But like um I absolutely so\nso one of the things we launched was\nLangmith agent builder which is a no\ncode way to build agents. One of the\ncool things that we have in there is\nmemory. Um, and so right now the way\nthat memory works is when you interact\nwith an agent, so it's not in the\nbackground yet. It's not like pulling\ndown its traces, but when you interact\nwith the agent, if you say, \"Oh, you\nshould instead of X, you should have\ndone Y,\" it will go to its own\ninstructions, which are just files, and\nit will edit those files. So then in the\nfuture, and so that's also kind of like\na a version of this. One thing we do\nwant to add is like the thing that runs\nevery night, looks at all the traces for\nthe day, updates its own instructions,\nand so\nthe dreaming thing.\nYeah. Yeah. Sleep time compute. Um,\nyeah,\nsleep time compute. Is that what it's\ncalled?\nThat's a term. Yeah, I think Leta came\nup with that. It's It's a great term.\nYeah,\nthat is good.\nLove it.\nAwesome. Okay, let's talk more about the\nfuture. Um, what are you what are you\nmost excited about? Sounds like you was\ntalking a lot about memory here.\nI like memory a bunch. Yeah, I mean I\nthink asking the agents to improve\nthemselves is I mean I think very very\ncool and can be useful in a lot of\nsituations. Not useful in all situations\nby the way. like if I'm chatting so so\nchat GPT added memory I don't actually\nreally use that feature that much and I\ndon't think it's created any more\nstickiness for me to use the product or\nanything like that and I think part of\nthe reason is when I go to chat GPT I do\nlike\neverything's a one-off thing like I\ndon't really repeat myself that much I\nhave like I'm asking about software I'm\nasking about food trips like everything\num in agent builder that you you build\nkind of like specific workflows for\nspecific things so I have an email agent\nUm, and I actually\nI know it's been emailing me for two\nyears.\nWell, so, okay, so I had an email agent\noutside of agent builder and it had this\nlike memory as part of it. We then built\nagent builder and I wanted to move it\ninto it and it didn't have all of my\nmemories and that was a big even though\nit had the same starter prompt and the\nsame tools and that was actually a I\nstill haven't fully switched over\nbecause it kind of sucks now compared to\nwhat it was before like the compared to\nthe other one and I if I just interact\nwith it then it will get better and it\nwill stop sucking but like that's where\nmemory I think can be like a real moat\nand I absolutely think that we're at a\npoint right now where LLM can look at\ntraces and change things about their\ncode. Um, and I think the question then\nbecomes how do you do that in a way\nthat's uh safe and acceptable to to\nusers. Um, but I I I think that's\nabsolutely uh something that we'll see\nmore of for specific scenarios, not all\nof them. Like I still don't know if this\nwould be useful in chat GPT in this form\nat least.\nYeah.\nHow do you think the UI around working\nwith long horizon agents will evolve?\nI think there probably needs to be like\na sync mode and an async mode. So long\nhorizon agents running for a long time,\nprobably default would be some sort of\nlike async way to manage them. Like\nyou're if if it runs for like a day,\nyou're not just going to sit there and\nwait for it to finish. You're probably\ngoing to kick off another one and\nanother one and do a bunch of work. And\nso I think this is where like async\nmanagement of things comes into play. I\nthink things like linear and Jira and\ncanban boards and maybe even email are\nuh interesting to look at for\ninspiration about like what it looks to\nlike to basically manage a lot of these\nagents. But I think for a lot of these\nat some point you're going to want to uh\nswitch into synchronous communication\nwith uh these agents because they come\nback with a research report and you want\nto give it feedback that it wrote\nsomething wrong. And I actually think\nchat's like reasonably good at that. The\nonly thing that I'll maybe say there is\nthat so many of these agents are now\nmodifying other things like files in a\nfile system that having some way to view\nthat like state is really important. And\nso you see this in coding um where IDE's\nids are still used when you want to go\nin and manually kind of like change code\nand and uh even when I kick off uh cloud\ncode when it finishes I sometimes I pull\nit up and look at the code that it\nactually wrote. Um and so I think I\nthink having a a way to view that state\nis interesting. One of the one of the\nreally cool things that Anthropic did\nwith their Claude co-work um when you\nset it up, you choose the directory that\nit's kind of like working in and you're\nbasically saying like this is your\nenvironment. Um and obviously like\nthat's what you do in coding as well.\nYou open your ID to a particular\ndirectory. But I think that's a nice\nmental kind of like framing is like this\nis your workspace. That could workspace\ncould be a Google drive. It could be a\nnotion page. Um, it could be anything\nthat like stores state and then you and\nthe agent are collaborating on that\nstate. You kick it off, you manage maybe\na bunch of these running asynchronously.\nThen you go into sync mode where you\nchat with it, but you also view this\nstate. And so that's kind of what I see\nright now.\nAnd this is like your agent inbox idea\nthen of, you know, to to enable the sync\nmode, your agent's going to have to need\na way of reaching you.\nYeah, exactly. And so yeah, so the agent\ninbox or something we launched that\nabout a year ago and and had this idea\nof like ambient agents that ran in the\nbackground and pinged you and the first\nversion of that didn't have a sync mode\nand so it would ping you and then you'd\ngive a response but then you'd kind of\njust wait for it to ping you again but\noftentimes like when I was switching in\nto email you and respond to you, I would\nI would I would I would say very small\nthings and I didn't want to switch out\nand wait like I I you're really\nimportant. So I wanted to like be in the\nsync mode in this conversation with the\nagent. And so one of the things we added\nwas this was was now when you open the\ninbox you're brought into chat and chat\nis very synchronous and that was\nactually a big unlock. So I actually\nthink having just an async mode. I don't\nthink that really works right now. Maybe\nin the future if they get so good that\nyou don't really need to like correct\nthem as much it gets more viable. But at\nleast right now I think we see people\nswitching from async to sync and back\nand forth.\nWhat do you think of code sandboxes?\nLike is is every agent going to have\naccess to a sandbox? Is every agent\ngoing to have access to or a computer?\nIs every agent going to have access to a\nbrowser?\nReally good question. Something we're\nthinking a bunch about. I think uh I\nthink coding has clearly worked more\nthan browser use so far. So at least in\nthe short term, it seems like if any of\nthose are going to be a key part there,\nit's going to be this code execution\npart. Um file systems, I'm completely\nfile system pled. I think in some form\nagent should have access to some file\nsystem coding. I'm maybe not as pilled,\nbut I'm probably like I'm like maybe\nlike 90% there. Like yeah, I think like\nit is definitely possible. There are\nit's maybe for like the longer\num tale of use cases. So maybe there's\nsomething where if you're doing\nsomething repeated, you need code less,\nbut I think file systems are still\nuseful because that repeated thing could\nbe generating a lot of context and and\nyou need to do context engineering. Um\nbut for the long tale of things,\ncoding's great and there's really no\nreplacement for that. browser use. Um, I\nthink the models just aren't good enough\nat it right now from what we've seen.\nUm, you could probably give like a\ncoding agent a CLI to do browser use and\nthere's probably some approximation\nthere. There's probably some people\ndoing some I think I have seen some cool\nstuff there.\nUm, and then computer use is like a\nweird hybrid of the two. Um, so if Yeah,\ncode sandboxes, I really like code\nsandboxes.\nYeah. Yeah.\nCool. Um, Harrison, thank you so much\nfor joining us today. you have\nconsistently seen the future on agents\nand it was really cool to have this\nconversation and talk about how context\nengineering has evolved to the current\npoint in time with with harnesses and\nand long horizon agents and so thank you\nfor for driving that future and thank\nyou for always chatting with us about\nit.\nThank you for having me on. I look\nforward to being back on sometime in the\nfuture and being completely wrong about\neverything I said today. So it's very\nhard to predict the future.",
  "transcript_chars": 46526,
  "ingested_at": "2026-05-12T00:49:29.067777+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 113620,
    "like_count": 331,
    "channel_id": "UCWrF0oN6unbXrWsTN7RctTw",
    "categories": [
      "People & Blogs"
    ],
    "tags": []
  }
}