{
  "video_id": "eor3eU9eReA",
  "channel_slug": "jamesbriggs",
  "channel_handle": "jamesbriggs",
  "title": "LangChain Streaming and API Integration",
  "duration_seconds": 2269.0,
  "url": "https://www.youtube.com/watch?v=eor3eU9eReA",
  "upload_date": "",
  "transcript": "In this chapter, we're going to cover\nstreaming and async in lang chain. Now,\nboth using async code and using\nstreaming are incredibly important\ncomponents of I think almost any\nconversational chat interface or at\nleast any good conversational chat\ninterface. or async. If your application\nis not async and you're spending a load\nof time in your API, whatever else\nwaiting for LM calls because a lot of\nthose are behind APIs, you are waiting\nand your application is doing nothing\nbecause you've written synchronous code\nand that well there are many problems\nwith that mainly it doesn't scale. So\nasync code generally performs much\nbetter and especially for AI where a lot\nof the time we're kind of waiting for\nAPI calls. So async is incredibly\nimportant for that for streaming. Now\nstreaming is slightly different thing.\nSo let's say I want to tell me a story.\nOkay, I'm using GBT4 here. It's a bit\nslower. So we can actually use string.\nWe can see that token by token this text\nis being produced and sent to us. Now\nthis is not just a visual thing. This is\nthe LM when it is generating tokens or\nwords is generating them one by one and\nand that's because these LM literally\ngenerate tokens one by one. So they're\nlooking at all of the previous tokens in\norder to generate the next one and then\ngenerate next one generate next one.\nThat's how they work. So when we are\nimplementing streaming we're getting\nthat feed of tokens directly from the LM\nthrough to our you know our back end or\nour front end that is what we see when\nwhen we see that token by token\ninterface right so that's one thing what\none other thing that I can do that let\nme switch across to 40 is I can say okay\nwe just got this story I'm going to ask\nare There\nany standard storytelling\ntechniques\nto follow? Use above. Please use search.\nOkay. So look, we we get this very\nbriefly there. We saw that it was\nsearching the web. And the way it's not\nbecause we told it, okay, we told the LM\nto use the search tool, but then the LM\noutput some tokens to say use the search\ntool that is going to use a search tool.\nAnd it also would have output the tokens\nsaying what that search query would have\nbeen although we didn't see it there.\nBut what the chat GPT interface is doing\nthere. So it it received those tokens\nsaying, hey, I'm going to use a search\ntool. It didn't just send us those\ntokens like it does with the standard\ntokens here. Instead, it used those\ntokens to show us that searching the web\nlittle text box. So, streaming is not\njust the streaming of these direct\ntokens. It's also the streaming of these\nintermediate steps that the LLM may be\nthinking through which is particularly\nimportant when it comes to agents and\nagentic interfaces. So it's also a\nfeature thing, right? Streaming doesn't\njust look nice, it's also a feature.\nThen finally, of course, when we're\nlooking at this, okay, let's say we go\nback to\nGT4\nand I say, okay, use all\nof this information\nto generate a long story for me, right?\nAnd okay, we are getting the first token\nnow. So, we know something is happening\nand we need to start reading. Now,\nimagine if we were not streaming\nanything here and we're just waiting,\nright? We're still waiting. Now, we're\nstill waiting and we wouldn't see\nanything. We're just like, oh, it's just\nblank or maybe there's a little loading\nspinner. So, we'd still be waiting\nand\neven now we're still waiting,\nright? This is an extreme example,\nbut can you imagine just waiting for so\nlong and not seeing anything as a user\nright now? Just now we would have got\nour answer if we were not streaming. I\nmean that that would be painful as a\nuser. You you' not want to wait\nespecially in a chat interface. You\ndon't want to wait that long. It's okay\nwith okay for example deep research\ntakes a long time to process but you\nknow it's going to take a long time to\nprocess and it's a different use case\nright you're getting a report this is a\nchat interface and yes most messages are\nnot going to take that long to generate\nalso probably not going to be using GT4\ndepending on I don't know maybe some\npeople still do but in some scenarios\nit's painful to need to wait that long\nOkay. And it's also the same for agents.\nIt's nice when you're using agents to\nget an update on, okay, we're using this\ntool, it's using this tool, this is how\nit's using them. Perplexity, for\nexample, I have a very nice example of\nthis. So, okay, what's this? Openi\nco-ounder joins Morati startless. Right.\nSo, we see this is really nice. It's\nwe're using pro search. It's searching\nfor news, showing us the results. Like,\nwe're getting all this information as\nwe're waiting, which is really cool.\nAnd it helps us understand what is\nactually happening, right? It's not\nneeded in all use cases, but it's super\nnice to have those intermediate steps,\nright? So then we're not waiting. And\nthen I think this bit probably also\nstreamed, but it was just super fast, so\nI I didn't see it, but that's pretty\ncool. So streaming is pretty important.\nLet's dive into our example. Okay, we'll\nopen that in Collab. And off we go. So\nstarting with the prerequisites same as\nalways lang chain optionally linesmith\nwe'll also enter our lchain API key if\nyou'd like to use linesmith we'll also\nenter our openi API key so that is\nplatform.openi.com openi.com and then as\nusual we can just invoke our lm right so\nwe have that it's working now let's see\nhow we would stream with a stream okay\nso whenever a method so stream is\nactually a method as well we could use\nthat but it's not async right so\nwhenever we see a method in line chain\nthat has a prefixed onto what would be\nanother method that's like the async\nversion of this So\nwe can actually stream using async super\neasily using just lmream.\nOkay. Now, this is just an example in to\nbe completely honest, you probably will\nnot be able to use this in an actual\napplication, but it's just an example\nand we're going to see how we would use\nthis or how we would stream\nasynchronously in an application further\ndown in this notebook. So, starting with\nthis, you can see here that we're\ngetting these tokens, right? We're just\nappending it to tokens here. We don't\nactually need to do that. I don't think\nwe're even using this, but maybe we\nYeah, we'll do it here. It's fine. So,\nwe're just appending the tokens as they\ncome back from our LM, pending it to\nthis. We'll see what that is in a\nmoment. And then I'm just printing the\ntoken content,\nright? So, the content of the token. So,\nin this case, that would be L. In this\ncase, it would be LP. It would be SANS\nfor so on and so on. So you can see for\nthe most part it's it's tends to be word\nlevel but it can also be subword level\nas you see sent imment is one word of\ncourse. So you know they they get broken\nup in in various ways.\nThen adding this pipe character onto the\nend here so we can see okay where are\nour individual tokens. Then we also have\nflush. So flush uh you can actually turn\nthis off and it's still going to stream.\nIt's still going to see everything but\nit's going to be a bit more. You can see\nit's kind of a it's like bit by bit when\nwe use flush it forces the console to\nupdate what is being shown to us\nimmediately. All right. So we get a much\nsmoother\num when we're looking at this it's much\nsmoother versus when flush is not set to\ntrue. So yeah when you're printing that\nis good to do just so you can see you\ndon't necessarily need to. Okay. Now we\nadded all those tokens to the tokens\nlist. So we can have a look at each\nindividual object that was returned to\nus. Right? And this is interesting. So\nsee that we have the AI message chunk,\nright? That's an object. And then you\nhave the content. The first one's\nactually empty.\nSecond one has that N for NLP. And yeah,\nI mean that's all we rarely need to\nknow. They're very simple objects, but\nthey're actually quite useful because uh\njust look at this, right? So we can add\neach one of our AI message chunks,\nright? Let's see what that does. It\ndoesn't create a list. It creates this,\nright? So we still just have one AI\nmessage chunk. Ah, but it's combined the\ncontent within those AI message chunks,\nwhich is kind of cool, right? So for\nexample, like we could remove these,\nright? And then we just see NLP. So\nthat's kind of nice little feature\nthere. I do I actually quite like that.\nBut uh you do need to just be a little\nbit careful because obviously you can do\nthat the wrong way and you're going to\nget like a I don't know what that is.\nSome weird token salad. So yeah, you\nneed to just make sure you are going to\nbe merging those in the correct order\nunless you I don't know unless you're\ndoing something weird. Okay, cool. So\nstreaming that that was streaming from a\nLM. Let's have a look at streaming with\nagents. So we it gets a bit more\ncomplicated to be completely honest but\nwe also need to things are going to get\na bit more complicated so that we can\nimplement this in for example an API\nright there is it's kind of like a\nnecessary thing in any case. So to just\nvery quickly we're going to construct\nour agent executor like we did in the\nagent execution chapter. And for that\nfor the agent executor we're going to\nneed tools chat prompt template lm agent\nand the agent executor itself. Okay very\nquickly I'm not going to go through\nthese uh in detail. We just define our\ntools add multiply exponentiate subtract\nand the final answer tool. Merge those\ninto a single list of tools. Then we\nhave our prompt template. Again, same as\nbefore. We just have system message. We\nhave chat history. We have query. And\nthen we have the agent scratchpad for\nthose intermediate steps. Then we define\nour agent using LEL. Lel works quite\nwell with both streaming and async by\nthe way. It supports both out of the box\nwhich is nice. So we define our agent.\nThen coming down here, we're going to\ncreate the agent executor. This is the\nsame as before, right? So there's\nnothing new in here, I don't think. So\njust initialize our agent things there.\nThen it's, you know, we're looping\nthrough\nlooping through. Yeah, nothing nothing\nnew there. So we're just executing,\ninvoking our agent, seeing if there's a\ntool call. Uh this is slightly we could\nshift this to before or after. It\ndoesn't actually matter that much.\nSo we're checking if it's final answer.\nIf not, we continue XR tools and so on.\nOkay, cool. So then we can invoke that.\nOkay, we go what is 10 + 10.\nThere we go. Right. So we have our agent\nexecutor. It is working. Now, when we\nare running our agent executor, with\nevery new query, if we're putting this\ninto an API,\nwe're probably going to need to provide\nit with a a fresh call back handler.\nOkay, so this is the coret handler is\nwhat's going to handle uh taking the\ntokens that are being generated by a LM\nor agent and giving them to some other\npiece of code like for example the the\nstreaming response for a API\nand our callback handler is going to put\nthose tokens in a a queue in our case\nand then our for example the streaming\nobject is going to pick them up from the\nqueue and put them wherever they need to\nbe. So to allow us to do that with every\nnew query rather than us needing to\ninitialize everything when we actually\ninitialize our agent, we can add a\nconfigurable field to our LM. Okay, so\nwe set the configurable fields here. Oh,\nalso one thing is that we set streaming\nequal to true. That's very minor thing,\nbut just so you see that there we do do\nthat. So we add some configurable fields\nto our LM which means we can basically\npass an object in for these on every new\ninvocation.\nSo we set our configurable field. It's\ngoing to be called callbacks and we we\njust add a description, right? There's\nnothing more to it. So this will now\nallow us to provide that field when\nwe're invoking our agent. Okay. Now we\nneed to define our callback handler and\nas I mentioned what is basically going\nto be happening is this callback handler\nis going to be passing tokens into our\nasync io Q object and then we're going\nto be picking them up from the queue\nelsewhere. Okay, so we can call it a Q\ncallback handler. Okay. And that is\ninherenting from the async callback\nhandler because we want all this to be\ndone asynchronously because we're we're\nthinking here about okay, how do we\nimplement all this stuff within APIs and\nactual real world code and we we do want\nto be doing all this in async. So let me\nexecute that and I'll just explain a\nlittle bit of what we're looking at. So\nwe have the initialization, right? It's\nnothing nothing specific here. We just\nwhat we really want to be doing is we\nwant to be setting our Q object\nassigning that to the class attributes\nand then there's also this final answer\nscene which we're setting to false. So\nwhat we're going to be using that for is\nwe our LM will be streaming tokens to us\nwhilst it's using its tool calling and\nwe might not want to display those\nimmediately or we might want to display\nthem in a different way. So by setting\nthis final answer scene to false\nwhilst our LM is outputting those tool\ntokens, we can handle them in a\ndifferent way. And then as soon as we\nsee that it's done with the tool calls\nand it's onto the final answer, which is\nactually another tool call, but once we\nsee that it's onto the final answer tool\ncall, we can set this to true. And then\nwe can start processing our tokens in a,\nyou know, different way essentially.\nOkay, so we have that. Then we have this\naitter method. This is required\nfor any async generator object. So what\nthat is going to be doing is going to\niterating through right it's a\ngenerator. It's going to be going\niterating through and it's going to\nsaying okay if our que is empty right\nthis is the queue that we set up here.\nIf is empty wait a moment right we use\nthe sleep method here and this is an\nasync sleep method. This is super\nimportant. We're using we're awaiting\nfor an asynchronous sleep. All right. So\nwhilst we're whilst we're waiting for\nthat 0.1 seconds our our code can be\ndoing other things, right? That that is\nimportant. If we if we use I think the\nstandard is time.leep that is not\nasynchronous and so it will actually\nblock the thread for that 0.1 seconds.\nSo we don't want that to happen.\nGenerally, our queue should probably not\nbe empty that frequently given how\nquickly uh tokens are going to be added\nto the queue. So, the only way that this\nwould potentially be empty is maybe our\nLM stops. Maybe there's like a\nconnection interruption for, you know, a\nbrief second or something and no tokens\nare added. So, in that case, we don't\nactually do anything. We don't keep\nchecking the queue. We just wait a\nmoment. Okay? And then we check again.\nNow, if it was empty, we wait and then\nwe continue onto the next iteration.\nOtherwise, it probably won't be empty.\nWe get whatever is from our inside our\nqueue. We get that out, pull it out.\nThen we say, okay, if that token is a\ndone token, we're going to return. So,\nwe're going to stop this generator,\nright? We're finished.\nOtherwise, if it's something else, we're\ngoing to yield that token, which means\nwe're we're returning that token, but\nthen we're continuing through that loop\nagain, right?\nSo, that is our generator logic. Then we\nhave some other methods here. These are\nl these are line chain specific. Okay,\nwe have on LM new token and we have on\nLM end. Starting with on LM new token.\nThis is basically when an LM returns a\ntoken to us. Line chain is going to run\nor execute this method. Okay, this is\nthe method that will be called. What\nthis is going to do is it's going to go\ninto the keyword argument since it's\ngoing to get the chunk object. So this\nis coming from our LM. If there is\nsomething in that chunk, it's going to\ncheck for a final answer tool call\nfirst. Okay, so we get our tool calls\nand we say if the name within our chunk,\nright, pro probably this will be empty\nmost of the tokens we return, right?\nBecause you remember before when we're\nlooking at the chunks here, this is what\nwe're looking at, right? The content for\nus is actually always going to be empty\nand instead we're actually going to get\nthe additional keyword argu\ntool calling, our tool calls as we saw\nin the the previous videos, right? So\nthat's what we're extracting. We're\nextracting that information. That's why\nwe're going additional keyword args,\nright? And get those tool the tool call\ninformation, right? Or it will be none,\nright? So if if it is none, I don't\nthink it ever would be none to be\nhonest. It would be strange if it's\nnone. I think that means something would\nbe wrong. Okay. So here we're using the\nwarest operator. So the warus operator\nwhat it's doing here is whilst we're\nchecking the if logic here whilst we do\nthat it's also assigning whatever is\ninside this it's assigning over to tool\ncalls and then with the if we're\nchecking whether tool calls is well\nsomething or none right because we're\nusing get here so if if this get\noperation fails and there is no tool\ncalls this object here will be equal to\nnone\nwhich gets assigned to tool cause here\nand then this this if none will return\nfalse and this logic will not run. Okay.\nAnd it will just continue if this is\ntrue. So if there is something returned\nhere, we're going to check if that\nsomething returned is using the function\nname or tool name final answer. If it\nis, we're going to set that final answer\nscene equal to true. Otherwise, we're\njust going to add our chunk into the\nqueue. Okay, we use put no weight here\nbecause we're we're using async.\nOtherwise, if you were not using async,\nI think you might just put wait or maybe\neven put put. No. Okay, you you'd use\nput if it's just synchronous code, but I\nI don't think I've ever implemented this\nsynchronously. So, it would actually\njust be put no weights for async. Okay,\nand then return. So, we have that.\nThen we have on lm end. Okay, so this is\nwhen line chain sees that the LLM has\nreturned\nor indicated that it is finished with\nthe response, line chain will call this.\nSo\nyou you have to be aware that this will\nhappen multiple times during an agent\nexecution because if you think within\nour agent executor, we're hitting the LM\nmultiple times. we have that first step\nwhere it's deciding, oh, I'm going to\nuse the add tool or the multiply tool\nand then that response gets back to us.\nWe execute that tool and then we pass\nthe output from that tool and all the\noriginal user query in the chat history.\nWe pass that back to our LM again. All\nright, so that's another call to our LM\nthat's going to come back. It's going to\nfinish or it's going to give us\nsomething else, right? So there's\nmultiple LM calls happening throughout\nour agent execution logic. So this on LM\ncall will actually get called at the end\nof every single one of those LLM calls.\nNow if we get to the end of a LM call\nand it was just a it was a tool\ninvocation. So we had the you know it\ncalled the add tool. We don't want to\nput the done token into our queue\nbecause when the done token is added to\nour queue we're going to stop iterating.\nOkay?\nInstead, if it was just a tool call,\nwe're going to say step end, right? And\nwe'll actually get this token back. So,\nthis is useful on, for example, the\nfront end, you could have, okay, I've\nI've used the add tool. The these are\nthe parameters and it's the end of the\nstep. So, you could have that your tool\ncall is being used on some front end.\nAnd then, as soon as it sees step end,\nit knows, okay, we're done with that.\nHere was the response, right? And and it\ncan just show you that. and we we're\ngoing to use that. We'll see that soon.\nBut let's say we get to the final answer\ntool. We're on the final answer tool and\nthen we get this signal that the LLM has\nfinished. Then we need to stop\niterating. Otherwise, our our streaming\ngenerator is just going to keep going\nforever, right? Nothing's going to stop\nit. Or maybe it will time out. I don't\nthink it will, though. So,\nat that point, we need to send Okay,\nstop, right? We need to say we're done.\nAnd then that will that will come back\nto here to our a iterator and to our\nasync iterator and it will return and\nstop the generator. Okay, so that's the\ncore logic that we have inside there. I\nknow there's a lot going on there. It's\nbut we need all of this. So it's it's\nimportant to be aware of it. Okay, so\nnow let's see how we might actually call\nour agent with all of this streaming uh\nin in this way. So\nwe're going to initialize our queue and\nwe're going to use that to initialize a\nstreamer. Okay, using the the custom\nstreamer that we just set, custom\ncallback handler, whatever you want to\ncall it. Okay, then I'm going to define\na function. So this is an asynchronous\nfunction. It has to be if if we're using\nasync. And what it's going to do is it's\ngoing to call our agent with a config\nhere. And we're going to pass it that\ncall the the callback which is the\nstreamer. Right? Note here I'm not\ncalling the agent executor. I'm just\ncalling the agent. Right? So the uh if\nwe come back up here, we're calling\nthis. All right? So that's not going to\ninclude all tool execution logic. And\nimportantly,\nwe're calling the agent with the config\nthat uses callbacks, right? So this this\nconfigurable fields here from our LM is\nactually being fed through and it\npropagates through to our agent object\nas well to the runnable serializable.\nAll right, so that's what we're\nexecuting here. We see agent with config\nand we're passing in those callbacks uh\nwhich is just one actually. Okay, so\nthat sets up our agent and then we\ninvoke it with a string. Okay, like we\ndid before. And we're just going to\nreturn everything. So, let's uh run\nthat. Okay, and we see all the token or\nthe chunk objects are being returned.\nAnd this is useful to understand what\nwe're actually doing up here, right? So,\nwhen we're doing this chunk message\nadditional keyword arguments, right, we\ncan see that in here. So, this would be\nthe chunk message object. We get the\nadditional keyword logs. We go into tool\ncalls and we get the information here.\nSo we have the ID for that tool call as\nwe saw in the previous chapters. Then we\nhave our function, right? So the\nfunction includes the name, right? So we\nknow what tool we're calling from this\nfirst chunk, but we don't know the\narguments, right? Those arguments are\ngoing to be stream to us. So we can see\nthem begin to come through in the next\nchunk. So next chunk is just it's just\nfirst token for for the add function,\nright? And we can see these all come\ntogether over multiple steps and we\nactually get all of our arguments. Okay,\nthat's pretty cool. So\nactually one thing I would like to show\nyou here as well. So if we just do token\nequals\ntokens sorry\nand we do\ntokens append token.\nOkay, we have all of our tokens in here\nnow. All right, you see that they're all\nAI message chunks.\nSo we can actually add those together.\nAll right, so let's we'll go with these\nhere. And based on these, we're going to\nget all the arguments. Okay, so this is\nkind of interesting. So it's one until I\nthink like the second to last maybe.\nRight. So we have these and actually we\njust want to add those together. So I'm\ngoing to go with tokens\none. I'm just going to go for\nuh\nfor token in we're going to go from the\nsecond onwards I'm going to TK plus\ntoken right and let's see what TK looks\nlike at the end here TK\nokay so now you see it's kind of merged\nall those um arguments here\nsorry plus equal\nOkay. So, run that. And you can see here\nthat it's merged those arguments. It\ndidn't get all of them. So, I kind of\nmissed some at the end there, but it's\nmerging them, right? So, we can see that\nthat logic where it's, you know, before\nit was adding the content from various\ntrunks. It also does the same for the\nother parameters within your trunk\nobject, which is is I I think it's\npretty cool. You can see here the name\nwasn't included. That's because we\nstarted on token one rather than token\nzero where the name was. So if we\nactually started from token zero and\nlet's just let's just put all them in\nthere, right? So from one onwards,\nwe're going to get a complete AI message\nchunk which includes the name here and\nall of those arguments and you you'll\nsee also here right populate everything\nwhich is pretty cool.\nOkay, so we have that. Now, based on\nthis, we're going to want to modify our\ncustom agent executor\nbecause we're streaming everything,\nright? So, we want to add streaming\ninside our agent executor, which we're\ndoing here, right? So, this is async def\nstream and we're sharing async for token\nin the Astream.\nOkay, so this is like the very first\ninstance. If output is non, we're just\ngoing to be adding our token. So the the\nchunk sorry to our output like the first\ntoken becomes our output. Otherwise,\nwe're just appending our tokens to the\noutput. Okay, if the token content is\nempty, which it should be, right,\nbecause we're using tool calls all the\ntime, we're just going to print content.\nOkay, I just added these as so we see\nlike print everything. I just want to\nwant to be able to see that. I wouldn't\nexpect this to run because we're saying\nit has to use tool calling. Okay, so\nwithin our agent, if we come up to here,\nwe said tool choices any. So it's being\nforced to use tool calling. So it should\nnever really be returning anything\ninside the content field, but just in\ncase it's there, right? So we'll we'll\nsee if that is actually true. Then we're\njust getting out our tool calls\ninformation, okay, from our trunk. And\nwe're going to say, okay, if there's\nsomething in there, we're going to print\nwhat is in there, okay? And then we're\ngoing to extract our tool name. If there\nis some, if there is a tool name, I'm\ngoing to show you the tool name.\nThen we're going to get the argu. And if\nthe argu,\nwe're going to see what we get in there.\nOkay? And then from all of this, we're\nactually going to we merge all of it\ninto our AI message, right? because\nwe're merging everything as we're going\nthrough. We're merging everything into\noutputs as I showed you before. Okay,\ncool. And then we're just awaiting our\nstream that will like kick it off. Okay.\nAnd then we do the the standard agent\nexecutor stuff again here, right? So\nwe're just pulling out tool name, tool\nlogs, tool call ID, and then we're using\nall that to execute our tool here. And\nthen we're creating a new tool message\nand passing that back in. And then also\nhere I move the break for the final\nanswer into the final step. So that is\nour custom Asian executor with streaming\nand let's see what let's see what it\ndoes. Okay, setting for the Bose equal\ntrue. So we see all those print\nstatements.\nOkay, so you can kind of see it's a\nlittle bit messy, but you can see we\nhave tool calls that had some stuff\ninside. It had add here. And what we're\nprinting out here is we're printing out\nthe full AI message chunk with tool\ncalls. And then I'm just printing out\nokay what are we actually pulling out\nfrom from that. So these are actually\ncoming from the same thing. Okay. And\nthen same here right so we're looking at\nthe full message and then we're looking\nokay we're getting this argument out\nfrom it. Okay. So we can see everything\nthat is being pulled out chunk by chunk\nor token by token and that's it. Okay.\nSo we could just get everything like\nthat. However, right? So I'm I'm\nprinting everything. So we can see that\nit's streaming. What if I don't print?\nOkay, so we're setting verbose or by\ndefault verbose is equal to false here.\nSo what happens if we invoke now? Let's\nsee.\nOkay,\ncool. We got nothing.\nSo the reason we got nothing is because\nwe're not printing.\nBut we don't if you are if you're\nbuilding an an API for example you're\nyou're pulling your tokens through you\ncan't print them to your like a front\nend or or print them as to the output of\nyour API printing goes to your terminal\nright your console window doesn't go\nanywhere else\ninstead what we want to do is we\nactually want to get those tokens out,\nright? But if but how do we do that? All\nright, so we we printed them, but\nanother place that those tokens are is\nin our queue, right? Because we set them\nup to go to the queue.\nSo we can actually pull them out of our\nqueue whilst our agent executor is\nrunning and then we can do whatever we\nwant with them because our code is\nasync. So it can be doing multiple\nthings at the same time. So whilst our\ncode is running the agent executor,\nwhilst that is happening, our code can\nalso be pulling out from our queue\ntokens that are in there and sending\nthem to like an API for example, right?\nOr whatever downstream logic you have.\nSo let's see what that looks like. We\nstart by just initializing our queue,\ninitializing our streamer with that Q.\nThen we create a task. So this is\nbasically saying, okay, I I want to run\nthis, but don't run it right now. I'm\nnot ready yet. The reason that I say I'm\nnot ready yet is because I also want to\ndefine here my async loop which is going\nto be printing those tokens, right? But\nthis is async, right? So we we set this\nup. This is like get ready to run this\nbecause this is async. This is running,\nright? This is just running like it's\nthere. It's it's already running. So we\nget to this, we continue, we continue\nthis, none of this is actually executed\nyet,\nright? Only here when we await the task\nthat we set up here. Only then does our\nagent executor\nrun and our async object here begin\ngetting tokens. Right? And here again,\nI'm printing, but I don't need to print.\nI could I could have like a let's say\nwhere this is within an API or\nsomething.\nLet's say I'm I'm saying okay send token\nto XYZ\ntoken right that's sending up tokens\nsomewhere or if we're maybe we're\nyielding this to our some sort of\nstreamer object within our API right we\ncan do whatever we want with those\ntokens. Okay I'm just printing them\nbecause I want to actually see them.\nOkay, but just important here is that\nwe're not printing them within our agent\nexecutor. We're printing them outside\nthe agent executor. We've got them out\nand we can put them wherever we want,\nwhich is perfect when you're building an\nactual sort of real world use case where\nusing an API or something else. Okay, so\nlet's run that. Let's see what we get.\nLook at that. We get all of the\ninformation we could need and a little\nbit more, right? because now we're using\nthe agent executor and now we can also\nsee oh we have this step end right so I\nknow all I I know just from looking at\nthis right this is my first tool use so\nwhat tool is it let's have a look it's\nthe add tool and then we have these\narguments so I can then pass them right\ndownstream then we have the next tool we\nuse which is here down here so then we\ncan then pass them in the way that we\nlike So that's pretty cool.\nLet's I mean let's see. Right. So we're\ngetting those fingers out. Can we can we\ndo something with them before I before I\nprint them and show them? Yes. Let's\nsee. Okay. So we're now modifying our\nour loop here. Same stuff, right? We're\nstill initializing our queue,\ninitializing our streamer, initializing\nour task. Okay. And we're still doing\nthis async for token streamer. Okay. But\nthen we're doing stuff with our tokens.\nSo I'm saying, okay, if we're on stream\nend, I'm not actually going to print\nstream end. I'm going to print new line.\nOkay.\nOtherwise, if we're getting a tool call\nhere, we're going to say if that tool\ncall is the tool name, I am going to\nprint calling tool name. Okay. If it's\nthe arguments, I'm going to print the\ntool argument and I'm going to end up\nwith nothing so that we don't go on to a\nnew line. So, we're actually going to be\nstreaming everything. Okay. So, let's\njust see what this looks like.\nOh, my bad. I just added that.\nOkay.\nYou see that? So, it go it goes very\nfast. So, it's kind of hard to see it.\nI'm going to slow it down so you can\nsee. So you can see that we as soon as\nwe get the tool name we stream that\nwe're calling the ad tool. Then we\nstream token by token the actual\narguments for that tool. Then for the\nnext one again we do the same. We're\ncalling this tool name. Then we're\nstreaming token by token again. We're\nprocessing everything downstream from\noutside of the agent executor. And this\nis an essential thing to be able to do\nwhen we're actually implementing\nstreaming and async and everything else\nin an actual application. So I know\nthat's a lot but it's important. So that\nis it for our chapter on streaming and\nasync. I hope it's all been useful.\nThanks.",
  "transcript_chars": 32363,
  "ingested_at": "2026-05-15T04:37:23.956638+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 2301,
    "like_count": 46,
    "channel_id": "UCv83tO5cePwHMt1952IVVHw",
    "categories": [
      "Science & Technology"
    ],
    "tags": [
      "python",
      "langchain",
      "langchain course",
      "langchain tutorial",
      "langchain agent",
      "langchain chatbot",
      "langchain crash course",
      "james briggs",
      "james briggs langchain",
      "langchain project",
      "langchain agent example",
      "langgraph",
      "langchain langgraph",
      "langchain langsmith",
      "langchain full course",
      "langchain openai",
      "langchain conversational agent",
      "langchain expression language",
      "ai streaming",
      "openai streaming"
    ]
  }
}