{
  "video_id": "QtDBjO4B_ug",
  "channel_slug": "assemblyai",
  "channel_handle": "assemblyai",
  "title": "Build a Voice Agent in an Hour with Claude Code | AssemblyAI Workshop",
  "duration_seconds": 3595.0,
  "url": "https://www.youtube.com/watch?v=QtDBjO4B_ug",
  "upload_date": "",
  "transcript": "All right, so uh let's get started. Um\nso for intros, uh my name is Dan. Uh I'm\na product manager here at Assembly for\nthe Voice Agent API. And uh in this\nworkshop, we're going to be running\nthrough uh building a voice agent\ntogether on the voice agent API with\nClaude Code. Um and uh yeah, it should\nbe really fun. Uh this is my first\nwebinar, so do bear with if there are\nany issues. Um uh but yeah, I'm hoping\nit's going to be almost as if like I'm\nsimulating being in a customer. Um and\nwe're going to run through the docs and\nlike building it together. Uh so you can\nfollow along and we're going to be\nbuilding an appointment setting voice\nagent uh with like a front-end\napplication. So um you can kind of take\nthis code and run with it if you have an\nidea. Uh but this is just showing you\nhow you can get started uh with the\nvoice agent API.\nRight, I think we can get started now.\nUh, so\nin terms of agenda, uh, we're going to\nrun through the, uh, setup and Oh,\nRobert. Yeah. Yeah, this is being\nrecorded. And, uh, thank thanks for all\nthe questions. Uh, definitely keep\nfiring them in as we're going. Uh, yeah,\nthis is going to be definitely more of a\ncollaborative uh, session rather than\nlike Q&A just being at the end. Um, but\nI'll run through like what is assembly\nfor the people that uh might be new to\nto us. Uh, what is the voice agent API?\nWe'll then do the live build together\nwith Claude code. Um, and then I'll show\nyou afterwards how you can iterate with\nthat like coding agent mental model that\nwe'll get into. And then at the end if\nwe if we have time, we'll do a deploy uh\ndeployment and we'll deploy to railway.\nSo uh by the end of this session, you\ncan have a appointment booking agent\nthat you can share with your colleagues\nor friends.\nUm, so to begin, uh, I want to demo what\nwe're going to be building. Uh, so this\nis an agent I created before this\nsession. We're going to try and recreate\nthis live, uh, on this workshop. Um, so\nlet's talk to it and like see how it\ngoes. And, uh, yeah, it'll be an\nappointment setting agent.\nHey, David. Uh, I'm looking to book a\nhaircut.\n>> Yeah, I was thinking this Friday.\nOh, sorry you can't hear the responses.\nLet me reshare my screen. Uh, this is\ntypical first webinar experience.\nOkay, let's rerun this conversation.\nThanks for calling that up.\n>> Hi, thanks for calling Aura Studio. This\nis David. What can I book for you?\n>> Hi, David. Uh, I'm looking to get a\nhaircut, please.\n>> Sure thing. We can definitely do a\nhaircut for you. What day were you\nthinking of coming in?\n>> Yeah, I was thinking uh this Friday.\n>> Let me check that for you. One sec.\nOkay, for this Friday, we have openings\nat 10:00 a.m., 11:30 a.m. or 100 p.m.\n>> Okay, anyway, we get the gist. Um, I'm\nguessing a lot of you guys are hopping\non this link. So, I think this railway\ndeployment is slowing uh slowing down a\nbit. Uh, but this is the agent we'll be\nbuilding. You can see the tools are\nexecuting live in the UI. Um, and I\nthink what's really great about this is\nlike you can go and build your own\napplication, your own UI. You display\nthe transcripts the way you want. You\ncan uh create like an interactive UI\nexperience with the tool calling. Um,\nand the voice agent API makes this\neasier than if you're plugging into some\nother orchestrator or some other\napplication where it's more like\nembedded into their systems. The voice\nagent API is really meant to like bold\ninto your application. So, you can go\nand build like your own unique voice\nagent application uh that that's for\nyour business and displays the way that\nyou want it to. Um, so yeah, that was\nthe demo of what we're going to build.\nUh so let's go back to the slides and um\nso what you need to work uh to work\nalong so if you haven't already\ndefinitely do sign up for assembly I\naccount and API key is going to be a\nrequirement for the voice agent API. Uh\nyou do get $50 in free credits for\nsigning up so no worries about adding a\ncredit card and you will need clawed\ncode. Uh this could be like the clawed\napp where it has the code section in the\napp or uh I will be using VS Code with\nthe claude code plugin. Um, so if you\nhave either of them, it should work. We\nwon't really be digging into the actual\ncode itself. It's just more about having\nan interface where you can chat to\nClaude, uh, give it like the\ndocumentation link and tell it what you\nwant to build. Um, so make sure you've\ngot that that set up. Uh, Oliver, yeah,\nit is indeed possible to tie into a\nphone call process. Um, if you connect\nit to like Twilio rather than connecting\nthrough the UI, you can then have the\ntool execute and like book an\nappointment, let's say, on cal.com.\nWe're not going to be getting into like\nactually executing business logic in in\nthis session, but we will be doing\nfollow-up sessions and you can check out\nour docs uh for instructions on how to\ndo that.\nUh so firstly, what is Ascendi? For\nthose that are new, uh we are a speech\nmodel uh company. So we create uh and we\nwe do the research engineering design uh\nfor the API of like building\ntranscription models uh that can like\ntake the audio and turn it into text. So\nlet's say if you're building a meeting\nrecorder, you can summarize a meeting\nand get notes afterwards. Or if you're a\ncall center company, maybe you want to\nbe transcribing all of those calls so\nyou can score them and make sure\neveryone's following on the script. Um,\nbut you can also build voice agents that\nuh don't require any human in the loop\nand use the transcription to pass to an\nLLM and then to like a speech generation\nmodel. And that's what we'll be focusing\non here today. So, we're using\nassembly's models under the hood um and\nand it's powering the voice agent.\nAnd for the voice agent API, uh it's\nit's one API that's using our models and\nwe provide the full loop. So rather than\nneeding to set up a speech to text\nmodel, a large language model, and a\ntext to speech model on your end, we\nhandle the orchestration uh because\nwe're uh like a model company as well,\nwe can be vertically integrated. So\nwe're making sure that the model plays\nwell with our custom orchestration. Um\nand we're also designing the API layer\non top. You might know from our uh\nspeechto text APIs. uh we definitely\nfocus on more like simplicity like\ndeveloper friendly um uh APIs and we\nwant to carry that into the voice agent\nAPI. Um we really felt like there was a\ngap in the market but a nice developer\nfriendly API for building voice agents\nthat like as I was describing before\nrather than uh like attaching an agent\nto your application you're like building\nyour application within the agent. Um\nand we'll get into more of what that\nlooks like. Um, but in terms of the\npricing, we're also trying to be\ncompetitive. So, at $450 an hour, we're\ncheaper than a lot of other options on\nthe market. Um, and this is because of\nus being vertically integrated. So, like\nwe we we run the speech to text model,\nthe text to speech model, the LLM.\nThat's why we're able to offer like a\nreally competitive price. Um, and I\nthink it uh creates for a stronger\nproduct where we're making sure that the\norchestration is using the model's\nstrengths um uh to create like a better\nvoice agent experience. Um, yeah. Hey,\nRahan. Yeah, great to see. And, uh,\nyeah, this is being recorded. So at the\nend um I can share this slide deck uh\nand also uh the recording uh definitely\non like LinkedIn or be on our YouTube\nMorris. Um yeah certainly so the voice\nagent API it's not necessarily designed\njust to be for like phone agents or just\nfor web agents. Um, it it's meant to be\nlike a a developer tool that you can go\nand build kind of any use case where you\nneed voice input and then like an AI\nlike uh responding back to you or doing\nsome kind of action. So definitely like\na voice coding agent uh can be created.\n>> Yeah, we will be sharing this recording.\n>> Yeah.\nAnd Morris regarding the uh coding agent\nlike to to get it uh to be more of a\nhuman in the loop answer only. It would\nbe that the tool result uh like you\nwould have a tool that is executing the\ncode logic for you and then uh in the\ntool result you only pass back what you\nwant the agent to respond with. So you\nkeep it more conversational rather than\nuh\nOh, I see. Oh, the chat doesn't display\nto everyone. Thanks, Meredith.\nMalik. Yeah, you certainly can. Um, if I\nclick answer live, is that going to\nThere we go. Yeah. So, you can indeed do\nthis enterprise scale with thousands of\ncalls. I would say if you're looking to\ndo thousands of calls definitely do\nreach out to us so we can make sure that\nconcurrency limits and uh maybe you do\nneed some legal and security\nrequirements met like uh like a baa um\nso so do reach out to us but certainly\nthis API is designed that you can uh go\nand build an application and scale and\nnot have to worry about concurrency\nlimits being hit or uh pre- purchasing\nuh usage like for the month. It's just\nusage based pricing\nand yeah so uh in this workshop the\nvoice a we're going to be building so\nthe step by step will be first we'll be\ndesigning a backend in Python using\nflawed code um and we're also going to\nhave an an HTML front end so the back\nend uh is used to uh generate a\ntemporary token to pass to the front end\nfor authentication and this is because\nwe don't want to be leaking like our API\nI key to the client. So this is more of\njust uh security and like designing the\narchitecture in a way uh that that you\ncan share with people safely. Um and in\nthe UI we'll then be displaying the\ntranscripts and tool calls uh like we\nsaw on the demo.\nUh so now we can get into the live\nbuild. U I'm going to bring up my IDE\nand we'll just run through this\ntogether. Um and uh yeah, we're going to\ntry our best to recreate the demo. It\nmight not be exactly perfect, uh, but\nthat's definitely the goal. So, I'll\nstop sharing my screen and let's, uh,\nlet's bring up my ID.\nOkay. So, here I'm starting with a a\nblank slate in uh, in VS Code.\nAnd we'll create a new folder in the\ndownloads\nand let me reshare my screen onto the\nnew tab.\nSo, if you're following along, yeah,\nmake sure you set up a a new directory.\nAnd uh once you have Claude chat brought\nup, uh this is where we can get started.\nSo, this might take some uh Zoom\nswitching because I'm going to have to\nlike show my browser and then the code.\nUh but first thing I'm going to do is\nI'm going to bring up our documentation\nand let me share my screen here. This is\ngoing to be fun.\nSo, now that we have the docs brought\nup, um, if you're following along, all\nall we're going to do is we're going to\ncopy this page. Um, or actually, we'll\neven just copy the link. Uh, and we're\ngoing to pass this to Claude. And\neffectively, that is all we need to do\nto give Claude the context of how to\nwork with the API. So if you browse to\nthe docs and just copy this link, this\nis all we'll need from the docs and then\nfrom there we can just go all in code or\nall include code. So to start with we'll\nsay here are the docs\nfor the voice agent API.\nAnd as we discussed what what we need to\ndo first is set up a Python backend\nserver uh which is going to do all of\nlike the the authentication flow. Um,\nand it's going to mean that we're not\npassing our API key in the browser. So,\nthis is just the first kind of\narchitecture setup. So, let's say here\nare the docs for the voice agent API and\nyou set up a Python back end with a HTML\nfront end that uses the Python back end\nto generate a temp token for all in the\nfront end.\nSo, we'll let that run.\nAnd generally this was the idea with the\nwith the voice agent API and the way we\nset up the docs is we really wanted to\nenable coding agent iteration for voice\nagents. Um so you might notice with with\nother like voice agent uh providers it's\nprimarily like a UIdriven experience um\nor it's like very deeply in SDKs and\nlike low-level programming. Uh so we\nfound that there was this like middle\nground of almost like uh how easy\nsuperbase is to use for databases. Uh we\nthought why couldn't there be something\nlike that for voice agents. Um and yeah\nthis was where the idea came about of\njust having like a really easy to use uh\nAPI. You don't have to worry about\npicking the models. It's just purely\nlike I'm giving Claude my business\ncontext and that business context is\ndriving how Claude uses the API and to\nset up my application.\nLet me just make sure there are no\nquestions.\nAwesome.\nSo, Meredith, um, this prompt, uh, this\nisn't in the docs, but I really, what I\nreally want to show here is that you\ndon't necessarily need like an MCP\nserver or a skill to use the voice agent\nAPI just with clawed web search and like\npassing it the docs uh, URL for the\nvoice agent API, it can crawl the rest\nof the docs and just with like blank\nclawed code, um, it's going to be able\nto scaffold this back end for\nand the front end just from reading\nreading the docs really. Um, so like I'm\nnot going to be touching any of this\ncode um or doing any like custom prompt\nmagic. It's uh purely me just telling\nClaude what to do and it's able to just\nbind the docs on its own.\nSo we can see Claude is setting\neverything up and I'm just going to\naccept permissions.\nSee here if assembly II serves TTS when\ncan we expect TTS feature to be live for\ncustomers like us as well. So this is\ncoming uh in the future. I I can't say\nwhen but uh like it is in the voice\nagent API. Um so if you need TTS\nuh for a voice agent, we do recommend\nusing the voice agent API. Um but TTS is\ndefinitely on our on our radar and like\nright now this uh we're not offering it\npublicly yet. uh because it's built in\nas part of our voice agent orchestration\nhow the TTS is working. Um but yeah,\nmore to come there. Do watch our change\nlog and be sure to check in on our\nLinkedIn and our YouTube for latest\nupdates.\nOh yeah, Mark, of course. Let me paste\nthe URL in. Sorry about that.\nLet me know if you can see chat to\neveryone.\nMaris, so uh is the claw code you're\nusing customized to build voice agents\nor can I use my personal cla code? This\nis a default claude code. So, uh, I\nhaven't got any MTP servers or skills\nset up. Um, I just use like plain clawed\ncode and I just give it the URL. So, you\ncan definitely use your personal claude\ncode. Um, you can also use the clawed\napp. So, rather than being in your like\ndevelopment environment, you can just\ngive the link to the clawed app like the\nclaude code app. Um, and it can go and\nscaffold uh that the back end and front\nend for you.\nSo now that claude code has gone and\nbuilt our back end and front end and\nit's done the temp token authentication\nuh which is key for like browser voice\nagent applications. Um we now have like\nour local host application ready. So\nlet's run this and uh let's see what\nhappens.\nI think the only thing we will need to\nadd is our API key. Um, so I'll stop\nsharing my screen so I don't leak my API\nkey. But, uh, Claude should be returning\nyou avail.\nSo, make sure you pass your API key\nthere. Uh, so we don't need Pipecat or\nLiveKit if we're using assembly.\nCorrect. Uh, because we're passing or\nlike we're providing the websocket\ntransport, you can just plug this\ndirectly into your application. You\ndon't need to have a separate voice\nagent worker.\nNot sure if this is covered already. Why\nwas there no business context shared? Um\nyeah, yeah, you should also follow\nfollow this as a first step. Uh the\nreason why there's no business context\nshared is because uh this is just like a\na getting started guide um of like\nscaffolding and and getting the\napplication ready. But the idea is that\nyou would then uh take this application\nthat we're building together and let's\nsay you are doing uh appointment\nbookings for uh for dentist uh like\ndentist receptionists uh you can then\ngive that information to claw code and\nhave it like customize and create the UI\ncurate the UI and the prompt uh for that\nbusiness context that you're working\nwith.\nDo we need assembly AI for voice agent?\nUm\nyeah, you would need an assembly AI\naccount to access our voice agent API.\nUm but we are providing like the full uh\npipeline like speechto text LLM and TTS.\nUm but you can indeed go and just take\nour speech to text models and build your\nown voice agent um on another platform\nif you do wish to. But this workshop is\nfor the voice agent API.\nUh Mark question on retention. uh we\nwill be publishing uh that very soon. Uh\nthat's on me really to to get that out.\nSo thank you for the reminder there. And\ndo feel free to email me uh at uh dint\nassemblyai.com\nuh and I can follow up there. But I'll\nfollow up with you after the session as\nwell.\n>> Uh Oliver, your question about security\nand privacy of the data. Um currently we\ndo not store any data so it's just efflu\nduring the session but we are going to\nbe adding session history. Um so for now\nthere's no data stored but we will be\nstoring that and adding a retention\npolicy soon. But uh thanks for thanks\nfor the question.\nOkay great. So I'm going to add in my uh\nAPI key and if you're following along um\ndo the same and we will then run the\napplication together. Uh let me know if\nif anyone's encountering any trouble\nwith that first prompt. Um and uh we can\ntackle it together.\nSo now I have my API key added in my\napplication.\nWe can now\nlet me just rename this\nWe can now run this application\nand we'll see what it has started with\nactivate.\num classic we encounter a first error on\nrun but uh for those who maybe aren't uh\ninto vibe coding yet the recommendation\nhere is just paste that error into\nclaude and it's going to fix it\nMarish are there best practices or user\nguides that you have for prompting\nclaude on your use case um yeah this is\nsomething we'll get into later more\nabout like context textual prompting\nbased on your use case. Uh to give a\nquick answer, um the best way to go\nabout this is if you have like uh like\nexisting humanto human calls of the use\ncase you're going to do. So if you uh\nhave like recordings of like uh\nappointment setting calls um transcribe\nthem with our speechto text model and\nthen passing those transcripts to claude\nas context for how to like prompt the\nvoice agent to speak like uh the the\nhumans do on those calls is uh the way\nwe recommend uh doing it and uh\ndefinitely we can dive more into that uh\nlater on in the session. I'll I'll save\nthat question and we can come back to\nit.\nI see. Um,\nokay, here we go. So, Claude did set up\na Python virtual environment for me. Um,\nif you're encountering any issues with\nthat, do just say to Claude like, \"I'm\nhaving this error. I'm feeling stacked\nright now. What should I do next?\" Um,\nthis is the same way that I pretty much\njust like uh brute force my way through\ncoding uh an agent like this and like\nbuilding web applications is difficult.\nSo, you might be encountering errors,\nbut just paste them back into claude or\nif you're really having trouble, paste\nit into the chat and we can break it\ndown together. Um, but currently what\nwe've done to reiterate the steps is we\nprompted claw to set up the back end and\nthe front end. Um, and now we're just\ngoing to run that locally. Um, so we can\nsee what it's created. So let's see here\nnow.\nSo as we can see Claude's created the uh\nfront end which uh is looking okay now\nand uh I can click to start and begin\nspeaking. So, why don't we\nbring this into our browser?\nWe'll bring it into the browser first.\nOkay,\nlet me share my screen and I will show\nyou the local application we have\nrunning.\nSo, here is where uh it's currently at.\nI can connect and let me make sure I'm\nsharing my audio here for you all.\nHere we can see we've already got a\npretty similar application to the Aura\nStudio demo that we started with. Uh so\nwe're making very quick progress just\nfrom a couple prompts with Claude. So\nlet's start a conversation.\n>> Hi, how can I help you today?\n>> Hey, can you hear me?\n>> Yes, I can hear you loud and clear. How\ncan I help you?\n>> Um I don't know.\nUh but there we can see uh just with one\nprompt and passing the links to the docs\nuh Claude was able to kind of crawl the\ndocs bind our connect from browser guide\nand this is really showing the power of\ncoding agents uh in using APIs. I think\nuh it's just able to like read the all\nthe docs so then you don't have to to\nworry about even reading docs nowadays.\nUm, and especially when it comes to like\naudio transport, uh, like sending audio\nover a web soocket can be really\ncomplex. Uh, but if you have a coding\nagent handling it for you, it really,\nuh, abstracts away that that complexity.\nUm, and fortunately, it's only one uh\nlike link you have to create from\nsending audio from the browser to our\nAPI. Um, so it's it's definitely\nmanageable more with a coding agent um\nthan than trying to kind of manually\ncode it. Um, so a question from Marish,\nwhich languages do you support? Some\nlanguages have dialects in EU like\nPortuguese and Spanish. Uh, what has\nbeen observed is that English or other\ncommon languages work great with voice\nagents. Um so the the languages that we\nsupport are English, Spanish, French,\nGerman, Italian and Portuguese. So these\nare the same languages that are\nsupported by our universal 3 pro\nspeechto text model which we're using in\nthe voice agent API. Um but we are\nexpanding these languages uh in the next\ncouple weeks which will also include uh\nfor the voice agent API. Um and it's a\ngreat point about dialects. Definitely\nsomething we've learned since launching\nthis is that like we have a couple\nSpanish voices, but the feedback we've\nbeen getting is that it's a very generic\nSpanish voice. Uh it's not like it's not\na a local or an accent that is like\nrecognizable to an area. Um so\ndefinitely an interesting learning we\nfound is that like uh it's really\nimportant to have a voice with an accent\nand that's definitely feedback we're\ntaking on board to drive like future\nproduct development. uh for TTS. Um but\nyeah, Brazilian Portuguese and European\nPortuguese is also uh another one and\nlike Latam Spanish versus European\nSpanish. Um so yeah, I'd say right now\ngeneric, but uh the vision of where we\nsee this going is we we want to have\nvoices like with local based accents.\nUm, so just to check in, uh, uh, we've\ngot our application now scaffolded. Um,\nlet me know if anyone's having any\ntrouble setting this up. Um, but we are\npretty much 80% of the way there. Now,\nlet's get on to the final 20% of uh,\nadding the tools uh, to our agent and\nmaking it about appointment setting and\njust generally making it look a bit\nnicer. So let's go back to uh claude\ncode and we can carry on.\nSo because we've already given the link\nuh in the first message that we did for\nthe docs, we can now say can you look in\nthe voice agent API docs how we can add\na create\nappointment tool just simulate the tool\nresult for now. So we will be simulating\nthe results um in this demo just because\nwe are short on time but um in our docs\nwe do have uh code examples for how you\ncan handle the tool uh cool event in\nyour code and have that cool real\nservices uh like uh kel.com or\ncalendarly or maybe a database or CRM\nthat you're using. Um,\nso you can definitely plug this into\nyour real systems and uh and run like uh\nbusiness uh use case calls. Uh, but for\nnow we will just be simulating it just\nto just to uh have an easy getting\nstarted guide.\nSo now Claude uh as you can see it's\nreading through the events uh to call\nreply or started\n>> Morris great question can you uh correct\nand improve the model so that I learned\ncustom words and way to speak. Uh so one\nfeature we do have in our documentation\nfor this is key terms. So you can\nprovide key terms to your voice agent.\nUm which is going to be like the key\nterms that you would want to give are\nrare words that like aren't really in\nany dictionary. So it's not a word that\nlike any typical speech text model could\nhave learned. So if it's like a a new\nproduct skew uh like for example if\nyou've created um some bug repellent and\nit's called like bug gone it might be\nquite difficult for a speech text model\nto to like recognize that out of the box\nbut if you can provide the key term uh\nwhich is also supported by the voice\nagent API it's going to bias the\nspeechto text model to recognize that\nterm and like when it hears something uh\nof that nature it's going to like select\nthe key term that you provide. Um, so\nthat's how you can like train the model\nto understand terms in your business\nthat um aren't generally understanded or\nunderstood.\nUh, a ways this is the assembly\ndocumentation link. I want to create a\nvoice agent. Yeah, that prompt uh\nabsolutely should be enough for Claude\ncode to go and create a voice agent and\nand to to kind of explain why why this\nis. So, Claude code is a really powerful\ntool that isn't just giving the\ndocumentation link to the model. Um, and\nyou know, as you know, like LLMs only\nhave memory from like 2024. So it's kind\nof weird how it can like if we give a\nlink it can um go and understand how to\nuse the API uh and go and build a voice\nagent but because claude code is like\nthis harness on top of the model um so\nit's like a it's built for coding so it\ncan like web search to look up the docs\nit can like scrape the rest of the\ndocumentation\num to to understand what it needs to do.\nJust a prompt like that saying, uh, I\njust want to create a voice agent, um,\nis enough context for Claude to then go\nsearch the documentation and and build\nan application. Um, so yeah, I recommend\npasting that prompt in and do let me\nknow how it goes. It should be all good.\nBroad Opus and Fable have training data\nup to 2026. Oh, awesome. Thanks. Thanks\nfor the heads up.\nDoes this feature only support\nintegration via API or with SIP and\nwebsocket as well? So yeah, currently it\nis websocket API integration, but we do\nhave a SIP pass through if that's\nsomething you're interested in,\ndefinitely give me a message or an email\nand we can help set up a SIP pass\nthrough. Um, so for anyone on the call\nwho is looking to deploy\ndeploy this to Telephony, um, we are\nworking on that right now. And for those\nwho want early access, uh we'd be happy\nto to work with you on setting that up.\nI won't be able to cover this in in this\nworkshop just yet. Um but we can\ndefinitely attach it to a phone number\nand uh we can host it for you.\nIn your experience, are some claw method\nclawed models better than others in\nbuilding voice agents? Claude versus\ncodecs. Um generally\nI would say there's no specific model\nthat's better. um for using the voice\nagent API. Now, if it was for like\nbuilding a voice agent in Pipecat or\nLiveKit where you're going and creating\non your own, I would say that maybe\nthere is some chance the coding model\nyou're using can matter. But for the\nvoice agent API, because we've\nabstracted away a lot of that\ncomplexity, even if you're using, let's\nsay, GPT 4.1, uh, but it was able to\nread our docs, it would be able to go\nand set it up\nfor complex voice agents. I do I do see\nwhat you mean how the model can matter.\nUm I think in terms of configuring the\nvoice agent API it wouldn't be a big\ndifference but maybe if it's more about\nhow do I take you know a thousand hours\nof of humanto human calls and then\nanalyze those calls to learn what prompt\nshould I give my voice agent that is\ngoing to perform the best then maybe a\nmodel like fable um which is the most\npowerful model available would be better\nthere um but I think that's something we\nwe have to explore with and we should do\nsome more research there.\nOkay, so now that we're back in the\ncode, u let's continue with setting up\nthe to uh to Oh, no, sorry. We we\nalready set up the tool calls. Uh what\ndo I mean?\nSo, they run client side. Okay.\nIs the tool call being shown in the\ntranscript output box?\nSo, we're just prompting Claude now to\ndisplay the tool call in the UI like we\ndid on that original demo. Um, and uh\nthis is what I think is really fun and\npowerful about the voice agent API is\nbecause it's so simple to handle these\nevents. Um, we can create like a more\ninteractive experience. Like right now,\na lot of agents are just built to answer\nthe phone and just do like basic tasks.\nBut I think what's really exciting is\nkind of like Siri on an iPhone. Siri\nisn't just um like speaking back to you\nand doing actions. Like it can actually\nmove things on your screen or open an\napp or like while it's thinking it can\ndo like that um like vibration of the\nphone and it makes it like a much more\nintuitive experience. Um, we definitely\nthink like for voice agents to take off,\nit has to be this really cool experience\nwhere it's like event driven and like\nthe voice agent can not just uh\nRobert, that's hilarious. I triggered I\ntriggered this. So, that's hilarious.\nAnd I'm probably doing it again. Um, but\nuh it's really cool how it's like it can\nsee everything on your screen and and do\nthings on your screen as well as um be\nable to speak back. I think like um like\nthe iPhone just has so many uh like\nmodalities that if a voice agent is able\nto use all of them, it like definitely\nmakes a much cooler app. Um I think like\na great example if anyone has used uh\nthe Portoola Tolens's app which is like\nthis companion voice agent friend. Um I\nthink it's like one of the best designed\napps because it's not just a voice a you\ntalk to but like things are are\nhappening and uh for consumer apps I\nthink um it definitely has made it uh a\nlot more fun to use than like other\ncompanion apps just for the fact that it\ncan change things on the screen alone.\nSo now that we've set up the tool um I\nthink this UI isn't looking very pretty.\nSo uh for the original use case it was\nlike a salon booking a haircut but um\nlet's say we are going to build aot\nbooking agent. So that is like uh if\nyour car it has to go through like a\nyearly pass here in the UK. Um so we can\ndo like an MOT um like a car mechanic\nbooking agent and we'll just make the UI\nlook nice as if this is like a hackathon\nproject. So, can you make this aot\nmechanic\nbooking voice agent\nUI? So, as you can see, I'm pretty much\nprompting uh Claude as if I'm like\ntexting a mate. Uh I definitely don't\ntry and do any kind of prompt\nengineering. I just let the context uh\ndo its work uh from searching the\ndocumentation. Um, funny enough, I think\nif you try and direct Claude too much on\nwhat to do and how to go about it, um,\nit can end up like really overfocusing\non that task, uh, when it like really\nClaude kind of knows how to do things\nlike better than we do. And it's just\nabout it's more about giving it\ndirection than telling it what to do.\nSo, as you can see, I'm just giving a\noneliner. Make this a car mechanic UI.\num and it can take uh take that context\nand I'm sure it's going to build a\nlovely looking UI.\nHow do you manage latency especially if\nthe voice agent is configured with\nmultiple APIs?\nSo um firstly I think making sure that\nyour APIs uh that you're executing in\nthe tools making sure that you get as\nmuch performance out of that as you can\nand um I think as well when it comes to\nlike it would it would depend on the\nmulti-step workflow. Um, if it's\nsomething like\nuh checking availability and then\nbooking an appointment in one go, I feel\nlike\nyou can reorganize how you do those\ntools in a more efficient way to cut out\nthe latency. So like rather than doing\nall these steps when someone is saying\nto like book an appointment all that you\nshould be triggering in terms of tool\ncalls is booking it and any background\ntasks that uh can be done like on their\nown that don't necessarily need the\nhuman input or they don't need to know\nthat like I don't say my email is this\nso then book the appointment with this\nemail.\num like you really in instead of trying\nto optimize the latency, try and\nsimplify the tools that you're doing and\nwe've found that that has resulted in\nlike better latency and overall better\ndesign of your voice agent. I definitely\nthink there is uh a lot of problems by\ntrying to add too many tools um where\nminimizing the tools is going to result\nin better tool calling accuracy and\nlatency.\nUh Morris asks, \"How would you get the\nvoice agent to work on your iPhone but\nstill read and write on your local\nharness and context located on your PC?\"\nUm so in this case, you would need to\ndeploy it to uh either like a deploy as\na web application or to a phone number.\nSo once we finish up this UI, we can run\nthrough deploying it to uh Railway,\nwhich is what I use to deploy the demo.\nUm, so then on your phone, you can just\nbring up like Safari or or Google and\nthen go to that link and then you can\nuse it on your phone.\nUh, a how do you spell uh the companion\napp? I can write it in chat for you. is\nuh Poland. You search that on the app\nstore. If you're interested in uh voice\nAI like me and you're just obsessed with\nany kind of voice AI companies and apps,\nI definitely would recommend trying out\nTons. It is really cool and I think it's\none of the best examples of like a UI\ndriven voice agent experience which uh I\nthink is is a market that's going to\nonly grow uh as time goes on. It won't\nbe just phones, it will be like in\nWhatsApp or in a phone app.\nSo\nnow that we refresh the UI, we have a\nreally cool professional looking uh\nvoice agent uh application here. So let\nme bring this into my browser so we can\nsee this a bit better.\nAnd it's pretty crazy. This is only from\nthree or four prompts. Uh if I scroll\nback up, this was one prompt\nand two prompts.\nYeah, four uh prompts essentially. And I\nhave like definitely a pretty nice\nlooking demo that I can uh go and deploy\nand uh share with a colleague or a\nfriend or even try and start my own\nstartup from. So, uh, let me make sure\nI'm sharing my audio and let's run\nthrough a\nlive example with our new agent.\n>> Thanks for calling Apex Auto. Are you\nbooking anote, a service, or a repair\ntoday? Oh, yeah.\n>> I'm looking to book my yearly.\n>> No problem at all. Could I have your\nvehicle registration, please?\nYeah, it's uh E L V44\nA G C.\n>> Got it. That is E LV44\nA G. And when would you like to bring\nthe car in?\n>> Um maybe the Saturday morning.\n>> We are open on Saturday from 8 until 1.\nWhat time on Saturday morning would you\nlike to drop the car off?\n9:00 a.m.\n>> That is all booked in for you. Your MOT\nfor ELV44\nAGC is all set for Saturday, May 24th at\n9:00 a.m. The total will be £54.85.\n>> Amazing. Thank you very much.\nAnd now that we see that was a\nsuccessful tool call, um I actually\nwanted it to fail because what I want to\ngo into very quickly uh next is because\nwe're working with tools and tools are\nlike notoriously difficult uh for voice\nagents. Something I just wanted to touch\non was uh if you have an agent where\nfirst it needs to check the availability\nand then it needs to like book an\nappointment using the context from that\nprevious tool. So, like you wouldn't\nwant the agent to book an appointment\nwithout checking that it's first\navailable. A design pattern that we\nrecommend is that in the handling of\nthat check availability tool, um you\nthen add in the the the agent to be able\nto to to book uh the appointment. So,\nlike let let me bring up uh in the docs\nhere. We call this progressive tool\nreveal. So like now that I've checked my\navailability, I can do a new update to\nmy agent configuration and I can remove\nthe availability tool and then add it\nback to have the book appointment tool.\nAnd that way you're you're going to see\nless hallucinations where the agent\nmight just decide to book the\nappointment without first checking. Um\nand you increase like the accuracy\nbecause you're only giving it one tool.\nUm, it's going to be able to call that\ntool uh much more frequently when it\nshould do.\nAR, does it matter that you're using\nheadphones? I've been struggling with\nnoise when using laptop speakers. Um,\none thing that you might want to check\nyou have enabled is acoustic echo\ncancellation.\nUm, we do recommend enabling it uh if\nyou're not using headphones, and this\nwould be because the speaker is going to\nbe playing out the audio and then the\nmic is going to be picking it back up.\nUm but if you enable uh acoustic uh echo\ncancellation which is available in the\nbrowser. So maybe here I should say can\nyou make sure AEC is enabled\nplease. And let me let me post this into\nthe chat as well. For anyone who is not\nusing headphones make sure you do have\necho cancellation enabled. It uh will\ncompletely solve those issues. And\nyeah, that's the prompt I would\nrecommend giving to claw to enable echo\ncancellation.\nUh, Tan, what are some of the out-of-box\nways you would recommend to decrease AI\nresponse latency? Uh, so for the voice\nagent API, you can control the turn\ndetection. By default, we do have it\nrelatively conservative at 1 second uh\nmin silence and 3 seconds max silence.\nAnd this is because we're focusing\npurely on accuracy for our out the box\ndefault experience. There's nothing more\nannoying than the agent interrupting you\nwhile you're saying, you know, a phone\nnumber and you want to give it your\nphone number to do a booking. But if\nyou're building a different kind of\napplication where maybe you don't need\nto take down someone's phone number or\nemail and you just want really quick\nback and forth uh back and forth\nconversation um you can reduce our uh\nmin silence and max silence parameters\nuh to your needs. So maybe you want to\nreduce min silence to 500 milliseconds.\nThat will cut the response time in half\nto what it is currently.\nUh so that's definitely a quick way to\nto increase the uh or decrease the\nlatency, but do be aware that because\nyou're decreasing the minimum amount of\nsilence for the agent to be uh like it\nso it can claim that it's a turn end. Um\nit does increase the like potential that\nthe agent ends the turn too early. So if\nsomeone is speaking too slow or is\nspelling something too slow, there is\nthat risk. So, make sure that it matches\nyour use case. And we generally\nrecommend and we've seen that more\nconservative settings do result in\nbetter customer outcomes. And sometimes\ntrying to make the agent too fast um\nisn't really worth it. Uh but it it\nmight depend on your customer base.\nUh this is what Claude has come up with.\nCan you check? Yeah, absolutely. Let's\nbring up this application.\nUh\nlet me see.\nLooks fantastic.\nI can start a conversation with it.\nLet's choose Ivy.\n>> Hi there. I'm your voice assistant. What\ncan I help you with today?\n>> Hey, um I don't know. What can you help\nme with?\nI can help you manage your schedule, set\nreminders, check the weather, or even\nfind information on a topic you're\ncurious about. Is there anything\nspecific on your mind right now?\n>> Awesome job always. Uh yeah, you've\nfully set up the scaffolding for the\nvoice agent API. So now you've got a\ngreat base to to to start with and uh\nyeah you can you can just prompt Claude\nto iterate on this and uh tune it for\nyour use case. But it's really cool to\nsee uh that that uh in this session\nyou're able to deploy to Cloudflare\nvoice agent application along with me.\nUm but what I want to go into next uh as\na ways has deployed it to Cloudflare I\nwant to show you all how you can go and\ndeploy uh the application on your own.\nAnd uh what I recommend to use is an app\ncalled Railway.\nSo let me bring up the link here. I will\npost this into chat for everyone. You\ncan sign up free and they have a really\nnice free tier. Um and I'll show you how\nwith the power of Claude. Um we'll\ndouble check that our acoustic echo\ncancellation is enabled, but I'll show\nyou how you can go to deploying this. uh\nto a website link so that you can share\nit with a friend.\nAny advice about using the voice agent\nto do interviews and analyze the data?\nUm so the voice agent API can definitely\nhandle interviews. Um some\nrecommendations that that we would give\nthere is to make sure that your turn\ndetection settings are conservative\nbecause typically in interviews someone\nis going to like take a lot of pauses.\nUh especially if you ask like an\nopen-ended question like tell me about\nyour career history. Someone might take\na couple minutes to answer that. So uh\nwe definitely recommend updating the the\nthe session like mid session to slow it\ndown when asking like an open-ended\nquestion. Uh in terms of documentation,\num we do have an interview agent sample\napp. Um I will remember to send you a\nlink to that. Um and definitely uh we\ncan exchange messages on email or\nLinkedIn to set up an interview agent.\nUm but yeah, we we don't have anything\nin our documentation there, but I really\nappreciate the call out and we should\nadd a sample app for an interview agent\nin there as soon as possible.\nSo, uh, acoustic echo cancellation is\nnow enabled. So, uh, if you were\npreviously having issues, you're not\nusing headphones and the agent was\nhearing itself, you should now see this\nfixed. Uh, if you're following along and\nhave enabled echo cancellation,\ncan the voice agent do a call summary\nafter the call? Um, so in like in terms\nof it being built into the voice agent\nAPI, we currently don't support this,\nbut we do have a product called LLM\ngateway, which you can let's say store\nthe um, you know, we can we can set that\nup right now. Actually, if I bring up\nthe LLM gateway documentation,\nso if you're following along, um I will\npost this is the link to the docs LLM\ngateway uh that if you go to claude and\nyou say can you store\nthe transcripts and at the end of the\ncall then them to LLM gateway and it can\ngenerate a summary\nthat's going to give um\nuh the\nuh coding agent the context of how to\nuse LLM gateway uh and then to to\nsummarize uh the call afterwards. Um\nyeah, we don't have this built into the\nthe product. Um but it is something\nwe're looking to do as we kind of\nmature. Um and um we do have a pretty\nstrong existing like speech\nunderstanding product suite that we'll\nbe using um to uh understand like you\nknow like the sentiment of the call or\num whether a tool call should have been\ncalled or not but definitely recommend\nLLM gateway for all your postprocessing\nneeds. Um so to round things out here um\nafter we set up the LLM gateway summary\nuh we will finish our railway\ndeployment. So I'll share um what we\nneed to do next. So in order to deploy\nto railway we need a GitHub repository\nof this agent that we've created.\nUm, so I might just pause on the LLM\ngateway summary, but say, can you get\nthis ready to deploy on railway\nand create a GitHub\nrepository\nand upload it there so I can deploy? And\nyeah, Railway is great for just if you\nhave your GitHub repository set up and\nyou ask Claude uh to make it ready to\ndeploy on Railway. Um it's going to uh\njust be super easy to then in the\nRailway UI we can deploy that repository\nand we'll run through that as soon as\nClaude is finished.\nUh Marish, how do you measure the\nsuccess/performance of your voice\nagents? though this would definitely be\nwith your LLM gateway uh like scoring at\nthe end of the call but I really do\nthink there's no better judgment than uh\nlike reviewing the call yourself. So um\none feature that we have coming out this\nweek is session history in the\ndashboard. So soon any call that you've\nmade to the voice agent API let's say\ntoday um or like in the future when\nyou're calling it you'll be able to go\ninto your assembly AI dashboard and\nlisten back to the audio of the call so\nof the users channel and the agent's\nchannel see the tool call events and the\npast transcripts um and then you can\nmake a judgment there on uh if the agent\nis performing well or not or you can uh\ntake that session history and pass it to\nLLM gateway to score for you. Um, so\nwe'll be posting documentation about the\nsession history endpoint when it's out,\nbut that that would definitely unlock\nbeing able to score your calls come at\nscale\nand thank you for the brilliant\nquestions.\nTan, by the way, I created a note bot\nfor Discord. We use assembly for our ST.\nAmazing product from you guys. looking\nforward to seeing the company grow. Uh\nthat's awesome to hear and uh that's\ngreat. I I would really use a uh a note\ntakingaking bot for Discord. Uh that's\nawesome. Uh in the past I I built a mini\ntool which was transcribing Discord\nvoice messages. So it's really cool that\nyou got calls working. I couldn't get\nthat to work before myself.\nUh, Morris, could the agent detect the\nperson interviewing you um as not being\nyour voice and answer a question in text\nbut does not react to you answering\ninterviewers question? Um, that would\nnot be supported now to answer the\nquestion in text, but you can kind of\nsimulate that by just muting the agent\nwhen it's responding um because it does\nprovide that text transcript at the end.\nBut um I'll put my email in chat for you\nMorates [snorts] if you would like to uh\nsend me an email and I'd be happy to\ndiscuss uh that further with you of how\nwe can support that.\nSo as Paul is running through getting it\nready to uh be deployed on railway\num it should be finished here in almost\na minute\nand that will take us right to the end.\nAnd thanks to everyone who stuck around\nfor the 58 minutes. This was really fun.\nWhat I will do is I will show um\nhow to deploy to railway on the original\nproject I did. So when you're in railway\nand you've connected your uh account, if\nyou just go to new project and then you\nhave your GitHub connected, this Aura\nvoice booking app was the original demo,\nbut I just told Claw to get it ready for\nbeing able to be deployed on railway and\njust within one click um it's going to\nstart that application for me. I think\nGitHub is having some issues which is\nfrustrating\nbut it is essentially once you have the\nrepository in railway you can just uh\nlike oneclick deploy and then you can\ngenerate a URL uh like this one here\nwith the railway.app and now this is\nsharable and you can share with a\nfriend, share with a colleague. It's\ngoing to stay running 24/7 and really uh\nyou could start a business from it which\nI think is really cool. Um, but we are\nat time everyone. Uh, really appreciate\nuh everyone uh asking questions and\nfollowing along. Uh, it's really cool to\nsee the agency you built. The recording\nwill be available. We'll be posting it I\nthink on LinkedIn or YouTube or Twitter.\nUh, we'll make sure that uh it gets out\nthere and you can see it. And uh, Robert\nuh really cool to hear that uh that you\nfound this great. And uh yeah certainly\nif if you have any questions uh or just\nwant to chat or want to carry on from\nthe session uh please do email me,\nmessage me on LinkedIn uh I'd love to\nhelp you all build something with the\nvoice agent API and do follow along for\nas we continue uh building out the\nproduct and making it even better and uh\nyeah hope you have a great rest of your\nday and uh if you're in London see you\nat the hackathon tomorrow.\nThank you very much everyone. All right,\nhave a great rest of your day. See you\nlater. Bye-bye.",
  "transcript_chars": 46681,
  "ingested_at": "2026-06-18T22:32:33.018047+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": null,
    "like_count": null,
    "channel_id": null,
    "categories": null,
    "tags": null
  }
}