{
  "video_id": "J2sbC8X5Pp8",
  "channel_slug": "dataindependent",
  "channel_handle": "dataindependent",
  "title": "World’s Fastest Talking AI: Deepgram + Groq",
  "duration_seconds": 705.0,
  "url": "https://www.youtube.com/watch?v=J2sbC8X5Pp8",
  "upload_date": "",
  "transcript": "what happens when you combine the\nfastest llm with the fastest text to\nspeech model hey my name is Greg hey\nGreg nice to meet you how can I help PTA\nwell you get a really fast AI I teamed\nup with a team at deepgram who sponsored\nthis video to pressure test their new\ntext to speech model but we're just not\ngoing to use any old llm not even GPT\n3.5 turbo will do we're going to use the\nnew grock API with the insanely fast\ntokens per second because I want to find\nout with latency how low can you go\nlet's review the pieces that we'll need\nall right we're going to need three\npieces to build our conversational Ai\nand it all starts with the audio and\nthis is the audio that's coming out of\nmy mouth into my computer's microphone\nso piece number one is we're going to\nneed a speech to text model or a\ntranscription model or an stt model as\nthe cool people like to call it and so\nlet's just say I like cookies well it's\ngoing to tell me hey Greg I like cookies\nand it's going to give me the string\nthat we have here and then with that\nstring we're going to go ahead and we're\ngoing to go pass that to our language\nmodel this is the llm that we are going\nto work with and then from there we're\ngoing to get get a response out the\nother end and it's going to say great me\ntoo they like cookies that that's\nawesome and then finally we're going to\nwrap it up with a text to speech model\nor a TTS and this is going to turn that\ntext that we just got back from the\nlanguage model and this is going to turn\nit back into beautiful audio and of\ncourse this process doesn't just go once\nthere's not just one command we're going\nto Loop this all around until we say an\nexit word or something that says hey\nthis conversation's over then we can\nexit the program so diving in on the\nfirst piece for the speech to text model\nwe're going to use deep gr Nova 2 this\nis their latest model and I found it to\nbe the fastest and the most accurate for\nour use case um so they also support a\nwhole bunch of different ones I thought\nthose was pretty interesting so they\nhave the bass Nova 2 model but then they\nalso have a Nova 2 meeting phone call\nFinance conversational AI these are all\ndifferent Nova models that have been\ntrained for different scenarios for\nexample if I'm doing a drive-thru app\nlike that one down at the bottom right\nthere well maybe I want to use the Nova\ndrive-through because the audio is a lot\nmore choppy who knows all right um they\nalso support streaming which is very\ncool the other really cool thing thing\nthat deep gram does when you do do\nstreaming is it does endp pointing and\nendp pointing is when it's going to\nnotice a natural break in the\nconversation and it's going to say hey\nit looks like the person is done talking\nhere so again I'm going to start\nspeaking I'm going to finish speaking\nand then deep gram is going to notice a\npause and when it notices a pause it's\ngoing to say hey I think Greg just\nstopped speaking I'm going to give you\nan endpoint meaning I'm going to set a\nflag that says I think this is done and\nso what it does is it has a flag within\nthe response that says speech final so\nwhile you're speaking it's going to say\nspeech final equals false but when you\nactually get the final data back after\nit's determined an endpoint it's going\nto say speech final equals true and then\nyou know you can go and continue on with\nthe rest of your app which is nice all\nright I want to show you the Single part\nthat just does this transcription of my\nvoice so you can see how this works the\nimportant part here is going to be this\nasync function called on message and on\nmessage means hey deep gr just sent us a\nlittle chunk of data we should go do\nsomething with it right what are we\ngoing to do with it well we're going to\npull out the sentence that it tells us\nkeep in mind this isn't the full thing\nthat I said but it's just a chunk of\nwhat it was because it sent us to it in\nlittle pieces so we're going to get our\nsentence and if I'm not done speaking so\nif it doesn't find that endpoint that we\njust talked about then I want to add the\nsentence to just a little I made this\ntranscript collector just means like\ncombine all these things together now if\nI am done speaking well add the piece to\nthe sentence go get the full thing that\nI just said and then we're going to go\nprint it out and then we're going to go\nreset the transcript that we had in here\nby the way all this code is available\nlinks in the description go check it out\nall right let's try this out here hi\nokay cool it seems to be like it's\nworking with\nus all right let me try saying a really\nlong sentence problems output debug\nconsole terminal ports yes it still has\nit for\nus see that's pretty cool because after\nI'm done talking it notices a natural\nbreak in the conversation and then it\nwill do the endp pointing for us you\njust need to make sure you don't talk to\nslowly or else it'll pick it up and it\nwon't be it won't be good for us now for\nthe llm piece what we're going to do is\nwe're going to use grock now grock is a\nnew model provider they don't actually\nmake models they're not in the open\nsource model game or any of the uh\nfoundational model game they're getting\nreally good at serving models so they're\nmaking custom chips that they call the\nlpu and they're really really good at\ndoing inference so think of these as\ncustomdesign chips that speed up these\nopen- Source models so they don't make\ntheir own models but but they're going\nto serve them to us really quickly the\ncool part is you can go try these models\nright now now the API I believe is still\nunderweight list but you can go and try\nthem in the UI so let's go check that\nout tell me a long poem about trees\nlet's see what we have here and you can\nsee here it just blows through all these\ntokens 526 tokens per second that is\npretty insane all right then let's check\nout what this looks like on the API side\nso I got two functions here one is in\nbatch and one is in streaming and this\nis just so we can kind of look at this\nbut either way I'm going to say explain\nthe importance of low latency llms let's\nsee how quick this does in batch here\nall right so it's pretty quick which is\nreally nice for all this text this will\nprobably take a while if you did gbt 4\nnow I want to show you the streaming\nside so let me to go ahead and clear\nthis let's do streaming and let's see\nwhat it looks like when we stream it out\nso write me a very long poem about topic\nand that is an insane amount of tokens\num this is really really cool let me\npause that though either way that's the\nllm side and so for the final side we're\ngoing to go from text to speech and the\nway we're going to do this is we're\ngoing to use deep grams Aura streaming\nthis is a new model that is just coming\nout and if you think about it deep\nGram's been in the transcription game\nfor a long time they have access to a\nlot of audio data which is really really\ncool so now they're starting to train\ntheir own models to go from text to\nspeech not just speech to text so what\nwe're going to do is is we're going to\nget our response from grock let's just\nsay it says I went to the park and we're\ngoing to pass that entire thing up to\ndeepgram then they're going to start to\ndo their processing now this is where\nthe cool part of streaming comes in\nbecause I want it to give me its data in\nchunks one at a time because I really\ncare about the time to First Data I I\ndon't want to wait till it's done all\nits processing till the end just to get\nmy first data here so the cool part is\nthat it's going to do its processing and\nthen it's going to send it back to my\napp and then I'm going to measure that\ndistance between when I sent a data and\nwhen it send me its first chunk back and\nthis is the streaming piece and this\nwill be the time to first um time to\nFirst Bite then what I'm going to do is\nright when I get that first chunk then\nI'm going to start playing the audio now\nyou may say well Greg what if it doesn't\nsend us the chunks fast enough us we're\nnot going to have the audio to play for\nus well luckily their models are so\nquick that it's above real time so they\ncan process one second of data in less\nthan 1 second so either way you're going\nto play your audio slower than it\nprocesses it so it works all works out\nin the end so either way it's going to\nsend us more chunks send us more chunks\nand then finally we're going to play the\nlast uh little bit of data here all\nright let's go ahead and see what this\nlooks like on the other end so now we're\non the text to speech side and then\nwe're going to do some cool stuff here\nso this is our streaming request so you\ncan see here we're going to make a post\nrequest we're going to give it the\ndeepgram URL for for the streaming uh\nfor the um text of speech and we're\ngoing to say stream equals true then the\ncool part here is for Chunk in response.\niter content so this is the request uh\nas R here so for iter Content this is\nall the different chunks of data that\nare coming back to us um we're just\ngoing to do a little bit of timing here\nto see when our first bik comes in but\nthen we're going to say hey write this\nchunk of data to our FF play that we had\nup above here beforehand and this is\nwhat will do the actual playing Force so\nit's going to write each chunk which is\nreally really nice okay all right let's\ngo ahead and try this out here the\nreturns for performance are super linear\nthe returns for performance are super\nlinear awesome and as you can see we get\na time to first bite at 272 milliseconds\nand so that's pretty quick all right so\nnow let's take a look at the entire\nprocess we have a conversation manager\nclass here and I have all the rest of\nthe code up here again links in the\ndescription if you want to go check it\nout so with our conversation manager I'm\nfirst going to start off by making a\nlanguage model processor and the reason\nwhy I do this is because I added just a\nlittle bit of memory to this\nconversation via Lang chain so as I'm\nstarting to speak to the bot it's going\nto keep track of those chat messages so\nwe can actually have a fruitful\nconversation then what we're going to do\nis while goodbye not in transcription\nresponse and lower so goodbye is my uh\nexit word so whenever I say goodbye it\nis going to exit the program so first\nwhat we're going to do is we're going to\nre uh hear the transcription and so this\nis uh me actually talking then we're\ngoing to go take what um was just said\nwith the transcription response and\nwe're going to go give that to the llm\nand then once we have that llm response\nwe're going to go give it um to our\ntextt to speech model which is what is\nactually going to do the talking for us\nand then we're just going to reset the\ntranscription response um we're going to\nhave this run basically forever um until\nwe want to stop and we say goodbye so\nlet's go ahead and run this and see what\nit actually looks like hi how's it going\nmy name's\nGreg hello Greg I'm here to chat how can\nI help you\ntoday yeah totally could you tell me a\nstory about the San Francisco\nweather\nsure San Francisco weather can be fogy\nin the morning and sunny in the\nafternoon it's quite\nunpredictable oh thank you for telling\nme and one more question what is my name\nagain your name is Greg wonderful thank\nyou very much\ngoodbye cool and as we take a look at\nhow this goes here you can see the\nlanguage model uh latency we're talking\na little over a second here about a\nsecond and then half a second so these\nare pretty slow in terms of the latency\ncompared to the time to for his bite now\ndeep gram was 230 278 and 300 and\nstaying under 300 is pretty Snappy so\nI'm happy with these results and this\njust goes to show that if you're going\nto optimize anywhere well the language\nmodel is probably your first spot to be\nand this is going to be the number of\ntokens that are produced the other thing\nI want to highlight here too is this\nlatency metric for deep gram also\nincludes the network latency as well so\nif my Wi-Fi is slow this is going to\nslow it down just a little bit now it\nwon't be a ton but that just goes to\nshow that the deepgram processing number\nis probably a little bit faster than\nthis too cool then to finish this off\nhere I just wanted to leave you with a\nfew considerations now what a lot of\npeople will do is they'll put in filler\nwords in their response to users in\norder to buy themselves more time\nbasically artificially putting in words\nto uh disguise the latency so as they're\nresponding or as the llm is thinking the\nvoice might\ngo yeah well okay now I have your answer\nand I'm just going to go for it so\nthat's a way to disguise as opposed to\njust having silence there now the other\nthing that I didn't show here was the\ninterruptions piece now you'll notice\nthat I never interrupted the AI as it\nwas talking the reason why I didn't show\nthat one is well it's pretty difficult\nand it would take a lot more code but\ntwo it's more of a software engineering\nproblem to interrupt a stream of audio\nas opposed to an AI problem which I\nwanted to highlight here but just keep\nthat in mind for when you're building\nyours now the last thing I'll say is um\nYohi I posted a tweet about this and he\nactually said something pretty\ninteresting okay this got me thinking\nwhen I was listening to someone talk I\ncan start forming our response before\nthey are done talking applying it to\nthis somehow stream the speech into the\nmodel so as I'm talking go give that to\nthe llm then have the model predict the\nrest of the user speech so as my first\nhalf of the sentence say is with the\nlanguage model have it predict what the\nsecond half of the sentence is going to\nsay then once it has that form a\nresponse based off the expected user\nspeech so once it knows what I think I'm\ngoing to say then go start generating\nyour response now this is probably\npretty expensive but keep in mind uh the\ncost of tokens and the cost of\nintelligence are all going down to zero\nso this would be a pretty cool bet to\nmake if you started to implement this\nwithin your own applications to start\nbuilding with text to speech head over\nto\ndeep.com TTS to get started and as a\nreminder all this code is in the\ndescription you can go ahead and go grab\nit I'll see you out there on Twitter and\nplease show me your speech models I\nwould love to see them we will see you\nlater my friends",
  "transcript_chars": 14299,
  "ingested_at": "2026-05-15T04:40:57.928178+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 74631,
    "like_count": 2393,
    "channel_id": "UCyR2Ct3pDOeZSRyZH5hPO-Q",
    "categories": [
      "Science & Technology"
    ],
    "tags": [
      "speech to text",
      "voice cloning",
      "deepgram",
      "large language models"
    ]
  }
}