{
  "video_id": "nMORNaE_qe4",
  "channel_slug": "dataindependent",
  "channel_handle": "dataindependent",
  "title": "2 Years of LLM Advice in 35 Minutes (Sully Omar Interview)",
  "duration_seconds": 2944.0,
  "url": "https://www.youtube.com/watch?v=nMORNaE_qe4",
  "upload_date": "",
  "transcript": "it lets you use AI in basically every\nnook and cranny of your day-to-day with\nthat model came out it actually opened\nup a lot of things that you could do we\nuse a lot of different providers and\nthat's because what we've seen with our\ninternal evals is that they're all so\nnuanced and different in like a variety\nof different ways but you also start to\nsee where they lack you'll get to an AI\nproduct you get it to 90% even 95% but\nthat last 51% is nearly impossible how\ndo you think about model distillation\nit's very powerful but you have to be\nvery\n[Laughter]\n[Music]\ncareful I just had an amazing\nconversation with soy Omar the CEO of\ncognus the company behind auto. not only\nis he one of the best llm practitioners\nthat I've met but you can tell he has a\nreally deep feeling for how these models\nare actually working he speaks from\nexperience in this inter interiew we go\nthrough his three tier system of\nactually ranking language models he\nshows us how he uses meta prompts to\ndevelop his real prompts that he uses in\nproduction he also shows us his cursor\ndevelopment flow where he actually has\nthe language model write the test first\nand then write the actual code and\nfinally he walks us through distilling\nperformance from large language models\nto small language models without losing\nperformance let's jump into it and let's\nsee what wisdom our friend suly has to\nshare uh the reason why we're doing this\ninterview here is because I see all the\ncool stuff you're sharing on Twitter and\nI'm like this guy clearly has not only\nlike a checklist learned uh ability to\nmanipulate these models but I can tell\nyou you feel them like you really feel\nhow these things are actually going and\nthe personalities and the nuances and so\nI want to dig in dig into that today\nyeah well thank you and I think it just\ncomes from playing with these things\nevery day day in day out and using them\nand pushing them to their limit and like\nas cliche as it is is just like\nsometimes you got to use them to Vibe\nwith them you know like like right so\nyeah yeah yeah it's so true well I tell\nyou what I want to start off with um one\nframework that I saw you document\nrecently which was your three tier model\nof language models so tier one through\ntier three so could you tell me like\nstarting at tier three what are those\nand how do you work your way up yeah so\nthat's a it's a framework that I I mean\nI don't even know if you want to call it\na framework but it's I like to\ncategorize it them into like based on\nintelligence and price which is\ncorrelated right like the less\nintelligent models are going to be your\ntier three models and then your more\nexpensive slower are going to be your uh\nmore intelligent model so the reason I I\nthought of it in three tiers was because\nof the application purposes so the way\nthat you use something like let's say 01\nso that would be like a tier one is\ndifferent than the way that you use\nsomething like Gemini flash which is\ntier three um and that's because they\nall provide different purposes one is\nsuper cheap super fast the other one's\nlike really smart and really slow so I I\nbroke it down to those three tiers and\nthe third tier is basically what I like\nto call just like the you know the\nWorkhorse the the ones that you're just\nconstantly using 247 and within that\ncategory I think there was three main\nmodels but it's kind of come down to two\nfor me personally so the first one is\nthe one that I think people are probably\nmore familiar with which is GPT 40 mini\nnow that model and is actually like I I\nreally really like it because it lets\nyou use AI in a way that previously you\ncouldn't like if you were to go back\nlet's say six months ago when we had no\ncheap models you had let's say GPT 4 and\nmaybe even Claude\n3.5 there was a lot of scenarios where\nyou couldn't just be like throwing that\nat like random problems like you\ncouldn't just be like hey I have this\nyou know 20page document I want you to\ngo paragraph by paragraph and like\nextract the details because\nrealistically like you know you're going\nto be paying a lot of money so with that\nmodel came out it actually opened up a\nlot of like things that you could do so\nthat was the the the the first one with\nwas gp4 mini and then the other one that\nI'm starting to really like is Flash so\nGemini flash is actually half the price\nof GPT 40 mini and those are the tier\nthree because like I said they they give\nyou a lot of optionality and the\ndifferent things that you could do that\nyou couldn't do before it lets you use\nAI in basically every nook and cranny of\nyour day-to-day right if whether it's\nyour coding and you wanted to look at\nlike you know 50 different files to\nsummarize to help another model for\nexample if you wanted to take a podcast\nand you know look at you know when did\nsomeone say a specific word in that\npodcast right you're not going to go to\na bigger model so that was that's what I\ncall the tier three um and then the\nsecond tier that I have is sort of like\nthe the middle obviously it's the middle\ntier and this is where I like to slot in\nthe actual gp4 Cloud 3.5 Gemini Pro this\nis where I think the majority of people\nyou use these models and and kind of get\nthe maximum usage out of them um and\nthen the last TI is obviously like the\n01 o1 preview and then what I like to\nclassify as thinking models yeah that's\nso cool so I want to dig in more into\nthe use case side so which use case\ntasks are you doing tier two with and\nthen I know that o 01 in tier one is\ngoing to be um it's not just oh I need\nit smarter it's almost like a different\ntype of task you're going to ask it to\ndo so how do you differentiate between\nthose two right so the way that I like\nto differentiate is I like like I pair\nthem so I will use 01 and I use this in\nmy dayto day it's like I'll go to Chad\ngbt and if you just go and say hey like\nto1 can you do this task for me one it's\ngoing to take a little bit of time\nyou're probably going to hit some rate\nlimits because it's highly limited and\nrealistically you're not going to use\nthe model the way that I think it was\nintended somewhat to be used so if you\nsay like hey how's it going like okay\nsure you could use it like that but\nrealistically you're better off using\nyou know the the tier two so how I use\nthe tier 2 is actually the most I use it\nthe most um obviously everyone uses it\nfor coding whether it's CLA 3.5 gp4 um\nusing it for like function calling or to\ncall tool calling like it it is\nobviously like a good balance between\nintelligence and price um and and that's\nkind of like what I use it the most\nwhether I'm writing whether I'm asking\nit to like hey help me edit an email or\nthings like that I'm using those like\nmiddle tier ones now how I actually use\nthat in tandem with 01 is I'll sort of\none of the cases I have is I'll come to\nChad GPT or Claude and I'll sit there\nand I'll just create a giant\nconversation about a specific topic so\nlet's say for example you know I'm deep\ndiving into a research topic and I want\nto learn more about now I'm not going to\nactually go straight into 01 because I\nfeel like one it's a bit slow what I'll\nwhat I'll do is I'll start the topic\nwith gp4 or Claude and I'll like add\nfiles because obviously I think right\nnow 01 doesn't support like files and\nweb search so there's a lot of\ncapabilities that o1 doesn't support and\nwhat I like to call is the context\nbuilding so I will just go and build as\nmuch context in this chat as I possibly\ncan or it could be you know in any\nplatform and and I'll sit there and\niterate I'll actually use voice mode as\nwell to sort of give context it's a lot\nquicker and that's another workflow and\nas soon as I have like you know let's\nsay like two to three pages worth of\ndocuments I'll actually take that and\npaste it into a chat with 01 or 01\npreview and I'll say Hey you know do\nthis gigantic task for me so for example\nI I'll give you one thing to use it for\nis like I was using it to generate use\ncases for my product and I was like okay\nI want to generate use cases and I want\nto understand you know what are some\npotential um customer segments and icps\nit's is like a pretty technical question\nand if I were to just go to 01 and ask\nit that it would have no context it\ndoesn't know what my product is it has\nno clue what my product does who my\ncustomers are and if I were to sit there\nchat with it well I'm going to hit that\nlimit but if I go to Claude or Chad gbt\nI can upload documents I can create this\nbasically a PDF and copy paste it into\n01 and then I can say generate me you\nknow personas icpas it does a lot better\nso that's sort of the the workflow and\nuse case that I have currently running\nwith like the the tier two and the tier\none models yeah yeah yeah one of the\nways that I found 01 works for me really\nwell is around actually D duplication so\nif I have a long list of items that say\nI've processed five different chunks\nwith the same type of workflow for each\nchunk well I'm going to have a list of\nduplicated items I give that whole thing\nto 01 it's actually really good at D\nduplicating and then I'll use one of the\ntier 2 models to do the structured\noutput after that since 01 doesn't yet\nsupport structured output and go from\nthere yeah that actually that's a good\none that's another thing I do as well is\nI'll take 01 and give me like a long\nverbos output and then take that and\nturn it into structured data sets with\nthe uh the tier two and even sometimes\nyou could even get away with using that\nwith the tier three because it's you\ndon't even need to worry about the\noutput you're just like hey I want this\nnicely formatted in whatever shape yeah\nyeah yeah for sure so it sounds like\nyou're using different models across\ndifferent providers too for different\nuse cases or do you stick with one all\nthe time yes so we use a lot of\ndifferent providers and that's because\nwhat we've seen with our internal evals\nis that they're all so nuanced and\ndifferent in like a variety of different\nways so obviously the big one Gemini\nmultimodal right off the bat like\nanything to do with videos or audios\nI'll go you know dive straight into that\nand and kind of use Gemini but you also\nstart to see where they lack so for\nexample a really interesting one is\nGemini models are really good at needle\nin the Hast stack and so if you say hey\nI want you to find one or two pieces of\ninformation in this you know giant long\npiece of text or video it's actually\nreally good but then I started to notice\nthat something like GPT 40 mini is a\nlittle bit of a little bit better\nreasoning over that so if I give it a\nlong piece of context and I say hey I\nwant you to sort of understand the\ncontext of it I saw I found the GPT 40\nmini is a little bit better so you start\nto see where one model does better than\nthe other model in specific area so like\nanother example is Claude 3.5 and GPT 40\nnow Claude is obviously everyone loves\nthat model it's a really good model but\none thing it's absolutely horrible at is\ntool use with structured outputs and\nyou'll start to see this if you it's a\nvery complex tool like I want you to\ncreate the very deep like a a nested\nJson a very you know long structured\noutput like a very large amount of the\ntime it fails and it gives you XML and\nit just breaks all your parsers whereas\nGPT 40 mini does a lot better job but\nthen the caveat is that gp4 o mini is\nnot as good at actually like thinking\nthrough the problem and acting as an\nassistant so there's always these like\ntiny trade-offs that you don't really\nlike notice one of the that we did was\nwe set up a\nlike one of the use case was to get\naround that was we set up Claude and GPT\n40 mini to work together where the tool\nuse for Claude would be to call GPT 40\nmini and we basically system where\nClaude could orchestrate GPT 4 mini to\ncreate the structured output so it would\nsay please do this so the user would say\nI want this task all GP all Claude would\ndo was relay that information to GPT 40\nmini 40 mini creates the structured\noutput and then I guess return so that\nwas like another use of like how we mix\nand match so many models across\ndifferent use cases yeah isn't it wild\nhow all these little mini Vibe tricks we\nhave to kind of like hack together in\nthe early days of llms here and then I\nthink back to how far we've already come\nlike because even like you know like\nJanuary of 23 we're dealing with like\n4,000 token context limits and gbt 3.5\nand all the hacks that we had then we've\nupgraded from them now but we still have\na bunch of hacks like the ones you're\ntalking about and so it just makes me\nthink we're never going to get rid of\nthe hacks and they're always going to to\nbe there for a long time I would say so\ntoo because yeah like you're right it's\nfunny looking back at it the hacks that\nyou used in 2023 were so different you\nwere hacking around context window and\nnow you're hacking around well tool use\nwhich didn't even exist a year ago right\nor like you know a year and a half ago\nso I I agree with you that we're always\ngoing to be Min maxing as a user of\nmultiple models you're going to be Min\nmaxing trying to figure out for your use\ncase for your product for your company\nwhere can I you know masch these\ntogether so that I get the best possible\noutcome for my users um and I know a lot\nof people have and I'm curious what you\nthink a lot of people have spoken about\nlike model routers and how you know at\nthe end of the day like a model is just\ngoing to pick it but my my personal\nopinion is I I think that it's going to\ncause a lot of unintended side like you\nknow side effects but I'm curious what\nyou think on like this whole idea of\nlike model routing because you know\nwe're talking what we're basically doing\nwe're internally with code model routing\nbut I'm curious what you think so\nwhenever I get asked a question like\nthis I think is there behavior in\npractice that tells me um what the\nprediction should be and you just\ndescribe basically you're doing model\nrouting on your own like in in in and of\nitself so that tells me yes model\nrouting will be a thing and I do still\nthink that fine-tuning models and having\nBoke small models is still too much\noverhead like it's really hard to do\nthat and manage them and do with them\nall right now all that is going to get\nso much easier so I would imagine that\nnot only will we have model routing for\ntask specific things against like some\nof the big ones where you have Vibe\nbased feels whether regards to\nstructured output or tool use or\nwhatever it may be but then also um for\ntask specific things um I will\nabsolutely do model routing so um I'm a\nfan I think it's hard I think it will be\nthe future we're not quite there yet\nthough that's for sure gotcha yeah like\nmy my my sentiment there was that there\nand it could be just because the models\njust where we're at right now what I've\nnoticed is and I'm sure you've seen the\nsame is where you'll get to an AI\nproduct you get it to 90% even 95% but\nthat last 5% is last 10 5 10% is nearly\nimpossible I find like it even you can\nrun all the evals you want you can run\nall the benchmarks getting that last 10%\nand I my thought process there is\nthat if you have the model sort of\nchoosing other models that adds to the\nvariance so it causes a lot more\npotential like you know that that's kind\nof where my thinking is and that could\njust be because like we're early like\nrealistically we're so early models have\nyou know multiple generations to get\nbetter uh so that was my thought was\nthat maybe in the future but right now\nprobably not because it's it's so hard\nto get a product in specifically like\nllms into production where you're\nhandling every potential Edge case uh in\na manner that gives you as high of an\naccuracy as you can and adding models\nthat you might not have an eval 4 could\ngive you an output that you didn't\nexpect yeah yeah totally uh well well I\ntell you what one of the other\ninteresting things that came up during\nresearch was your opinion on what is\nkind of becoming known as model\ndistillation so you have a really really\ngood model you perfect the output from\nthere but then you realize wow I can\nactually come up with a little bit of a\nbetter prompt here and give it to a\nsmaller model so that you have it's\nfaster and it's cheaper so can you talk\nme or walk me through how do you think\nabout model distillation in your own\nworkflow yeah so that's a something I\nthink about a lot and it's one of those\nthings where you need to be very careful\nbecause it's very it's very powerful but\nyou have to be very careful because it\nrequires a lot of work and the reason it\nneeds a lot of work is\nbecause you need to have a a good data\nPipeline and understand what you're\ndistilling so one of the things and\nmistakes I made previously with the\nproduct was that we went we had GPT 40\nthis was actually before GPT 40 it was\ngp4 turbo and we used it and it was slow\nand we're like hey let's distill that to\n3.5 open AI has a has a really nice um\nway to do it so we did that and the\nproblem was that we didn't have good\nenough evals we didn't have a good\nenough data set so as the potential you\nknow the various areas grew that people\ncould use the product we would notice\nokay we have to revert back to gp4\nbecause 3.5 was at that time not good\nenough now where I do see distillation\nin our workflow is when you have a\ndefined eval set you have like all your\nbenchmarks and you have a very good data\npipeline where you can say okay\nin this 500 example set I'm using Claude\n3.5 Sonet or or you know 0an for example\nI have my data set and you can use a\nbunch of different there's a lot of\ndifferent companies that provide you\nwith like ways to manage your impr\nprompts and evals whether it's Brain\nTrust or Langs Smith and then you can\nvery accurately uh detect and determine\nthe accuracy of the distilled model then\n10 out of 10 times I would use it um and\nthe easy and it's actually really easy\nlike to actually distill the model down\nit's like it's like it's a single API\ncall the challenging part is making sure\nthat you don't regress your product when\nyou do uh the distillation but I think\nit's one of those things that it's going\nto become more and more apparent as the\ntooling around distillation becomes like\nbetter I know there's a couple companies\nworking on it like open pipe is one of\nthem um and I know open AI straight up\noffers you that so I think as the\ntooling gets better you're going to see\nthis pattern in production\nof companies launching with the biggest\nbest model they collect a bunch of data\nthey have a good e set and engineering\nteam to support that then they go and\nthey distill it to whether open you know\nGPT 40 mini or an open source model yeah\nthat's beautiful my favorite line with\nthat is the whole make it work make it\nright make it fast and so it's like look\nyou're going to use the biggest one to\nstart us off but then you're going to\nmake it fast eventually and go from\nthere um this is awesome I tell you what\nthough so I know you're a practical\nperson I would love to jump into like\nyou actually showing us some of the ways\nthat you use these tools and I think a\nreally cool starting off point would be\nI know that you're a fan of prompt\noptimizers or like meta prompt writing\nand so yes because you had you had a\ntweet and literally said pretty good\nchance you won't be prompting from\nscratch in two to three months so I\nwould love to see the way you kind of\nprompt engineer your way from like an\nidea to like I'm going to go use this\nthing okay yeah hopefully my prediction\nuh ages well because I feel like it's\nbeen a month since I said that and I\ndon't know if we're two to three months\naway from it but okay let me yeah I just\nto add some context I do a lot of this\nsort of meta prompting where I'll come\nin with a problem what is what is meta\nprompting let's start there you come in\nwith a general idea of what you're\ntrying to do you have a problem that\nyou're trying to solve like\nrealistically if you're coming in you\ndon't know what problem you have that\nyou're trying to solve with an AI it's\nit's sort of useless so an example would\nbe um the other day I was trying\nto get uh one of the models to write\nlike me which to this day I I cannot for\nwhatever reason and I was like I came\ninto it and I came into Chad GPT and I\nhad all my examples and I was like okay\nwhat do I write and I normally I would\nwrite something like you know you you\nwrite like a basic promp structure and\nthe reality is that prompt is probably\nnot that good so what meta prompting or\nwhat I like to think about this work\nthis idea is that you come in with an\nidea hey I want to have an AI right like\nme I have examples and then I just give\nthat to 01 or claw and I say please\ncreate the prompt for me and that's sort\nof what I like to think of like this I\ncome in with a a rough idea of what I'm\ntrying to do I don't really know\nspecifically how to optimize it I'll go\nto these models and say hey like\nactually give me this promp structure\nand it does a pretty good job so that's\nkind of the the rough idea of how it\nworks but let's let me should we just\nhop into like yeah I would love to jump\ninto it if you could share your screen\nand then are you using just a regular\nchat interface or are you going to\nanthropics workbench and doing their\nprompt Optimizer I I just used the chat\ninterface because I feel like the prompt\nI mean people some people do use it I\nand I think you can start with it um but\nI just find it easier because I can\niterate a lot better I can say hey start\nlike this and do that so let's actually\ndo it but I I want to start and say do\nyou have some sort of task that like we\nshould we start we should start with\nlike a rough idea because I like do you\nhave any like what what's the task we\ncould Dem let's do a straightforward one\nlet's do what I guess I'll give you a\nfew options you tell me what you think\nis best we could do the classification\none which is very standard hey I have\nsome data sources or can you please\nlabel them for me um we could do either\nlike uh unstructured to structured\nextraction so like extracting insights\nfrom a piece of text or we could do uh\nidea generation that's always a fun one\ntoo okay let's do the let's do the\nextracting text one and I think that's a\ngood one so let's say we I like to\nalways preface it with like the problem\nor what we're trying to do so again what\nI like to come into it is like all right\nI have a problem I'm trying trying to do\na specific task and usually this is like\nmy blank State slate starting point so\nlet's say the task that I'm trying to do\nis I have a large piece of text and I\nwant to you know turn that piece of text\ninto something else some sort of\nstructured output and it's it's funny\nbecause a lot of people say like oh is\nit complicated it's really like I just\ncome to Chad GPT and I or or claw and I\nbasically say that so the way that I go\nis I'll say you know you could use\nClaude or or chagy PT I haven't found\nwhich one is really better again and\nthis is kind of going back to my\noriginal workflow is what I'll do is\nI'll actually start with gp4 or Claude\nand I'll get like a rough idea for a\nprompt and I'll copy that and I'll give\nit to 01 and then I'll start to compare\nacross all three to see which one like\nmakes the most sense so let's say for\nexample in this one I am grabbing\ntranscripts from podcasts and I want to\nknow like you know I want a nice like\nstructured output for all of the key\nexciting moments let's say that that\nlike the problem space so now you could\ncome in and you could create a prompt\nand says okay given this video I want\nyou to do this or I come to CL and say\nlook like and actually the other\nworkflow that I I wish I could demo is I\nuse voice a lot so I don't know if um if\nyou use voice a lot but I've notice that\nwith voice here I don't use it a ton\nyeah it hasn't entered my workflow yet\nbut I'm I'm voice curious so I want to\ntry actually see this let's see this\nokay so I have I have something here I\nwant to show you the whole workflow that\nI use so that I\nso and let me know if you need a\ntranscript I have one handy for us\nactually yeah could you could you toss\nme it there and then I will use it I'll\ncopy paste this okay let me know when\nyou have the transcript and then mm\nsmall plug this is MFM Vault website I\nput together that does insight\nextraction from my first milon there we\ngo Okay cool so let's say our goal is to\nextract insights now my workflow is I\nhave a tool that transcribes it so I\nthink it works so let's say I'll just\nexactly show you how to do it okay hey\nuh I need a bit of help creating a\nprompt uh for a uh use case so what\nwe're doing right now is taking podcast\ntranscripts and trying to extract all of\nthe key moments key insights so I need\nyou to create a a nice uh prompt that\nwill you know help us do that and I'll\nI'll give I'm going to put in the prompt\nas well later on the actual transcript\nbut I need you to create the prompt SL\nsystem\nprompt so boom so that's that's actually\nsort of how I do it it's there's no\nsigns to it and I I'll sit there and\nkind of like here and I'll copy this and\nI'll actually do this I'll go into Chad\nGPT I'll paste it and I'll actually also\nplace it into\nCLA and it's going to go and it's going\nto give me like a uh starting\npoint and so right off the bat like if\nyou're maybe not as good at prompting or\nyou're new to prompting like you can\nread this like obviously if you're more\nexperienced and you kind of know like\nwhat you're doing these kind of prompts\nare like pretty obvious but for a lot of\npeople they'll come in and be like okay\ncool I have a a good starting point so\nthen all I'll do is I'll look at this\nsay okay the following is a podcast\ntranscript identify so and I'll compare\nit to here so right off the bat I don't\nknow if you which one you think is\nbetter but I'm looking at this and I\nlike the claw output better um little\nbit\nmore uh what's it called clear Direction\nso I'll actually copy this and I'll be\nlike okay we have a rough outline I\nliked the first pass I liked the one\nfrom\nuh cloth I'll take that and I'll go back\nto Chad GPT and I'll open up a new tab\nand then I'll\nsay let's go to o1 preview so then I'll\nactually do the same thing um I'll say\nand I'll actually give it more context\nso I'll say something along the lines of\nand again I I'll go back to the voice\nmode here I'll say hey um you're going\nto help me optimize a prompt so I\nalready got another AI model to give me\na rough idea for this prompt I want you\nto look at it and tell me if there's any\nareas in the prompt that we could\nimprove um so I'll give you the prompt\nand I'll actually give you the prompt\nthat I gave to the I AI that generated\nthis\nprompt so it's going to go and then I'm\ngoing to go like this so this is sort of\nhere you\nknow\noriginal prompt to AI I'll paste that in\na sec um what's amazing is just how you\nspeak to it just like a human like it's\nnot complicated it's literally just\nbeing clear in your\ndirections it's something\nthat I recently started to do and\nI think it's a very a lot of people talk\nto the AI as if it's not a human but\nthey perform the best when you just\nspeak to it naturally and I found that\nvoice is the best modality to do that in\nbecause it's very hard to sound robotic\nwhen you're talking to like the the chat\nit's like you have to just talk\nnaturally um and then I found that it's\nit's also a lot faster like if I were to\nsit here and type that it would take me\na lot so here I'll go here I'll T I'll\npaste this um original prompt you know\nand then I'll say Okay cool so I like\nthat one and now this is the second pass\nand now this is where again kind of\ngoing back to the workflow that I use\nright is I'll come in here and iterate\nwith voice on this specific subset of a\nproblem which is generating this kind of\nlike like a prompt we sat there with\ngp24 we sat there with Claude iterated a\nbit um and then I'm I'm like okay I have\na rough idea this prompt looks somewhat\ngood and then I'll come back to 01\npreview and I'll say okay cool I want\nyou to optimize this and I haven't found\nlike I don't have a real scientific\nmethod to which one is best because I\njust kind of sit there and and this is\nkind of where I have like a good first\ngeneration of the prompt realistically\nI'll put this into production I'll write\na couple of like you know uh evals I'll\nsay okay how does this actually perform\nand then kind of iterate back but this\nis sort of my starting point so we'll\nlet this go\num okay so\nhere and then it gives me some\nthings can you please generate the new\nprompt now all right cool it gives me\nthe revised prompt so it it did gives\nyou finally the answer yeah and and sort\nof you can see here and you can OB say\nhere this is just for the sake of this\nand now what I'll do is I will take this\nand then I will actually go to and this\nis my full workflow we can use any model\nbut let's say we're going to use um you\nhave a preference of which model you\nwant to test out the actual uh\ntranscription we can actually do I'd\nlove to hear which one you think and why\nand let's just test it out let's let's\ntest it out so now we go to studio so\nand you see what I mean it's like\nthere's all these different models I'll\ngo to Studio which is Gemini now we're\ngoing to go to Gemini which I found so\nspecifically Gemini Pro uh better at\nsorts of these these sort of tasks um\nand now I'm here with Gemini Pro which\nI'm going to take and grab the prompt I\ncrafted with 01 put it into the system\nprompt of uh what's it called Gemini Pro\npaste in the the transcript and we'll\nsee how it goes all right beautiful yeah\nthat sounds\ngreat all right let's copy this\nhere okay this is how the sausage is\nmade yeah it's it's this is how I like\nto think of like the first generation of\na promp or I'm not really sure where I'm\nstarting off with obviously like is this\nsomething that I would use in production\nprobably not because you want to test it\nout and have a lot of back and forth um\nbut okay cool can I is there a way to\ncopy paste the transcript you're just\ngonna have to select all down at the\nbottom\nthere that would be nice to copy the\ntranscript actually I think I might add\nthat feature in there yeah it's a let me\nsee if I can just this I'm\non all\nright cool now we go grab this okay and\nthen I'll obviously like do a second\npass to make sure that this actually\nmakes sense key moments obviously yeah\nokay this looks pretty good time stamp\nthree to takeaways extract one sentence\ndiscussion themes theme name\num yeah like Okay cool so here I'll\npaste this in and we'll let it we'll let\nit run here so I'm using Gemini\nPro um all right 177,000 tokens and and\nfor for people who are curious like\nGemini Pro\nI I talked about this recently is that a\nlot of models can't actually reason over\na large context like um but for\nsomething like Gemini Pro anything under\n100K tokens it's uh it's pretty good at\nlike being able to synthesize a a\nrelatively intelligent answer so\nhere okay that's really\ncool and now yeah key moments how you\nleverage CrossFit I'm actually curious\nto like see how it this would do against\nlike you know other benchmarks because\nwe don't really know if this is a good\noutput or not and that's where the whole\npoint of evals is but there you go you\nhave how I went from an idea to\ngenerating like a full I guess optim air\nquote here optimize prompt and the\nreason for that is just like for me to\nsit here and write this probably would\nhave taken like an hour hour and a half\nmaybe like give or take depending on how\ngood you are but you know we just did it\nlive in whatever 10 minutes so yeah\nthat's super super cool I love that um\nso then out of curiosity what are you\nusing for prompt management so I saw a\num a tweet by the CEO of prompt layer\nJared and he's like yeah I see everybody\nthey go through the same they go through\nthe same world first their prompts are\njust hard-coded in their code and then\nsecond their prompts are hard-coded in\ntext files but they're still in their\ncode base and then third you actually go\nto a prompt manager what what are you\nusing for prompt management so for\nthat's an interesting one we obviously\nwe use GitHub for our our our prompts\nyeah so we use a lot of a couple of\ndifferent things maybe maybe we're not\nlike we're not prompt managing correctly\nbut we just have our prompts that we\nstore in Langs Smith and sort of I'll\njust have data sets and I'll compare\nthat prompt to that data set so for\nexample we have a giant data set of like\na thousand examples that I I run or test\nagainst different models different\nprompts and that prompt is just like\nstored you know in in the data set and\nthen whenever I want to change the\nprompt I'll actually change it and\nduplicate data set paste in the new\nprompt and like my version so to speak\nso the actual prompt stays in my\ncodebase with the latest version of like\nthis is the the source of Truth and all\nprevious other versions are different\ndata sets where I can see how they\nperform so for example if I want to go\nback to a prompt that was like you know\nlet's say from a week ago I just look at\nthe data set that was from a week ago\nand I can see the prompt is there and I\ncan also see how it performs so that's\nhow I manage uh like an inversion it I'm\nnot sure if that right approach but\nthat's how way I do it sir so in your\ncode is the prompt that's being called\nis it actually in your code or are you\ncalling out to langub and Lang Smith\nevery single time it's in the code so\nthe the our code it's in GitHub and the\nnice part is because it's just all\nVersion Control like I could look at the\ngit history and I can actually see okay\nthis person changed this line is as well\nwhich is nice so I have the line by line\nversion controlled from git um and then\nif I want to see the full prompt I can\nlook back at like a you know the the\ndata management tool yeah that's very\ncool um I tell you what I had one more\ndemo on here that I was like this would\nbe so cool if so Su would show us how we\nuse this um it's a cursor one actually\nso I saw that you tweet you you said do\nI actually have the llm write the test\nfirst then the code it helps a ton which\nthat's a framework I don't see too many\npeople doing of course there's test\ndriven development but like not in\npractice not usually I'm not seeing a\nlot of people do that could you walk us\nthrough like how do you write that test\nfirst and then how do you ask it to\nwrite code right after that yeah okay\nthis is one that I the reason I started\nto do was because the problem I was\nfacing the model just kept messing up\nlike every single time it was within our\ncode base and I was like this is this is\na waste of my time the model can't\nfigure it out how about I just get it to\ngenerate the test first and then if the\ntest works then it can maybe look at the\ncode and say where the issues are\nbecause models guess what if a test\nfails you can grab the error output give\nit back to the model and say hey like\nplease decipher that so let's actually\nsee if I can like I can spin up um like\na little mini project or something or\nyeah yeah let's see here if I can spin\nup something new I actually think this\nis really cool and this is like\nsomething like really truly not enough\npeople are doing this and if it legit\nhelps you write better code because it\nmakes sense you have the test that's\nsupposed to run successfully and it can\nuse that as instructions and it can use\nthat to like test to make sure it's\nactually working I'm surprised not a lot\nof people not more people are doing this\nwhere it's like right that's like it's\njust a lot easier for the llm to like do\nthat and then your code\nis I guess like you know less spaghetti\nbecause you're not you don't you're not\nworried about you know if something\nchanges like the model like you start\nwith the tests and it's really easy for\nthe model to generate it okay so I got I\nthat took a little time I got a uh a\ncursor here so this is just a super\nquick um let me just grab the screen\nhere super quick here so I have this you\nknow super basic thing we can just\nterminal we can run it and I can go you\nknow button\nindex.ts Hello World um now I should\nlike to start with cursor and I'll just\nsay something along the lines of like\nliterally and again I actually don't\nknow how to write tests and fun so I can\njust go to cursor I open up command I\nand for those you don't this is like the\ncomposer it lets you uh coordinate and\ncreate file so I'm going to say you know\nI'm using FN uh for now create a test uh\nfile for a method and then make the\nmethod uh that let's say for now\nreverses a string super simple um and oh\nI guess I'm out of slow request\nunfortunately okay wow so it what it'll\nfirst do is it'll create the test right\nand this is obviously a really simple\nexample and so here I'm happy with this\nall right I'll I'll just accept this um\nand now right off the bat like there's\nyou know how many whatever five tests\nhere so obviously I have the actual\nfunction so here in this example just\nreversing string now the nice part is I\ncan go here I can say you know bun I\nguess it's uh reverse\ntest.ts um and I can again debug with\ncomposer this is a nice part I can just\ngo debug with I\ngot up I got up\nmy man I'm out of the\nfree that's how much I use cursor I I\njust always blow through the budget but\nOkay cool so here it like you know\npasses the test but let's actually say\nthat like we are using something a\nlittle bit more complicated than\nreversing uh a string now I can go into\nhere and I can say let's just not\nreverse it let's just say like let's\njust break the code let's just say here\nwe'll split it like this okay um return\ndot okay so now if I go here I go test\nif I go test file so all these tests\nfail right now obviously this like a\npretty simple example and it's almost as\nsimple as just clicking this button that\nsays add to composer and then I say um\nyou\nknow please fix the reverse method due\nto errors and now the nice part is here\ncursor will pull in that terminal\nthat'll throw you know the errors where\nit happens and what cursor will do is\nthey'll look at that and they'll say hey\nlook I see what the issue is and it'll\njust fix it so this is kind of what I\nlike to call of like I don't actually\nhave a name for it yet maybe llm test\ndriven development whatever you want to\ncall it but it's like you come in and\nyou describe what you're trying to do\nhere the llm writes the tests for for it\nand then it's going to write the method\nand then what you can do is have it run\nand now if the method itself like this\nfunction which is reversing a string is\nis complex or confusing it will be able\nto sort of like essentially agentically\nair quote here fix itself if that makes\nsense it'll test the code see if it\npasses the tests if not it'll update the\ncode and then sort of do that until it\ncan you know pass the test and all you\nhave to do is make sure that the tests\nyou're writing are correct and and I use\nthis a lot for obviously for simple\nfunctions it's not that useful but when\nyou have code that is across a couple\ndifferent files you know in a in a\nmodern code base it's not just a single\nfunction it's like you have like you\nknow a bunch of different files and and\nstuff connecting and ones that require a\nlot of like conditionals or\nlike they're not as simple as this it's\nlike that's where I found that whenever\nI would try to get like cursor or sorry\nI get like son it to oneshot it it would\nfail every single time but then a second\nthat I was like okay please let's write\nthe test for it and then I would sit\nthere and kind of help it write the test\nit was able to debug itself a lot better\nand go through these like bigger maybe\nmeteor functions that normally wouldn't\nbe able to even like 01 and 01 mini\ncouldn't solve but a second that I would\napply this like test driven development\nwhatever you want to call it the model\nwas able to look at the output see where\nit messes up adjust the code and kind of\niterate on itself like that that's cool\nso not only does this test first mindset\num it's kind of like a prompt\nengineering technique it's almost like\nthink out loud but it's almost like\nwrite the goal first and then tell me\nwhat you think we should do for it but\nyou also get tests out the other end and\nso you get a little bit of extra utility\nas a byproduct\nexactly it's a it's a win-win you get a\nlittle bit of both and to me that was\nthe one thing I never understood why\npeople haven't done more of because you\nwould think well if it pass all the\ntests the the code is like you know\nyou're happy that it passed the test but\nit's something that I haven't seen a lot\nof people do yeah yeah for sure well\nthat's awesome well that's fabulous\nthank you for showing me the cursor\nexample one of the questions I love\nasking is I want to know what the smart\npeople are talking about right now like\nin AI so like as You observe on Twitter\nin your circles what are the smart\npeople talking about that's a good\nquestion oh man I\nthink what I see a lot of people talking\nabout is sort of the you know what's it\ncalled like test time compute like 01\nthinking I see a lot of people talking\nabout those I see a lot of people\ntalking about having think those\nthinking models do more agentic sorts of\ntasks um and basically bringing this\nwhat I like to think of as an agent as a\nfor Loop inside to the model uh thinking\nprocess having and training the the\nmodel to just innately be able to call\ntools like and we saw that I think a\ngood example that is uh computer use\nright from anthropic right they they\nobviously fine-tuned in on that so I see\na lot of people talking about that um I\ndo see what I started to notice is\npeople starting to talk about whether\nwe've hit some variation of a wall I\ndon't know if you've seen it too and\nI've hearing a little rumors that you\nknow Cloud 3.5 Opus is not up to par and\nlike the the new Gemini model is not as\ngood so I I've hearing that as well um\nand what else are people really talking\nabout and I think I think we spoke a lot\nabout the other things model\ndistillation um and the other thing I'm\nstarting to see more of is people being\na little bit not I guess talking more\nabout evals like I I think a lot of\npeople didn't really talk about it and\npeople are saying hey like from a\nproduct perspective if you want your\nproduct to be good you need to write\nevals which are just a way of writing\ntest so that's kind of what I seeing and\nI don't know if you've SE anything\ndifferent but just from what I've heard\nfrom people talking yeah let me think is\nthere any anything else I would\nadd to that list um the one thing people\naren't talking about it but I think it\nwill be a big deal when it actually\ncomes out is the whole feature\nengineering um weight manipulation uh\nlike the Golden Gate uh Claude\nanthropic I'm still waiting for access\nto that because that is going to be an\nalternative to prompt engineering and I\nhave no idea like how easy it's going to\nbe to work with what kind of results\nwe're going to get but I'm excited test\nthat whenever it comes out yeah I I I\nremember seeing that I was like I was\nblown away and I kind of forgot about it\nso that I'm actually interested to see\nif they ever will ever let you have that\nmuch inoperability with those models\nlike maybe there's like no no we're good\nsorry we're shelving it like you're not\nallow to touch it right but that be\nreally interesting yeah for sure for\nsure um awesome two more questions here\nlast one I love hearing about what is in\npeople's tool kit so I've seen you use\nExel draw on Excel draw on your YouTube\nvideos I've seen you use repet I've\nheard Rumblings about VZ what else is in\nyour toolkit that is in your kind of\nday-to-day workflows okay okay so\nthere's a lot I guess yeah you got you\ngot a couple V Zer obviously there's\ncursor um excal draw I like it for\ndrawing little diagrams um the other one\nI guess that I use a lot is the\nplayground from anthropic and from open\nAI uh which is like different than chat\nGPT I use that to iterate on prompts\num I use this yeah the the one that I\nuse for transcribing uh the actual audio\nis called whisper flow it's the one\nwhere I like I have a hotkey that I\npress and it takes the voice and\ntranscribes it into the inputs that you\nsaw me use um the other tooling that I\nuse I mean we can go do you want to go\ninto the technical side or are we just\ngoing to leave it at like the high level\nI let's let's not go like I don't want\nto know your entire teex stack but like\nwhat is in like the cool AI stuff that\nlike you're you're you're grabbing for I\nthink that's pretty much it I think um I\nthink you got it there I I there's not\nmany other tools that I honestly use\nlike I just like I a lot of it's yeah\nlike just writing the code Langs Smith\nis one actually I will say that we we we\nuse Lang Smith a lot for eval that's\nlike the other one um but yeah that's\npretty much it from from me I think you\nnailed it vzero cursor excal draw um OBS\nif you're recording videos yeah yeah\nyeah yeah for sure um all right last\nquestion and this is kind of off topic\nfrom the AI side but I know people would\nbe interested in it so you've had a few\nbangers on Twitter like just some things\nthat just absolutely pop and as somebody\nwho does a little bit of Twitter himself\ntoo I can look at a tweet and be like\nthat person thought about it and they\ndid a really good job as to how they\narchitected and constructed it and I\nnoticed that with yourself so what hits\non Twitter and what what's your advice\nfor people who like want to do better on\nit oh man okay so Twitter is just this\nhilarious platform that the algorithm\nchanges a lot so it's you kind of got to\nget a feel for what works and what\ndoesn't and luckily the cost so for\nanyone's looking to grow the cost to\npost on X Twitter is zero like you don't\npay anything if it doesn't do well no\none cares so it's the one platform where\nthe cost is literally zero because\nyou're just typing so type things away\nhow I craft a banger it's like a mixture\nof what I see trending so what I see\nwhat people are talking about and\nthere's two ways to craft a banger one\nis you have to be controversial I'm\nyou're are not going to craft a banger\nif you're not controversial now there's\npros and cons if you're posting that\nkind of stuff all the time people will\nbe like hey you're just posting\nclickbait so you got to be careful with\nit you can't be like this is insane and\nevery single tweet starts with that like\nno one and no one's going to believe you\nbut start saying something controversial\nand the most important part of crafting\na banger is your hook it I can tell like\nhonestly I'll post something and I can\ntell within 20 minutes if it's going to\nbe a banger or not and it's basically\nhow natural does it come that's one\nthat's like how natural did this thought\ncome to me and how well did I craft that\nhook everything in\nbetween like you could you can kind of\nsit there in minmax but the the that's\nhow I sit there and sometimes I'll sit\non something and I'll be like oh man\nlike I just don't know the right way to\nsay it so I won't post it but then it'll\njust come to me and I'll be like all\nright I got this I all the words I'm\nusing the right structure it's like the\nthe right timing and and that's kind of\nwhat goes into crafting it so um the one\npiece of advice that I will give from my\npersonal experience is don't spend too\nmuch time on a tweet because I unless\nyou're doing educational there's there\nshould be a diagram where the more time\nyou spend thinking about a tweet the\nworst it does because I swear the\nmajority of my bangers I spend like 15\nminutes thinking about I'm like all\nright I'm just going to post it you know\ngrab a coffee I come back and end blow\nit up and then all of a sudden you see\n1.4 million\nviews oh man do I have time I have I\nhave to I have to tell you the story of\nhow the started do I have time for that\nyeah yeah let's hear it okay so because\nit's so relevant to the Banger tweet\nso my company we we started like a year\nand a half ago and right this is around\nthe time that agents like people were\ntalking about them but didn't have any\nclue this was let's say March\n2023 and at this time I I was no one\nactually knew of my account I literally\nhad I had been posting tweets and no one\nreplied you know the classic zero views\nyou know that's just what happens and\nthen and I remember I saw someone else\npost something about Auto GPT and I saw\nit and I was like it looks pretty cool\nbut I ignored it and then it came up\nagain and I was like no I can't I cannot\nnot ignore this like this seems\nsomething very interesting and i' been\nbuilding actually like AI projects side\nproject before this and I was like you\nknow what let me like try this thing out\nand obviously I tried it and back then I\nwas like dude this is insane agents AI\nis gonna be crazy so when I was like I\njust posted about it and like I didn't\npost anything crazy and I was like oh\nyeah this is thing is kind of cool it's\npretty crazy and it like got like I\nthink that was the first post that got\nover a thousand likes and I was like\nwait a minute wow and then I was like\nhold up hold a second then I saw this\ntrend that people wanted to do something\nabout like AI agents and it's\ninterestingly enough I like thought back\nto an episode of am like my first\nmillion so funny and and I remember them\ntalking about like there's sometimes you\nsee like this opportunity and I was like\ndude I got to sit here and I got to do\ntwo things first I got to craft\nsomething I got to make a product that\npeople want to use and I got to figure\nout the right Twitter thread and\nnarrative and story to craft to get\npeople on it so that weekend I spent the\nwhole weekend building vzero of cognosis\nwhich was like our previous product in\nthe meantime posting Twitter bangers and\nthreads about how AI agents were going\nto change everyone's life and every\nsingle post was getting like a million\nviews I'm not even exaggerating oh and I\nwas like dude and and I was like okay\nand all I would be posting I was like it\nwas was kind of Click baity I was like\nthis is going to change your life and\nthen getting like million view million\nviews and I post the product like I was\nlike Hey like here I built this thing\nfor you people to go and try because I\nknow from what you've been telling me um\nyou don't want to go through GitHub and\nI and I posted out and it was literally\nbuilt it in like three days and within\nlike two days we got 50,000 users so my\ngoodness that is so crazy the the\ncraziest two weeks and the most\nstressful two weeks of my life and it\nstart started all from how can I craft a\nbanger tweet so I I will say that that\nwas why it's so relevant and so funny it\njust shows how powerful uh writing well\nand writing with the right timing and\nstructure given what's happening can\npotentially you know help you start a\ncompany so and with that that is an\nabsolutely beautiful story to end on\nsuly thank you very much for joining us\ntoday oh dude it it was a pleasure I I\nenjoyed it and hopefully my workflow is\napplicable to other people people can\nlook at it and see that like hey using\nAI is just not that hard you just got to\ntalk to the computer and it'll do stuff\nfor you",
  "transcript_chars": 51176,
  "ingested_at": "2026-05-15T04:40:44.326753+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 44633,
    "like_count": 1338,
    "channel_id": "UCyR2Ct3pDOeZSRyZH5hPO-Q",
    "categories": [
      "Science & Technology"
    ],
    "tags": []
  }
}