{
  "video_id": "8OJC21T2SL4",
  "channel_slug": "dataindependent",
  "channel_handle": "dataindependent",
  "title": "The 5 Levels Of Text Splitting For Retrieval",
  "duration_seconds": 4140.0,
  "url": "https://www.youtube.com/watch?v=8OJC21T2SL4",
  "upload_date": "",
  "transcript": "one of the most effective strategies to\nimprove the performance of your language\nmodel applications is to split your\nlarge data into smaller chunks the goal\nis to give the language model only the\ninformation that it needs for your task\nand nothing more this practice is the\nArt and Science of text splitting it is\none of the first and most foundational\ndecisions a language model practitioner\nwill need to make text splitting takes a\nminute to learn but in this video you're\ngoing to learn the five levels of text\nsplitting that squeeze out more\nperformance from your language model\napplications using the same data that\nyou already have now there's something\nfor everyone in this video for the\nbeginners we're going to start from the\nvery Basics and for the advanced STS I'm\ngoing to give you plenty that you're\ngoing to want to argue with me on but\neither way I guarantee you're going to\nlearn something along the way this is\ngoing to be a longer video and we're\ngoing to cover a lot but that's on\npurpose I want to take our time and I\nguarantee that if you make it to the end\nyou're going to have a solid grasp on\nchunking Theory strategies and resources\nto go learn more for those that are just\njoining us my name is Greg and I'm\nexploring the AI space through the lens\nof business value you see models are\ncool stats are cool but I want to find\nout how businesses will actually be\ntaking advantage of AI and language\nmodels this video will be split up into\nsix different sections first we're going\nto talk about Theory we'll talk about\nwhat splitting and chunking are why we\nneed them and why they're important I\neven made a cool tool called chunk\nvi.com to help us visualize along the\nway then we're going to jump into the\nfive levels of text splitting for each\nlevel we're going to progressively get\nmore complex and introduce topics along\nthe way for you to consider when you're\nbuilding your own language model\napplications for level one we're going\nto talk about character splitting this\nis when you split your documents by a\nstatic character limit for level two\nwe're going to talk about recursive\ncharacter text splitting this is when\nyou start with your long document and\nthen recursively go through it and split\nit by a different list of separators for\nlevel three we're going to talk about\ndocument specific text splitting so if\nyou have python Docs or JavaScript docs\nor maybe PDFs with images we're going to\ninclude multimodal in this level as well\nfor level four this is where it gets\ninteresting we're going to talk about\nsemantic splitting so the first three\nlevels were all naive ways of splitting\nthese levels focused on the physical\npositioning and structure of the text\nchunks these first three levels it's a\nbit like sorting a library based off the\nbook sizes and shelf space rather than\nthe actual content of the books but in\nlevel four here we're not just going to\nlook at where the text sits or its\nstructure instead we're going to start\nto delve into the what and the why of\nthe text the actual meaning and context\nof these chunks it's like understanding\nand categorizing the books by their\ngenre and themes instead and then with\nlevel five we're going to talk about a\ngentic splitting so we're going to look\nat an experimental method where you\nactually build an agent-like system\nthat's going to review our text and\nsplit it for us and then to finish it\noff we're going to end with some dessert\na bonus level that shows the advanced\ntactics that start to creep a little\nBeyond Tech splitting but are going to\nbe important for your overall knowledge\nabout how to do retrieval in general my\ngoal isn't to prescribe the best or most\npowerful method you'll see why that's\nactually not possible my goal is to\nexpose you to the different strategies\nand considerations of splitting your own\ndata so you're able to make a more\ninformed decision when you're building\nthis is part of a larger series on\nretrieval I ofo and if you want to check\nout more or get the code for this\ncontent head over to fullstack\nretrieval.com and I can go send to\nlastly I do a lot of workshops with\nindividuals and teams if you your team\nor your company want to chat live or do\na custom Workshop just feel free to\nreach out so without further Ado let's\njump into it first we're going to start\noff with a theory behind text splitting\nwhat is it and why do we even need to do\nit in the first place you see well\napplications are better when you give it\nyour own data or maybe your user's data\nbut you can't pass unlimited data to\nyour language model and there's two main\nreasons for this number one applications\nhave a context limit this is an upper\nbound on the amount of data that you can\nactually give to a language model you\ncan see the context windows on open AI\nwebsites for their own models and number\ntwo language models do better when you\nincrease the signal to noise ratio let's\nsee what Anton co-founder of chroma has\nto say about this distracting\ninformation in the model's context\nwindow does tend to measurably destroy\nthe performance of the overall\napplication so instead of giving your\nlanguage model the kitchen sink and\nhoping the language model can figure it\nout you want to prune the fluff from\nyour data whenever possible Now text\nsplitting or chunking is the process of\nsplitting your data into smaller pieces\nso you can make it optimal for your task\nand your language model now I really\nwant to emphasize this point the whole\ngoal of splitting your text is to best\nprepare it for the task that you\nactually have at hand so rather than\nstarting with hey how should I chunk my\ndata your question should really be\nwhat's the optimal way for me to pass\nthe data that my language model needs\nfor my task our goal is not to chunk\njust for chunking sake our goal is to\nget the data in a format where it can be\nretrieved for Value later so let's talk\nabout retrieval in general so in the\nbigger picture the act of gathering the\nright information for your language\nmodels is called retrieval this is the\norchestration of tools and techniques to\nsurface up what your language model\nactually needs to complete its task\nlet's take a look at where chunking fits\ninto the retrieval process so here we're\ntaking a look at the full stack\nretrieval process we have everything\nfrom your raw data sources to your\nresponse here if you want an overview\nabout this entire process head over to\nfullstack retrieval.com where I do a\nseparate tutorial on this now the\nimportant part is we're all going to\nhave our raw data sources down at the\nbottom here and they eventually need to\nmake it into our knowledge base right\nhowever we can't just put our raw data\nsources we're going to need to chunk\nthem which is what this video is about\nnow right when you do your data loading\nthis is where your chunking strategy is\ngoing to come into play how you choose\nto split up your documents is a very\nimportant decision as you go through\nthis you'll see that there isn't one\nright way to do your chunking strategy\nor really your retrieval strategy for\nthat matter for example take a look at\nthis tweet from Robert hir for those\nthat need a translation Robert is\nbasically saying that he employs many\nalternative strategies across his\nretrieval stack what works for him may\nnot work for you the last thing I'll\ncomment on is the topic of evaluations\nevaluations are super important when\nyou're developing your language model\napplications you won't know if your\nperformance is improving without\nrigorous testing one of the most popular\nretrieval evaluation Frameworks out\nthere is ragas I encourage you to go\ncheck it out I won't be covering those\ntoday because it's more of a retrieval\ntopic rather than this narrow Niche that\nwe're going to be covering today plus\nthey're very domain specific and\napplication specific if you want my\ntaken evals please head over to\nfullstack retrieval.com and you'll get a\nnotice when I start to cover it all\nright now that is enough talking for now\nI finally want to get into some code\nlet's move on to level one character\nsplit all right so level one charact\ncharacter splitting before we jump into\nthat I want to talk about the chunking\ncommandment your goal is not to chunk\nfor chunking sake your goal is to get\nour data in a format where it can be\nretrieved for Value later I'm placing so\nmuch emphasis on this point because it\ndoesn't matter what your chunking\nstrategy is if it doesn't serve your\nDownstream task keep that in mind as we\nkeep going here so level one character\nsplitting this is the most basic form of\nsplitting and this is when you're going\nto chunk up your text by a fixed static\ncharacter length let's talk about what\nthat means first the pros it's extremely\nsimple and easy the cons it's very rigid\nand doesn't take into account the\nstructure of your text and to be honest\nI don't know anybody that does this in\nproduction I don't let me know if you do\ncuz I'm curious the two concepts I want\nto talk about for this one we're going\nto see what chunk size is and we're\ngoing to talk about what chunk overlap\nis but let's use examples to explain\nthose for our text this is the text I\nwould like to Chunk Up it is an example\ntext for this exercise cool we got that\none so before we talk about packages\nthat help us do this automatically I\nwant to show you how we do this manually\nfirst just so you can appreciate the\nnuances for how cool some of the stuff\nis so in order to create our chunks I'm\ngoing to first create an empty list of\nchunks now my chunk size is going to be\n35 this stands for 35 characters so I'm\ngoing to count 35 characters in count\nthat as chunk one next 35 characters is\nchunk two I'm going to run through this\nI'm going to create a range and the\nrange length is going to be the length\nof my text that I have up above here and\nthen um the iteration step or the the\nstep we're going to take is the chunk\nsize so we're going to skip ahead every\n35 characters I'm going to get the chunk\nI'm going to unpend it and let's see\nwhat our chunks are this is the text I\nwould like to CH Unk up it is an example\ntext for this exercise first off\ncongratulations you just did your first\nchunking exercise do you feel like a\nlanguage model practitioner yet I sure\ndo let's keep on going here there's a\ncouple problems with this this is the\ntext I would like to Chu well it's stuck\nin the middle middle of the word that's\nno good um how do how are we supposed to\nknow how long this 35 character length\nwe could change this to 40 and then all\nof a sudden this goes a little bit\nfurther but dang again we have it one\nmore time that's giving us a hard time\nthat's not too good um so yes it's quick\nyes it's simple but uh we need to fix\nthis problem now before we move on to\nlevel two I want to talk about Lang\nChain's character splitter so their\ncharacter splitter that's going to do\nthe exact same thing for us but it's\ngoing to be through a lang chain\noneliner and the way that you do that is\nyou're going to initialize a character\ntext splitter you're going to tell it\nhow much of a chunk size you want you're\ngoing to tell it how much of a chunk\noverlap you want we'll talk about that\nin a second and when you do a blank or\nan empty string as a separator we'll\ntalk about that in a second too that\nmeans it's just going to split by\ncharacter and Lang Chain by default they\nwill strip the white space and so they\nwill remove the uh the spaces on the end\nof your chunks I don't want them to do\nthat quite yet so I'm going to say false\nso this is we just made our character\nsplitter now we're actually going to go\nand split the documents and so I'm going\nto say dot create documents and this\ncreate documents function it expects a\nlist and because our string up above is\njust a plain old string I need to wrap\nit in a list right here let's go do that\ntext splitter so now what we get\nreturned is we get three chunks however\nthey look a little bit different than\nplain strings the reason why is because\nthey're actually a document object now\nin Lang chain a document object is uh\nwell an object that holds a string but\nit can also hold metadata which is\nimportant for us to understand when we\nstart doing more Advanced Techniques so\ndon't get scared documents uh still have\nour string and they're held within page\ncontent this is a text I would like to\nchump it's same thing that we had up\nabove cool that makes sense so let's\ntalk about overlaps and separators here\nso I'm going to make the character\nsplitter again it's going to be 35\ncharacters but this time we're going to\nhave a chunk overlap of four now what\nthis means is that the tail end of Chunk\nnumber one is going to overlap a little\nbit with the head or the beginning of\nChunk number two and the overlap is\ngoing to be 4 characters so the last\nfour characters of Chunk one will be the\nsame four characters of Chunk two let's\nsee what this looks like here and again\nI'm just going to make these and what we\nhave is this is the chunk or this is the\ntext that I would like to we still have\nthe first same split the chunk size is\nthe same but check this out now the\nfirst four characters of Chunk number\ntwo are the same four characters as\nchunk number one because we have that\nchunk overlap all right now when I was\ngetting ready for this exercise I\nthought I remember a tool that actually\nvisually showed you different chunking\nTech techniques with uh highlights of\nthe different chunks but I couldn't find\nit so I ended up making a tool and that\ntool is called chunk viz.com and this is\njust a quick snippet of it but I want to\nshow you this while we're on the topic\nso chunk number one was this first\nbeginning part and then we have the\noverlap and then we have chunk number\ntwo and then the overlap and then chunk\nnumber three but if you want to try this\nout for yourself it's kind of cool you\ncan go to chunk fis.com and then what\nyou'll get here is you'll get a tool\nwhere you can input um different text\nand uh play with your different chunk\nsizes that we get get out so let's go\nback up and let's grab the text that we\nhad I'm going to bring this over here\nI'm just going to replace this and so\nyou can see here that right now our\nchunk size is one with no overlap that\ndoesn't make any sense although it's\nvisually cool it won't do any good for\nus because we have 83 chunks here what\nare you going to do with 83 one\ncharacter chunks I don't know I'm not\ngoing to do much with there but as you\nstart to increase this number you can\nsee that these different chunk sizes are\ngoing to start to get bigger so we're\ngoing to take this all the way up and as\nwe go through I'm going to put it at\nwhat we had it before which is 35 so\nthis is a text that I would like to and\nyou see it ends right in the middle of\nthe CH just like we had beforehand let\nme zoom in just a little bit more here\nit ends just in the CH that we had\nbeforehand now if I start to introduce\noverlap you can see that now we have a\nlittle bit of an overlapping section so\nwe had chunk oversiz or chunk overlap\nbefore this is the text that I would\nlike to so it chunk one still ends here\nbut now there's the overlap that comes\nwith it so what's cool is you can switch\nthis around to yourself you see above a\ncertain mark it ends up being I want to\nget rid of this over\nover a certain Mark well this chunk size\nis bigger than the document that we have\nso it's just going to encapsulate\neverything but you can go to chunk\nvi.com and go play around with this\nwe'll take a look at one more of these\nsections uh in a minute here cool so\nlet's go down there those are characters\nthose are separators fabulous and so the\nnext thing I want to talk about is\nseparator so beforehand we just had a\nblank string as a separator which means\nyou're going to split by character right\nhowever if we specify another separator\nin this case I'm going to do CH well\nlet's see what that says here this is\nthe text that I would like to and you\nsee here that the CH is missing and the\nspace is missing because I removed the\nuh strip whites space and so it's\nDefault true so that's gone and you see\nthat this word is supposed to be chunk\nup but it's not anymore because we\nremoved the CH it's not too helpful when\nwe do ch um you could do the letter e if\nyou wanted and it's going to be a little\nbit different either way unless you know\nexactly what you're doing I wouldn't\nsuggest messing around with the\nseparator to try to get better results\nhere all right so that's the Lang chain\nside of the house the next next one I\nwant to show you is the Llama index side\nof the house so they have what they call\na sentence splitter but also I'm going\nto use their simple directory reader\nbecause this time instead of just using\na a static string that I put in the code\nwe're actually going to load some essays\nfrom a directory so I'm going to make\nthe sentence splitter and this time I'm\ngoing to have a chunk size of 200 and a\nchunk overlap of 15 and then I'm going\nto load up some essays so my input files\nI'm just going to load up one essay this\ndata is also in the repo so if you go\nand clone this repo you can also get\nthis data pretty easily this is going to\nbe a Paul Graham essay and this is going\nto be his MIT essay so we can go check\nout Paul Graham's MIT essay and we can\nsee what it is right here you can go and\nread it I have it loaded up for you so\nlet's go ahead and load this now we have\nour documents but this document is just\ngoing to be let's let me show you here\nlet's check\nout let's check how long this document\nis and it's just one big long document\nbecause the entire essay was loaded into\nthis variable here however we want to\nchunk it up and the way we're going to\nchunk it up is with our split splitter\nwe're going to say get nodes from\ndocuments now you may be asking hey Greg\nwait what's a node well llama index is\nnomenclature for a chunk or a subsection\nof a doc is going to be nodes and so\nthat's what we're going to get here same\nthing so now that we have our nodes I\nwant to take a look at one so as you can\nsee this node is quite long based off of\nthe amount of text that's in here but\nthere's some really cool information\nthat comes out of the box so first of\nall this node has an ID and we can see\nhere that it's a text node because they\ndelineate from other nodes so we have a\nnode ID so we can go and uh use it later\nand deal with its uniqueness and then we\nalso have some metadata so we can tell\nwhere it came from we can have last\nmodified date Etc but then one of the\nother parts I like a lot is going to be\naround node relationships so here we\nhave a relationships key and we can take\na look at other relationships this node\nhas so node relationship we can see the\nsource node that it came from but then\nalso we can take a look at the next node\nso what node actually comes next and\nthis is really helpful for when you\nstart doing some traversing across your\ndocuments but either way I won't go too\nfar into that one well congratulations\nwe just finished level number one let's\nhead off to level number two recursive\ncharacter text splitting so you'll\nnotice that in our previous level one we\nsplit by a static number of Chunk sizes\neach time so 35 characters by 35\ncharacters however there's other\nchunking mechanisms that will actually\nlook at the physical structure of your\ntext and it will infer what type of\nChunk sizes you should have so instead\nof specifying by 35 characters you can\nthen specify well give me every new line\nor give me every double new line and\nthat's what recursive character text\nsplitter does so what it's going to do\nis it's actually going to have a series\nof separators and it's going to\nrecursively go through these documents\nand it's going to start at its first\nseparator and it's going to first Chunk\nUp by every new double new line that you\nhave here for any chunks that are still\ntoo large after that first iteration\nit's going to go to its next uh\nseparator and that's going to be just\nnew lines and then it's going to go to\nspaces and then it's going to go to\ncharacters so now I don't need to\nspecify 35 characters or 200 all I can\ndo is I can just pass it my text and it\nwill infer what the structure should be\nnow the cool part about this one is if\nyou think about how you write text\nyou're probably going to separate your\nideas by paragraphs and those paragraphs\nare separated by double new lines this\nmethod takes advantage of that fact and\nso we can start to be smart about which\nseparators that we use to take advantage\nof how humans naturally write a text all\nright so let's go ahead and let's check\nthis out we're going to do Lang chain\ntext splitter we're going to do the\nrecursive character text splitter now\nagain I'm going to take some text but\nthis one's going to be a little bit\nlonger that we have here right and so\nlet's do that text and then let's put it\nthrough our recursive character text\nsplitter this is going to be a 65 uh\ncharacter limit we're going to pass it\nthrough and then all of a sudden you can\nsee that we get a whole bunch of\ndifferent chunks here now I'm not even\nsure how many chunks let's see how many\nwe actually have we have six 16\ndifferent chunks all right so with that\none of the most important things I\ndidn't understand about the for end of\nsentence end of sentence day you can see\nhere that what's cool is that we're\nending on Words quite often and that's\nbecause words have spaces in between\nthem and that is one of the separators\nthat we try out so this is cool now\nwe're not splitting in between words\nanymore however we are starting to split\nin between sentences that's not so good\nthat's not so fun um one of the ways\nthat we can combat that is we can\nincrease the chunk size because our\nhypothesis is that if we inre increase\nthe chunk size we can start to take\nadvantage of the paragraph splits a\nlittle bit more all right so now what\nI'm going to do is I'm going to increase\nthe chunk size to 450 still chunk\noverlap of zero and let's see what we\nhave here one of the most important\nthings I didn't understand about the\nworld with without child was the degree\nto which performances are super linear\ncool so there's a period here's a period\nand here's the end of it now what's\ninteresting is all three of those let me\nscroll down to this viz again this is\nthe same exact uh string that we have\nthose are all three different paragraph\nbreaks hm that's pretty interesting so\none of the important things to note here\nis look how these these paragraphs are\ndifferent lengths if I use level one in\nthe 35 character split or any character\nsplit I'd start to cut in the middle of\nthem but now I can get these paragraphs\ngrouped together and the hypothesis\nbehind this method is that these\nparagraphs will hold semantically\nsimilar information that should be held\ntogether so the recursive character text\nthis is pretty awesome let's go take a\nlook at what this looks like at chunk\nvi.com I'm just going to go ahead and\ncopy this go back to chunk fis.com let\nme put this text in there and you can\nsee if we did the character splitter of\n35 uh text we have 26 different chunks\nwe're chunking all over the place this\ndoesn't make any sense but what I'm\ngoing to do is I'm going to actually\nscroll down I'm going to select the\nrecursive character text splitter and\nnow I still have a chunk size of 35 but\nwe're going to increase this one so the\nfirst thing I want to show you is as I\nstart to increase this notice how I go\nbetween 35 and 36 the split the first\nchunk here doesn't switch sizes that's\nbecause it's looking for the space to\nactually split on and because there's a\nspace here it snaps to the nearest word\nthis is why it's so cool so I'm going to\nincrease this let's see when it finally\ndoes split and there it goes it just\njumped up to a degree but either way let\nme save you for this here I'm going to\nselect the chunk size I think we wanted\nmaybe like 450 I forget what it was let\nme zoom out just a little bit let's go\nfour maybe four I mean either way look\nit now we're splitting all these three\ndifferent paragraphs and we can even\nincrease the size and it doesn't really\ndo much for us but all of a sudden if I\ngo too big well it's going to chunk the\nfirst two paragraphs together because\nthat's um this 493 is around the size of\nthis these two combined anyway you can\ngo and split this again you can get so\nbig that it finally takes over the third\nparagraph all right so that's it for\nlevel two congratulations recursive\ncharacter text splitter if I'm starting\na project this is my go-to splitter that\nI use each time the ROI for your energy\nto split up your docks is pretty awesome\nit's a oneliner it goes really quick\nthere's no extra processing that's\nneeded so if you're looking for a go-to\nplace to start I recommend with level\ntwo the recursive character text\nsplitter let's move on to level three\ndocument specific splitting so up until\nnow we've been splitting just regular\nOld Pros we've had some static strings\nand we have had some Paul gram essays\nbut what if you have markdown what if\nyou have python docs what if you have uh\nJavaScript docs there's probably a\nbetter way to split on those because we\ncan infer more about the document\nstructure from special characters within\nthose documents because when you have\ncode you start to have some code\nformatters and we can take advantage of\nthose\nall right so the first I want to look at\nhere is for markdown so we're still\ngoing to have something that's like the\nrecursive character text splitter\nhowever we have a lot more separators\nnow the reason why this is so cool is\nbecause let's take a look at this first\nseparator it's a new line and then it's\ngoing to be a pound symbol which\nindicates a heading within markdown and\nthis Rex here means one pound symbol\nbetween one and six times so it's a new\nline followed by a header H1 through H6\nwhy would we do this well headers\nusually denot what what you're going to\nbe talking about so this is a cool way\nto try to group similar items together\nso these are the Lang chain Splitters if\nyou have your own different package you\nmight see other Splitters but if you\nwant to see the Lang chain side of the\nhouse you can head over to their GitHub\nand you can go find on Lang chain Libs\nLang chain Lang chain texts spitter dopy\nyou can see that they have a markdown\nlanguage and here are the Splitters that\nthey actually end up using all right so\nlet's go ahead and load up our Lang\nchain markdown splitter I'm going to do\na chunk size of 40 which is again this\nis really really small my first go-to\nfor chunk sizes is going to be anywhere\nbetween 2,000 4,000 5,000 6,000\ndifferent chunks and as contact lengths\nfor language models get better and their\nperformance with large context gets even\nbetter well you're going to start to\nincrease this a whole lot because it can\ninfer what you want there all right so\nlet's go ahead and do that and then so\nhere's some uh markdown text fun in\nCalifornia uh H2 driving blah blah blah\nblah blah blah cool and let's split\nthese up and so we can see here uh the\nfirst document is fun and California\ndriving and what's cool is that it split\nit on these headers and so split on\nheader here etc etc split on header here\nwhich is nice that's markdown you can do\nthis also for python but instead of\nusing the markdown Splitters you're\ngoing to want to have your own python\nSplitters so again Lang chain is going\nto split on classes uh functions\nindenting functions so these might be\nmethods within your class or you might\nhave double new lines new lines spaces\nand characters they have a python code\ntext splitter let's go ahead and run\nthis let's see what we got chunk size of\nand we scroll and we can see that we\nhave uh this whole class is enveloped in\nwithin one document which is cool\nbecause that's what we'd want but then\nwe have uh P1 equals John equals person\nwe have this stuff and we have the\nranges right here um I put this over on\nthe uh chunk uh chunk fizz.com and you\ncan go ahead and you can throw this in\nthere you can go to python Splitters and\nlet's see how how long this that was 100\nlet's go ahead and bump this up to\n100 we have it at 100 right here and\nLang chain I couldn't figure out how to\nundo the strip whites space with them\nwithin the JavaScript version to make\nthis tool which is why I'm a little\nhesitant to show it but either way you\ncan see that we're splitting right there\nthat's what we'd want nice all right\nsame thing we have uh JavaScript uh we\nalso have just a bunch of different um\nseparators that we have here this is\nvery similar but in this case we're\ngoing to uh do our recursive character\ntext splitter again but now we're going\nto specify which language we want it to\nsplit by so here's our text recursive\ncharacter text splitter. from language\nthis time and we're going to do a\nlanguage which is going to be language.\nJS chunk size 65 let's go through there\nlet's do it and then all of a sudden we\nsplit up our JavaScript code as well\ncool those are all strings those are all\npretty easy because they might just be\nin txt files which is simple enough for\nus to work with however what if you have\nPDFs everyone loves talking about PDFs\nthey especially love talking about\npulling tables from PDFs this is because\nthere's a lot of old school industries\nthat still put information inside a PDF\nso when it comes to chunking you're not\nonly just going to split text but you\njust want to pull out all the different\nelements within your documents and some\nof those might actually be tables it\nmight be pictures it might be graphs\nlet's take a look at how we do this here\nso the way I'm going to do this is I'm\ngoing to load up a PDF right here and\nthis is just going to be a Salesforce\nFinancial PDF I went over and I just\npulled one of the random PDFs that we\nhave here so I can have a table all\nright um so we have this and the way I'm\ngoing to do it is I'm actually going to\ndo it via unstructured now unstructured\nis another Library we can check go check\nthem out they're at unstructured doio we\nget your data llm ready so they have\nsome really cool parsers that you can\nuse which is going to be advantageous\nfor when you start to get more\ncomplicated data types or your data gets\na whole lot more Messier so for example\nif you had a million PDFs that you\nsomehow needed to get into a structured\nform unstructured would be who you win\nwith that not a sponsored video by them\nat all all right so we're going to load\nup two uh Elements by them it's going to\nbe partition PDF and then elements to\nJson we're going to get our PDF and if\nwe take a look at this Salesforce PDF\nhere you can see that we have a few kind\nof just like notes or whatever then we\nhave some paragraphs but then we have\nthis table and this is where I'm going\nto start to um Place more emphasis on\nand then we're going to do uh partition\nPDF we're going to give it the file name\nand then we're going to give it some\nunstructured helpers this is just a\nlittle bit of config for them so if we\ntake a look here let's load it up let's\nsee what elements I actually found and\nso I found a whole bunch of them here it\nlooks like we just have some narrative\ntext this is going to be your regular\ntext but then we have this table and\nthat's actually what I want to double\nclick on and so what I'm going to do is\nI'm just going to go grab the fourth\nfrom the last element which is going to\nbe this table right here 1 2 3 4 and\nlet's look at what the HTML looks like\nso here we have the HTML and now you\nmight be saying well Greg how come the\nHTML is important why would you just\nwant to read it like a table well tables\nare easy for us to read They're not so\neasy for the language model to read\nhowever the language model has been\ntrained on HTML tables not only HTML but\nalso markdown but in this case I want to\npull out the HTML table because the\nlanguage model is going to be able to\nmake more sense of this than I can so\nwhen I pass my data to the language\nmodel I'm going to pass the HTML or you\ncan pass markdown whatever works for you\nand if you want to see what this\nactually looks like you can head over to\nan HTML viewer and you can see what the\num unstructured actually pulled out\nwhich is pretty cool nice so that's how\nyou do tables within PDFs but now let's\nsay you have images within PDFs or maybe\nyou have images elsewhere how you going\nto take advantage of those\nhow are you going to extract those well\nlet's take a look at how you do that and\nI'm going to use unstructured once more\nthis time I'm going to do uh their\npartition PDF which is the same as last\ntime and here I have a fine-tuning uh\nvisual fine tuning paper you can go and\ncheck this out let's go look at the\narchive one I'm going to download the\nPDF just so you can see it because\nthere's this wacky photo up at the front\nall right so here's the same one I'm\ngoing to load up that page that I had\nand then I'm going to get the uh\npartition PDF ready for me and this time\nI'm going to do extract images in PDF\nequals true so now it's going to extract\nall the parts for me which is the\nchunking process but it's going to treat\nthe images separately which is nice also\niner table structure blah blah blah\nlet's go from there image output\ndirectory path this is in this repo uh\nso you don't have to reload the images\nif you don't want to but either way I'm\ngoing to load those up and I don't want\nto make you wait this does take just a\nlittle bit of time so let me come back\nto you 1 minute later awesome so that\njust finished up loading for us let's\ntake a look here so I know it's kind of\nsmall on the screen but I they found\nlooks like 15 or so different images or\n16 different images on here and that was\nall extracted from the PDF that I\nsupplied now the interesting thing about\nthis is how are we going to make those\nimages useful right we would need to\ntake an embedding of them because we're\nprobably going to do semantic search\nlater however embedding models usually\ndon't cross paths between text and\nimages meaning there's embedding models\nfor images and there's embedding models\nfor text but generally their Vector\nlength isn't going to line up and if you\ndon't have the same model for each one\ndoing similarity search between the two\nmay give you a hard time yes I know for\nall the perfectionists out there there\nis something called the clip model which\nis going to do embeddings for both\nimages and text so you can take\nadvantage of them however the tech is\nstill not quite there and I haven't\nfound the same performance with clip\nthat I have with other ones so I'm going\nto show you a different method here I I\nwill note this to say that in the future\nand when you're watching this there may\nbe really good models that do both of\nthese if it's true then I would take\nadvantage of those uh instead of the\nmethod I show you but either way let's\ntake a look here so what I want to do\nactually is I want to generate a text\nsummary of each image and then I'm going\nto do an embedding of that text summary\nso now what I can do is I'm going to go\nto semantic search maybe the text\nsummary will get returned for me if so\nthen I can pass that image to a\nmultimodal llm or I can just use the\ntext summary on its own to answer my\nquestion or do my task cool the way\nwe're going to do this is we're actually\ngoing to use langen and here we are\nwe're going to load up uh chat open Ai\nand we're going to use the gp4 vision\npreview model all right so I'm going to\nload that up and I made a quick function\nhere this is just going to convert the\nphysical file on my local machine to Bas\n64 which we uh then can go and pass to\nthe language model so string to base 64\nnow we have this image string let's go\ntake a look at what this looks like just\nlooks like a bunch of gobbly go which\nthat doesn't mean much to me but it will\nmean something to open Ai and I'm glad\nthat it does all right let's close this\nlet's get that out of there so what I'm\ngoing to do is I'm going to use the GPT\nfor vision one again and we're GNA\nconstruct a human message this just\nmeans it acts as if it's coming from the\nhuman content type please give me a\nsummary of the image provided be\ndescriptive and then we're going to pass\nit an image URL and here we are the URL\nis we're going to pass in our image U\nbase 64 we had there let's go ahead and\npass that over and let's see what openi\nthinks the image actually is I haven't\nshown you what the image is but let's\nlook at the summary the image shows a\nbaking tray with pieces of food like a\ncookies or some baked goods arranged\nLoosely to resemble the continents on\nearth as seen from space hm what but do\nyou know there you go yeah that makes\nsense so now when I do my retrieval\nprocess I can either just use this text\nin lie of the picture if I don't want to\nwork with a multimodel llm or I can do\nsemantic search have this summary get\nreturned and then pass this image over\nto the language model the llm all right\nso that seems about right so what I've\ndone in level three here is emphasizing\nthat your chunking strategy really\ndepends on your data types so in this\ncase I was pretty explicit about that\nand I showed you what it would mean for\nPython and JavaScript and if you have\nimages but in your industry in your\nvertical you may have different data\nformats and you'll want to pick a\nchunking strategy that is going to adapt\nto those data formats because remember\nthe ultimate goal is that you want to\ngroup similar items together so that you\ncan get them ready and prepared for your\nlanguage model task in the end now\nyou'll see there that I even made an\nassumption that you want to group\nsimilar items together I'm just saying\nthat because generally you're doing\nquestion and answer and generally you\nwant to combine similar items together\nfor context to answer a question however\nif you're not doing that maybe for some\nreason you want to combine opposite\nitems together in in which case your\ntrunking strategy be a lot different I\ndon't know of anybody who actually do\nthat but let's get back to the tutorial\nhere all right so now we're moving on to\nlevel four semantic chunking now the\ninteresting part about levels 1 through\nthree here is we all took physical\npositioning into account doesn't it seem\nkind of weird that we would split up a\ndocument with the intention of grouping\nsimilar items together we just assume\nthat paragraphs have similar information\nin there what if they don't what if we\nhave really messy information and doing\nrecursive character text splitting\ndoesn't really do anything for us you\nknow I saw this tweet from lonus we can\ngo and take a look at this one he\nsays weird idea chunk size when doing\nthis when doing retrieval augments\ngeneration is an knowing hyper pram and\nfeels naive to turn it into a global\nconstant value I totally agree now he\nrecommends could we train an end to-end\nchunking model I didn't want to go quite\nthat far because I think there's a\nlittle easier step that we could try\nbeforehand and I wanted to do an\nexploration but now what I'm going to do\nis I'm going to do an embedding based\nchunking method it's a little bit more\nexpensive and it's definitely more work\nand it's definitely slower than what we\ntalked about for the first three but it\nstarts to take the meaning and the\ncontent of the text into account to make\nour chunks the analogy I looked up\nbeforehand is imagine the first three\nlevels that's like having a bunch of\nbooks and putting them on a bookshelf\ndepending on their size right and the\nbookshelf size but what if you want to\ngroup the books together by genre or by\ntheme or by author well then you\nactually need to know what the books are\nabout and that's what we're going to try\ndoing level four here all right so when\nI thought about level four what I wanted\nto do was obviously semantic chunking\nand I chose an embedding based way to do\nthis so what I wanted to do was I wanted\nto take embeddings at certain positions\nof our document and then I wanted to\ncompare those embeddings together right\nso if two embeddings are close to each\nother distance-wise\nwell maybe they're talking about the\nsame thing that's the assumption that\nwe're going to make if they're further\nfrom each other that means that they're\nmaybe not talking about the same thing\nright so what I imagine is I we'd have a\nbig long essay and then with those I'd\ntake an embedding of every single\nsentence that we have right and then I\nwant to compare those embeddings\ntogether now the comparing the\nembeddings that's going to be the\nimportant part and where all the magic\nis going to be for this and I did two\ndifferent methods that I wanted to share\nwith you the first one is I did\nhierarchical\nthat's a mouthful hierarchical\nclustering with positional reward so my\nfirst thought is well you know let's\njust do a clustering algorithm and let's\nsee which embeddings are clustered\ntogether and then let's assume that\nthose are the chunks that we're going to\nhave but one thing I wanted to do was\ntake into account short sentences that\nappear after a long sentence you know\njust like that I wanted the you know to\nbe included with that long sentence\nbecause it's likely needs to be relevant\nwith it and so I added a little bit of a\npositional reward so hierarchal FAL\nclustering generally is just going to be\nbased off of distance but I added a a\nlittle uh extra sauce to it and did some\npositional reward all right this one was\nokay but it was kind of messy to uh to\nwork with and it wasn't as logical as I\nwanted it to be and I couldn't really\ntune this um intuitively like I wanted\nso I wanted to find something just a\nlittle bit easier as an exploration for\nme so the what I did was is the next\nmethod was to find break points between\nsequential sentences so I got embedding\nnumber one of sentence number one and I\ncompared that to sentence number two's\nembedding and I measured the distance\nbetween them and then I got two compared\nit to three and then three compared it\nto four and so on and so forth um I do a\nvisual with this and so I guarantee it's\ngoing to make more sense in a second\nwe're going to use Paul Graham's essay\nand what I'm going to do is I'm first\ngoing to split all of my different\nsentences and I'm going to do that just\nvia some rejects with a period a\nquestion mark or an explanation point\nthere's likely a lot of better ways to\ndo this don't come at me with that but\neither way we have\n3177 different sentences in this Paul\ngram essay all right so what I want to\ndo is I want to start adding more\ninformation to each one of these\nsentences so it's like I have a lang\nchain document but I'm just going to do\nmy own to show you how we're going to do\nthis so instead of having a list of\nsentences I want to have a list of\ndictionaries of which the sentence is\nplaced in it I'm going to add in the\nindex just CU it's fun why not and let's\ntake a look at these first three after I\ndo that now I have a list of\ndictionaries and one of the keys is\nsentence and now we have these different\nsentences up here CU if I were to go\nhere let me just show you what this\nlooks like the single sentence list I\nwant to do this with this the first\nthree again it's just a list of strings\nnow these list of strings are list of\ndictionaries all right cool well now\nwhat I want to do is I actually want to\ndo some combining of the sentences like\nI said if I just did sentence one\ncompared to sentence two compared to\nsentence three it was a little noisy it\nwas kind of all over the place and it\ndidn't tell me much I thought you know\nwhat if I combined the sentences so\nthere's a little less movement from each\none cuz now what I want to do instead of\ncomparing one to two comparing to three\ncomparing to four Etc I'm going to\ncompare the embedding of sentence 1 2\nand 3 combined with sentence 2 3 and\nfour combined then compare that with\nsentence 3 four and five combined so\nit's a little bit more of a group I did\na just a small little function you could\ntake advantage of here I have a buffer\nsize of one means one sentence before\nand one sentence afterwards you can do\nwhatever you want go and switch around\nwith this and go play with it I won't go\nthrough this code but I've commented it\nso you can follow along if you want now\nlet's take a look at what that does so\nhere we have our original sentence but\nnow we have our combined sentence all\nright and this combined sentence is\ngoing to be what comes uh before and\nafter it because this is the first one\nthere's nothing before it's only after\nso um get funded by Y combinator is the\nsentence of number two nice so we have a\ncombined sentence here which is want to\nstart or startup that's what sentence\nnumber one is and then uh something in\nthe grad school and that's what sentence\nnumber uh three is cool now that we have\nthose what I want to do is I want to get\nan embedding of the grouped sentences of\nthis combined sentence key so I'm going\nto use open Ai embeddings and let's go\nthrough this and I'm going to get all\nthe embeddings um which is basically get\nthe combined sentence for X in each one\nof those sentences and this is going to\nbe we're going to go grab all those\nwhich is really nice we have our\nembeddings now I need to put those\nembeddings with its proper list all\nright so sentence uh with the now I'm\ngoing to make a new key the combined\nsentence embeddings and I'm just going\nto go through and add those and let's go\ntake a look at what that looks like now\nCU of course this is fun and I like\ndoing this in an iterative nature so we\ncan take one step together at a time all\nright so sentence want to start a\nstartup here's our combined sentence\nembedding so now we have this embedding\nfor what's up here all right cool um\nwell now what I want to do is I want to\nadd one more metric to it and I know we\nkeep on going here but hopefully you're\nstill following along I want to add the\ndistance between the first uh sentence\nand the second group of sentences I want\nto add that to the first sentence so I\ncan see how big is the jump with the\nnext one all right so what we're going\nto do is we're going to get the\nembedding of the current thing we're\ngoing to get the embedding of the second\nthing the second group that it comes\nwith we're going to get the distance\nwe're going to append the distances\nbecause we're going to do something with\nthis later but then we're going to start\nwe're going to get a distance to next so\nhow far is the distance between the\ncurrent embedding with the next one\nlet's go ahead and let's run this and so\nnow we just got that and this is added\nto our sentences but we have our\ndistances here too so let's just take a\nlook at the first three distances\nawesome so this is 08 so this means that\nsentence number group number one is 08\ndistance away from group number two and\ngroup number two is0 2 distance away\nfrom group number three hm that's kind\nof interesting why is group one further\naway from group two than group two is\nfurther away from group three I don't\nknow but we're going to do something\nwith this in a second let me show you\nthese sentences look like one more time\njust because we're doing this\niteratively oh boy I added a whole bunch\nhere there's too many I should have did\njust did the first three okay um we'll\ngo through this let's scroll all the way\ndown to the bottom we got a long ways to\ngo okay now we finally have it distan to\nnext because this is the first one you\ncan see that it's 08 that's what we just\nsaw above cool let's close that up uh we\nhave our distances here but now we're\nall data people we're all having fun we\nall want to see some visuals I want to\nsee some visuals let's do that any data\nscientists out there will laugh at this\nbecause I've typed this more in my life\nthan I think I've ever should import\nmatplot live. pyplot as PLT that is just\nabsolutely muscle me memory for me right\nnow all right so now we plot our\ndistances hm cool so this is our\ndistance now it looks kind of random\ndoesn't it well I mean a little bit you\ncan see here that it looks like there's\na little bit of some e and flows this\none there's a little bit more distance\nin between so what this would mean in\nEnglish is that for some reason the\nchunks here are more dissimilar from\neach other than uh further down than\nchunks that are grouped together but\neither way what's interesting to me is\nthat we have some outliers up at the top\nhere you can see here there's these\npoints up the very top and that tells me\nhm maybe there's good break points there\nbecause two groups are so dissimilar\nthat they should actually be chunked up\nand they shouldn't actually be together\nbecause if there's a long distance in\ntheir embedding space maybe they're not\ntalking about the same thing all right\nso I want to show you this one more time\nbut let's iteratively build another\nvisualization to further emphasize the\npoint that I'm trying to make here all\nright first thing I'm going to do is I'm\njust going to plot the distances let's\ngo down that's the exact same that we\nthink that we had beforehand all right\nthe next thing I want to do I just want\nto do a little bit of formatting don't\nhate me for this first thing I'm going\nto do is I'm going to do a y upper bound\nthis means the bound of the upper y\nlimit because you as you can see right\nhere there's not enough cushion up here\nfor me it's visually too off I want to\nfix this all right and then we're going\nto do a y limit of from zero to the Y\nupper bound which means how long is your\ny AIS and then we're going to have an X\nlimb of how long you want this to be on\nthe X limit because you see here there's\nthese buffers in between I don't want\nthat all right let's get rid of that and\nanyway we go through this we have more\nspace up the top we got rid of the sides\nall right what's next let's see what we\nhave here I don't even know what's going\non here I wrote this code I I still\ndon't I'm just kidding I do um\nbreakpoint percentile threshold so what\nwe need to do is we need to somehow\nidentify which outliers do we want to\nsplit on now there's a million ways you\ncould go about doing this and I'm really\nexcited to hear alternative methods for\nyou um that you may have in the back of\nyour mind for me what I ended up doing\nwas I just did a percentile base I\nwanted to identify the outliers so using\nthe different points as a distribution I\nwanted to find out hey which points are\nin the top 5% because if it's in the top\n5% well those are probably going to be\noutliers for us so I only want to take\nthese um upper these upper distances\nright here and the way I'm going to do\nthat is I'm going to specify the\npercentile I want to take and then I'm\ngoing to use numpy and I'm going to go\nmp. percentile I'm going to pass it my\ndistribution of distances and I'm going\nto get the break point percentile\nthreshold it's going to be 95 and then\nwhat I want to do is I want to draw a\nline on the graph showing where that\nthreshold is so I'm going to draw a a\nhorizontal line right across and the Y\nis going to be the breakpoint distance\nthreshold this will be a number that\nsays hey what is the 95th percentile of\nall these all right let's take a look at\nwhat that looks like and here we go so\nnow anything above this line is going to\nbe in the 95th percentile these will be\nmy outliers where I'm going to\neventually make my chunks so everything\nthat's in right here up until this one\npoint that'll be chunk one then we have\nchunk two then we have chunk three chunk\nfour chunk five blah blah blah and going\nfrom there because again one one last\ntime I know I keep on repeating this but\nI really want to hammer this home the\nhypothesis is that if there's a big\nbreak point then a chunk should be uh\nsplit up at that point and so this is\nwhere we're end going to end up doing\nthat all right then what we're going to\ndo is we're going to see how many\ndistances are actually above this one\nand so I want to get the number of\ndistances that we have meaning the\nnumber of break points the number of\nthings that are going to be above that\nthreshold and then I'm going to do PLT .\ntext this is just a fancy way to put\nsome text on your visualization let me\ngo ahead and do that and so we can see\nthat we have 17 chunks I put that down\nin the corner right there H that's kind\nof interesting all right um then what\nwe're going to do is we're going to get\nthe indices of which points actually are\nabove the breakpoint meaning which are\nthe actual outliers because this break\nuh this break breakpoint distance\nthreshold this is just a single static\nnumber but I need to get a list of\nnumbers to find out where these break\npoints and these chunks actually need to\nbe met so for I IND distances um if x is\nabove the breakpoint distance threshold\nso I'll get a BN a bunch of indices\nthere that doesn't do anything different\nfor us but then what we're going to do\nactually let me just do this because I\nthink this actually would be helpful I'm\ngoing to look at these indices now what\nwe have here these are 17 different\nchunks we can look at this it says 16\nbut that's cuz there's an extra one\nadded to the front there should be a\nzero right here um or this could be the\n317 at the end so what this means is\nbetween uh between sentence zero and\nsentence 23 we want to make our split\nand make our break all right because at\nthis at number 23 it says Hey here the\ndistance to the next one was quite big\nso we want to include number 23 on this\none okay either way let's go from there\nlet's go do some more uh let's do some\nfancy colors on here so what I want to\ndo is I want to add some colors I just\nset my own custom ones right here then\nwhat I'm going to do is I'm going to go\ndo uh a vertical span meaning you're\ngonna have a vertical shading in the\nbackground of your um the background of\nyour graph here and you're going to do a\nstart index and an end index the reason\nwhy I do a for Loop here is because you\nadd them one at a time but the start\nindex and the end n index will be what\nis in our indices above thresholds\nanyway's go through there and you can\nsee here I cheated a little bit let's\ntake away this text we can see that we\nhave our different chunks right here so\nnow we have chunk zero chunk one chunk\ntwo blah blah blah and go go all the way\nthrough there but of course we want some\ntext on here to really make it more\nexplicit and you can see here chunk blah\nblah blah blah blah uh this last one was\ngiving me a hard time so I actually had\nto do just just a little bit of custom\ncode for that one uh that splits it up\nuh that's just a little bit of a\nBand-Aid don't don't add me for that one\neither I was too lazy to figure that one\nout Okay cool so we go through there so\nhere's all of our different chunks now\nthis wouldn't be a good chart unless we\nput on some graphics or some titles as\nwell so now we'll do a title A Y Lael\nand an X\nlabel and then there we go hm that's\nkind of cool uh Paul Graham essay chunks\nbased off embedding\nbreakpoints H that's pretty interesting\nbut uh a good visualization doesn't do\nanything for us we can't really pass\nthis to the language model and it's not\ngoing to know what to do with it so what\nwe're going to do is we're going to\nactually get the sentences and actually\ncombine them so here's a bunch of code\nhere um I won't go through it too much\nbut the the tldr is that you're going to\nappend all these different pieces in\nyour chunks so like I said beforehand\nyou're going to go from chunk zero to\nchunk 23 and you're going to combine\nthose and that'll be your first chunk\nlet's go through there then let's take a\nlook at what this actually looks like\nlet's go through this and so we have the\nuh chunk number zero about a month of\nneed phing cycle we had something called\na prototype day you might think they\nwouldn't need any more motivation cool\nthey're working on their cool new idea\nthey have funding for an immediate\nfuture and they're playing the long game\nwith the only two outcomes wealth or\nfailure hm you think motivation might be\nenough so the hypothesis here is that\nthese two are actually getting split up\nat a spot where there's a big breakpoint\nwith it so it's kind of interesting to\nsee where the uh semantic splitting\nactually happened there you go and now I\nwant to reemphasize that this isn't\nperfect of course but I think this is an\ninteresting step towards doing chunking\nbecause if I were to think if I'm going\nto hypothesize out in the future what is\nchunking going to be like uh well as\ncompute gets better as language models\nget better there's no way we're going to\ndo physical based chunking anymore\nunless the um structure of our documents\nis uh we can make those big assumptions\non it we're probably going to do a smart\nchunking and I think this is a really\ncool um way to go towards that all right\nso that's level four again experimental\nplease let me know what you think please\ngive me other ideas for how we could\nmake this a little bit better and this\nis going to be a little plug if you want\nto see more of these experimental\nmethods that I do I shared this out on\nTwitter so um other people got to see\nthis early now let's move on to level\nfive a gentic chunking so if we went off\nthe deep end with level four um we're\ngoing into the ocean here we're going\ninto the Mariana Trench and we're going\nto the very bottom and we're going to do\nsome cool things so my motivation for\nthis side is I asked myself hey Greg\nwhat if a human were to do chunking how\nwould I do chunking in the first place\nand I thought well I would go get myself\na piece of scratch paper cuz I can't do\nall that in my head I'd start at the top\nof the essay and assume the first part\nwill be in a chunk well because the\nfirst little bit it needs to go\nsomewhere we don't have any chunks yet\nso of course it's going to go in a first\nchunk then I would keep going down the\nessay and evaluate if a new sentence or\npiece of the essay should be a part of\nthe first chunk if not then create a new\none then keep doing that all the way\ndown until the S on the essay until we\ngot to the end and I thought to myself\nwait wait a minute this is pseudo code\nwe can make something like an agent to\ndo this for us now I don't like the word\nagent quite yet because we're not quite\nthere yet a lot of people like using I\nthink there's a lot of marketing around\nit so yes I call this agentic chunking\nbut I'm going to call this an agent like\nsystem the way that I like to Define\nwhat an agent is is there's some\ndecision making that needs to go on in\nthere and you use the language model to\nmake decisions which you're going to\nhave an unbounded path but the language\nmodel will help you guide that via your\nuh decision-making that you do and so so\nI thought man this is pretty cool I'm\ngoing to try this out all right so now\nlet's go into level five and talk about\nwhat I found here so one first design\ndecision that I need to make is how do I\nuh want to give different pieces of my\nessay to the language model and I\nthought man well there's still that\nproblem with the short sentences you\nknow so around this time there was a\ncool paper that came out all about\npropositions what is a proposition well\na proposition is a sentence that can\nstand on its own so there's another\nagent-like system it's kind of just a\nprompt we'll go over that in a second\nhere but it's going to take a sentence\nand it's going to pull out propositions\nwhich are little itty bitty Legos that\ncan stand on their own let's talk about\nwhat that means Greg went to the park he\nlikes walking if you were to pass he\nlikes walking as a chunk to the language\nmodel language model is going to be like\nWTF who is he however if we change this\ninto propositions it will make less\nsense from a reader's perspective\nmeaning it doesn't look great for us to\nread but it makes a lot more sense for a\nlanguage model Greg went to the park\nGreg likes walking that's pretty\ninteresting if you want to take a look\nat more proposition work and I might do\na whole another video on this let me\nknow if you want me to because I think\nthis is really cool uh Lang chain came\nout with proposition based retrieval so\na new paper by Tom Chen um I haven't met\nTom Chen yet but Tom I'm a big fan of\nyour work if you want to chat I'm very\ndown at talk in the image that they\nshowed prior to the restoration work\nperformed between blah blah blah blah\nblah blah you have this big long\nparagraph and then it's going to split\nup into different propositions and then\nuse that for your retrieval because\nthese sentences can stand a bit more on\ntheir own let's go ahead ahead and do it\nI'm importing a whole bunch of Lang\nchain stuff here I'm not going to go\nover each one except for the for the\ncool parts now one of the cool Parts\nhere is going to be this Lang chain\nimport Hub hey Greg what's the Lang\nchain Hub I should probably do a whole\nanother video on this either but they\nhave this thing or Lang chain came out\nwith this thing called The Lang Hub this\nis within their Lang Smith Suite but a\nlang Hub is just going to be where they\nshare prompts around so this person I\ndon't know who this is um looks like\nit's been viewed a lot YouTube\ntranscript to article\nact as a uh expert copywriter\nspecializing blah blah blah so Lang\nchain will help host prompt templates\nthat you can use for your own so it's an\neasy way to just go share prompt\ntemplates so here we have this whole\nentire prompt and here we have a token\nwhere you can go and grab it and then if\nyou wanted to grab this prompt yourself\nthen you can go and just grab it like\nthis the uh advantage of this is you can\nshare prompts a whole lot easier but\nthen two if you want prompts to be\nupdated with the latest and greatest\nthinking well you can just keep on\npulling from there which is nice\nso I'm going to pull uh hub.\nwhf proposal indexing all right so let's\ngo you view the proposition prompt here\nuh wfh proposal indexing here's a\ncitation that comes with it and here's\nthe chat prompt template I won't go\nthrough all this but the interesting\npart is split the compound sentence into\nsimple sentences maintain the original\nphrasing from the input whenever\npossible and then they actually give one\nexample here so they have an input one\nhere and then they have an output about\nwhat they want the language model to\nOutput hm interesting I just went and\ncopied this code I put that in right\nhere I'm also going to uh get my\nlanguage model going we're going to use\nGPT for a preview because I want the\nlong context and good processing power\nall right so within this object is going\nto be the prompt then I'm going to\ncreate a runnable which combines The\nPrompt and the language model and this\nis via the Lang chain expression\nlanguage and with that language you can\ndo just this pipe operator and you can\ncombine those right there and then the\nnext thing I'm going to do I tried doing\nsome uh extraction of these propositions\nvia the uh recommended way but I found\nit was giving me a hard time so I just\nmade my own extractor from it and the\nway I'm going to do that is I'm going to\njust do the pantic extraction method so\nI'm going to create a pantic class here\nand then I'm going to do extraction\nchain create extraction chain with p uh\npantic and I'm going to pass it in the\nsentences pass in the language model\ncool we have that and then I'm going to\ncreate a small little function which is\nthis the get propositions which is going\nto be hey go and get this Pro go and get\nthe list of propositions from the thing\nthat I give you because it takes a\nrunnable and then it's going to do the\nextraction\nand it's going to return the\npropositions for us all right so now I'm\ngoing to do Paul Grahams superlinear\nessay cool we have that I'm going to\nsplit the essay into paragraphs now this\nis a a design Choice hey Greg aren't we\ndoing just chunking again not really\nbecause this is like a very loose chunk\nthat doesn't really matter I could pass\nit a sentence at a time I could pass it\nto I could pass it paragraphs see how\nmany paragraphs we have we have 53\ndifferent paragraphs nice let's take a\nlook at one of the paragraphs just\nbecause I like um being super explicit\nhere\nuh let's do number two nice one of the\nmost important things I didn't know etc\netc we can look at number five cool\nthose are just paragraphs for us all\nright then what we're going to do is\nwe're going to go get our propositions\nand so we're going to do essay\npropositions and so I'm just going to go\nthrough this go through each paragraph\nI'm just going to do the first five\nbecause there's kind of kind of a lot of\ndata here I'm going to get the\npropositions let me come back to you\nwhen this is done one minute later\nawesome so we have our propositions\nlet's take a look at a few here again\nbecause we're doing this iteratively\nwith each other all right I'm going to\ntake a look at a\nfew you have 26 propositions cool so our\nfive paragraphs resulted in 26\npropositions the month is October the\nyear is 2023 at the time past I did not\nunderstand something about the world\ncool so now we're starting to pull out\nindividual facts about what the um what\nthe paragraph is about lovely so now\nwhat I want to do is I want to use an\nagent-like system that is going to go\nthrough each one of these iteratively\nand decide hey should this be a part of\na chunk that we already have or should\nit not there there's no package that I\nsaw that did this for us yet because\nthis is a quite experimental method but\nwhat I did is I ended up making a a\ngentic chunker this isn't a package yet\nso you can't go uh import this anywhere\nbut I'll show you the code that's\npowering this awesome so here's the code\non how this works now I'm not going to\ngo through this in detail because that's\nnot the uh point of this tutorial but\nI'm going to go through the highle\npieces here the way that it works is\nyou're going to have uh AC equals a\ngench chunker okay cool and then you're\ngoing to have your list of propositions\nthis could be sentences but propositions\nwill be best cuz that's how I designed\nit to work and then what you're going to\ndo is you're going to add your\npropositions to the class and then what\nit's going to do is it's going to start\nto form chunks for you then we're going\nto pretty print the chunks all right so\nlet's go through and see how this works\nthe real magic happens in the add\npropositions so if we go up to the top\nhere so we're going to add proposition\nnow if it's your first Chunk meaning\nyour first proposition your first chunk\nthen you're not going to have any chunks\nand chunks is going to be a property of\nthis class if chunk size equals zero\nmeaning you don't have any yet then\ncreate a new chunk hm totally makes\nsense well what do you want the chunk to\nbe about well let's go find out where\ncreate new chunk is and on create new\nchunk what we're going to do is we're\ngoing to create a chunk ID which is just\ngoing to be a random uh uu ID or a\nsubset of one then we're going to get a\nchunk summary and a chunk title the\nreason why we do this is because when we\nadd a future thing we need to know well\nwhat are our chunks already about right\nbecause that'll tell us whether or not\nwe need to add it so we're going to have\na summary which is going to have a lot\nof good detailed information and we're\ngoing to have a title right so get new\nchunk summary all that this is doing is\nlooking at the propositions that are\ncurrently in the chunk and then it's\ngenerating a summary about what that\nchunk is about and then the chunk title\nthis is just a few words that kind of\ngive you a quick glance on what it\nactually is now there is a parameter you\ncan set when you do this do you want to\nupdate your summary and title because as\nI was going through this um as I added\nsay proposition number one it was about\none thing but then once you added\nproposition number two and three and\nfour to the chunk you may need to update\nthe summary or update the chunk because\nnow the chunk is kind of just slightly\ndifferent right it's kind of as if you\nhad a uh centroid and it's moving just\nslightly but you want to capture that\nEssence all right and then what we do is\nwith each one of the chunks we have a\nchunk ID we have a proposition or we\nhave a list of propositions a title a\nsummary and then the chunk index is just\num when was this chunk made its number\ncool uh let's go back to where we were\num okay so that's if if you add uh your\nfirst proposition to your empty list of\nchunks and then you're going to return\nhere meaning you're going to stop you're\nnot going to go any further because well\nyou've already added it to a chunk it's\nall good cool but let's say that wasn't\nthe case well let's go find a relevant\nchunk this part I thought was actually\nkind of cool too in this relevant chunk\nyou're going to have a proposition now\nyou want a proposition to go in and you\nwant a chunk to come out right and so we\nhave a a simple little prompt here\ndetermine whether or not the proposition\nshould belong to any of the existing\nchunks and then I do an example here I\nhave some other good stuff but what I'm\ngoing to do here is I'm G to pass it a\nuh string of what our current chunks\nactually look like and those current\nchunks are going to be groups of three\nthings the ID of the chunk the name of\nthe chunk and the summary of the chunk\nbecause what I want it to do is if it\ndeems that yes this proposition should\nbe part of a chunk well then what I want\nyou to do is I want you to Output the\nchunk uh ID for me and this is how I can\nthen extract which chunk it should\nactually be a part of all right so we're\ngoing to go through there we got all of\nthat um and let's just say that a chunk\nID uh does come out meaning it should be\nadded to another chunk well what I'm\ngoing to do is I'm just going to add\nthat proposition to the chunk nice so\nyes I found a chunk it should be a part\nof cool I'm going to add this\nproposition to it then when you add this\nproposition to it this is where I was\ntalking about beforehand um if you want\nto generate new metadata then it will go\nand generate a new summary and title if\nyou don't want to do it then it doesn't\nhave to so let's say you didn't find a\nnew chunk ID meaning you uh wanted to\nadd a proposition it didn't find a new\nchunk so you need to actually make a new\none uh no chunks are found create a new\nchunk and that's the same thing that we\nhad up above and you go from there so\nthat's really the meat of the entire\nthing and if we're going to go back to\nour repo here let's go do this for our\nessay that we had all right so I'm going\nto uh do the agentic chunker I'm going\nto say from AC or uh make AC and then\nwhat I want to do is I want to add each\none of the first 20 propositions but you\nknow we only had 23 so let me just this\nwhole thing here and then what I did was\nis I have a lot of print logging if you\ndon't want that cuz you think it's\nannoying just go print logging equals\nfalse but let's step through this so now\nit's adding our first chunk which is the\nmonth is October no chunks duh because\nyou don't have any so it's created a new\nchunk which is\n51322 it's called date and times yeah\ncool makes sense that's probably where\nI'd want this to go the year is 2023\nchunk found date and times oh yeah duh\ncuz what it did is it's saw there's a\nchunk there's a date and times chunk I'm\ngoing to add this one to it too too okay\ncool I was a child at some past time no\nchunks are found because there's only\none at this time and it doesn't think\nit's part of that one and so created a\nnew chunk personal history H nice okay\num at the past time I did not understand\nsomething important about the world it's\nadding it to personal history cool makes\nsense the important thing I did not\nunderstand is that the degree in which\nreturns for performance are super linear\nit didn't find any chunks and so it\ndoesn't think it's part of date and\ntimes or personal history it made a new\none called\nreturn performance and returns\nrelationship cool teachers and coaches\nimplicitly told us returns were linear\nuh chunk found it's adding it to the\nreturns chunk nice teachers and coaches\nmeant well no chunks blah blah blah and\nwe go all the way through here and so\nit's kind of interesting but what I want\nto show you is an an instance where a\nchunk name was updated so adding um you\nget what you you get out what you put in\nWas Heard A Thousand Times by the\nspeaker it's adding it to the Mis\nconceptions in per performance of\nreturns and relationships wait where did\nthat title come from oh wait it's the\nsame chunk ID but the name has been\nupdated because you're actually updating\nthe chunk as uh the chunk TM and the\nchunk title as you go along here so as\nwe go through here we're going all the\nway through it did this a whole bunch of\ntimes I want to see what comes out the\nother end so you can pretty print the\nchunks cool so it looks like we have\nfive chunks chunk number zero has this\nchunk ID um this chunk contains\ninformation about the specific dates and\ncool and here are the propositions that\nwere added to that chunk nice so this is\nthe content of our chunk that we want to\nactually pull in this chunk contains\nReflections on someone's childhood blah\nblah blah okay cool oh dang oh this is a\nbig one so now we have a bunch of\nstatements that were pulled from the\nessay that are all about superlinear\nreturns across different fields H all\nright cool and let's look at this last\none or second to last one teachers and\ncoaches K cool consequences of inferior\nproduct quality on business viability\nand customer base nice all right so\nthat's kind of interesting now we're\nstarting to group similar items together\nso if we have a question that pops up\nabout product quality and business\nviability well here's the chunk that you\ngot to look at now is this perfect well\nnot quite yet because you could get some\ncomplicated questions where you may want\nmultiple things for different chunks but\nI think this is a really interesting\nDirection on how we'd start to move\ntowards there and if we actually want to\nget these chunks because we want to go\ndo something interesting with them maybe\nproper index them then you can go and\nget those and get the list of strings so\nthere you go that's a gentic chunking\nnow again it's slow and it's expensive\nbut if you bet that language models will\nspeed up and they'll get cheaper which\nI'm guessing they will then this start\nthis type of method starts to come into\nplay finally what I want to do is I want\nto congratulate you on finishing the\nfive levels of text chunking and text\nsplitting we are almost done but I\nwanted to throw in a bonus level in\nthere right so this bonus level is going\nto be through alternative\nrepresentations now much like before\nlike our chunking commandment we need to\nthink about how we're going to get our\ndata ready for our language model right\nchunking is only one part of that story\nit's how you're actually going to split\nyour texts but the next step that you're\ngoing to take in front of that is you're\ngoing to get embeddings of your text and\nyou're going to go throw those in your\nknowledge base right well there's\ndifferent ways you can get embeddings of\nyour text and there's different\nquestions that'll come up like should\nyou get embeddings of your raw text or\nshould you get embeddings of an\nalternative representation of your raw\ntext that's what we're going to talk\nabout this bonus level let's go through\nthis very briefly\nthe first thing that we're going to talk\nabout is multiv Vector indexing this is\nwhen you do a semantic search but\ninstead of doing semantic search over\nthe embeddings of your raw text you get\nit off of something else like a summary\nof your raw text or hypothetical\nquestions of your raw text all right\nlet's do this first one which is\nsummaries and how we'd actually do that\nso we're going to get our super linear\nessay one more time I'm going to blow\nthrough this part since we already\ntalked about it but what I'm going to do\nis I'm going to get my six chunks from\nour document and I'm going to say\nsummarize the following\ndocuments and and that's going to be a\nchain that we have there and then we're\ngoing to do this but we're going to do\nthis in batch which is one of the other\nnice things about Lang chain uh\nexpression language and now we have all\nof our summaries and here's the first\none so this is a summary of our first\nchunk that we have right now what I want\nto do is I want to get an embedding of\nthis summary instead of the embedding of\nthe chunk like I I would have with the\nold method and the way that we're going\nto do this is we're going to get our\nVector store ready we're going to use\nchroma today and we're going to get just\na doc store ready which is just going to\nbe the inmemory bite store that link has\nand then we're going to pass in a multiv\nvector retriever this is a cool Lang\nchain abstraction where you can um\nbasically go semantic search off one\nthing but then return another thing for\nyour final language model all right\nwe're going to go through this there's a\nwhole tutorial that they have on this in\nfact I have a whole tutorial on this as\nwell at fullstack retrieval.com so I'm\ngoing quick through it but that's CU I\nwant to show you a whole bunch here\nwe're going to get our summary docs\nwe're going to add those and we're going\nto uh change those into proper Lang\nchain documents instead of the actual\nstrings that they were and then here's\nwhere the cool part happens we're going\nto add them to the vector store which is\nalso going to get the embeddings for us\nwe're going to add the summary docs and\nthen we're going to add our normal docks\nto our normal Dock Store and our\nretriever nice um cool and then what we\ncould do here is we could go to our\nRetriever and we could go get the\nrelevant documents and so now what I'm\ngoing to do is with my query this is\ngoing to go match on the summary\nembedding rather than the raw document\nembedding but when these docs are passed\nback to us these are the raw documents\nso there's a little happening behind the\nhood here um but again I encourage you\nto go to fullstack retrieval.com and go\ncheck out theor tutorial on this all\nright there's one method for us the\nother method you could do is well\ninstead of summaries I want hypothetical\nquestions this one's really nice if you\nanticipate a Q A bot that you're making\nbecause then you can start to anticipate\nwell which questions are people going to\nask the language model will generate\nquestions for you and there's a your\nhypothesis is that there's a high higher\nlikelihood that um these questions will\nend up matching for you and you'll get\nbetter uh document matching cool another\none you can do is you can do a parent\ndocument retriever so now this case the\nhypothesis with this method is that if\nyou subset your document even more\nyou'll get a better semantic search\nhowever in order to answer the question\nof whatever hypothetical question you\ncould have you actually want what's\naround that small document so yes that\nsmall document will will do good\nsemantic search but you actually want\nthe buffer around it so what comes\nbefore and after it and another way to\nsay that is you want the parent document\nthat that small chunk actually comes\nfrom and or in order to show you this\none I'm going to do a um I'm going to do\na tutorial from llama index now I also\nhave another tutorial on this on full\nstacker tre.com if you want to go check\nthat out so with L uh with llama index\nwe're going to do their hierarchical\nnode parser and what you're going to do\nis you're going to give a list of Chunk\nsizes so they're going to get chunks\nthat are two uh 20 48 512 and 128 here\nand let's go do our Paul Graham essay\nlet's see how many nodes we actually\nhave this is 119 nodes that's cuz 128 is\npretty small for us here and if we take\na look at one of these relationships\nhere we and this is one of the smaller\nones because this is something at the\nend and that's the 128 you can see that\nthese chunks are quite small but and we\nare just looking at the relationships\nhere you can get a source a previous a\nnext and then the important part here is\nthe parent so if you were to do this via\nthe Llama index way yes you matched on\nthe\n128 chunk because that has good semantic\nsearch but you actually got the 512 or\nthe 2048 right I won't show you how to\ngo all the way through that because\nagain that's a whole separate tutorial\nthen the last thing that I'll show here\nis around the graph structure so\nsometimes you're going to go over your\nraw text and instead of chunking your\nraw text you actually want to extract a\ngraph structure from that text because\nthere's a lot of entities within your uh\nwithin your text I'm going to do that\nvia the diffbot and this is the diffbot\nTransformer we're going to go I'm going\nto say say Greg lives in New York Greg\nis friends with Bobby San Francisco is a\ngreat City but New York is Amazing Greg\nlives in New York and let's see what\nactually pops out of over\nhere so now we have this so now we have\nactually have a graph document and one\nof our nodes is node ID Greg type person\nproperties K Greg we have another Bobby\nnode which is cool we have another node\nhere which is an entity location it's\nentity New York we have another uh\nrelationship node so this kind of be\nlike an edge and this is a social\nrelationship between Greg and Bobby so I\nwon't go through all these but either\nway you can start to see how now you can\nuse a graph structure to answer\nquestions about uh a person or your\nspecific text but that's a little\noutside the chunking side either way\nthat is actually the rest of the\ntutorial and I want to congratulate you\non what is quite a long video I'm not\nsure how long this is probably one of my\nlonger ones that we have yet either way\nI'm excited that you're here my name is\nGreg camrad and I am on a mission to\nfigure out how Ai and business business\nare going to overlap the vend diagrams\nwith each other this has been a lot of\nfun thank you for joining me we will see\nyou later",
  "transcript_chars": 81783,
  "ingested_at": "2026-05-15T04:41:04.544322+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 141858,
    "like_count": 4861,
    "channel_id": "UCyR2Ct3pDOeZSRyZH5hPO-Q",
    "categories": [
      "Science & Technology"
    ],
    "tags": []
  }
}