{
  "video_id": "7AmhgMAJIT4",
  "channel_slug": "dataindependent",
  "channel_handle": "dataindependent",
  "title": "What is the perfect memory architecture? | Sam Whitmore",
  "duration_seconds": 694.0,
  "url": "https://www.youtube.com/watch?v=7AmhgMAJIT4",
  "upload_date": "",
  "transcript": "[Music]\nThank you, Nicole. And thank you, um,\nHarrison and Langchain and Greg for\norganizing and hosting. Actually, one of\nthe first things I did with memory was\nwith Harrison on the original memory\nimplementation in Langchain. So, very\nfull circle. Um, cool. So for those of\nyou who do not know new computer and\nwhat we do, we have dot which is a\nconversational journal. It's in the app\nstore. You can use it now. We launched\nthis last year. So we've been working on\nmemory in AI applications since 2023.\nUm cool. So take us back to 2023. The\ntime GPT4 state-of-the-art. We have\n8,000 wind length token uh prompt. Very\nslow and very expensive. So I want to\nwalk you through some of the things that\nwe tried initially, lessons we learned\nalong the way and how we kind of evolve\nas underlying technology evolves. So\nwhen we started, our general goal was to\nbuild a personal AI that got to know\nyou. It was pretty unstructured. Um, and\nso we knew that if it was going to learn\nabout you as you used it, it needed\nmemory. So we were like okay let's just\nbuild the first build the perfect memory\narchitecture and then the product after\nthat. Um so we started out being like\nokay maybe we can just extract facts as\na user talks to dot and search across\nthem you know use some different\ntechniques and we'll have great memory\nperformance. So we learned pretty\nquickly that this wasn't really going to\nwork for us. So imagine a user saying,\n\"I have a dog. His name is Poppy.\nWalking him is the best part of my day.\"\nSo early extraction, we'd get things\nlike, \"User has dog. User's dog is named\nPoppy. User likes taking Poppy for\nwalks.\" There's a lot of nuance missing.\nSo like you can tell a lot about a\nperson from reading that sentence that\nyou can't tell from those facts. That\nwas pretty quick realization for us. We\nthen moved on. So we were like maybe if\nwe try to summarize everything about\nPoppy in one place then it's going to\nperform better. We decided that we're\ngoing to make this universal memory\narchitecture with entities and schemas\nthat were linked to each other. This was\na UI representation of it. Um so users\ncould actually browse the things that\nwere created. Um and they had different\ntypes and on the back end there was\ndifferent form factors with JSON blobs.\nThis is real example from our product at\nthe time. So I sent it a bachelorette\nflyer and it made like a whole bunch of\ndifferent memory types with schemas\nassociated. Um\nso you can see here that like this is\nwhat the backend data look like. There's\ndifferent fields and we had a router\narchitecture that would kind of generate\nqueries that would search across all of\nthese um in parallel. And what we found\nwas that it worked okay, but there was\nkind of some base functionality that was\nstill missing. Um, oh, this was a funny\nexample. Um, Jason, my co-founder, was\nsending it pictures and it made him a\ndrunk text category as a schema, which\nwe were like, that feels like a heavy\nread. Um, but anyway, so the schemas\nwere kind of fun. Um but yes so\nbasically we also saw that when we\nexposed this to users there was like too\nmuch cognitive overhead for them to\ngarden their their database like there's\na lot of overlapping concepts and people\ngot stressed by actually just monitoring\ntheir memory base. So again we're like\nokay let's just go back to basics here\nand figure out like what do we want our\nproduct to be doing and let's reexamine\nhow we want to build memory from that.\nSo we looked again at like what a\nthought partner should have to do to\nactually be really good as a listener\nfor you. So we realized like should\nalways know who you are and your core\nvalues. It should know basically like\nyou know what you talked about\nyesterday, what you talked about last\nweek and again like who Poppy is, if\nPoppy is your dog, who your co-founder\nis, stuff like that. And it also needs\nto know about like your behavior\npreferences and how it should adapt to\nyou as you use it. So we ended up making\nfour kind of parallel memory systems. So\nthe schemas that you saw didn't really\ngo away. They just became one of the\nmemory systems, the entities. And it's\nfunny seeing Will kind of say some of\nthe same ones. So it's like an example\nof convergent evolution because we kind\nof made these up ourselves, but\nbasically like\ntheory of mind. Um here's mine. It's\nkind of just like who am I? What's\nimportant to me? What am I working on?\nWhat's top of mind for me right now?\nEpisodic\nmemory is kind of like what happened on\na specific day. Here's kind of like a\nactual real example soon after I had my\nbaby last year. Um here's like another\nentity example. We ended up stripping\naway a lot of the JSON because it turned\nout to actually not improve performance\nin retrieval across the entity schema.\nSo, we kept things like the categories\nif we wanted to do tag filtering, but um\na lot of the extra structure just ended\nup being like way too much overhead for\nthe model to output. And finally, we\nmade this thing called procedural memory\num which is basically like triggered by\nuh conversational and situational\nsimilarity. So, what you're looking at\nhere is this intent. And if you're a DOT\nuser, you'll probably recognize this\nbehavior. says, \"Choose this if you have\nsensed a hidden or implied emotion or\nmotivation that the user is not\nexpressing and see a chance to share an\ninsight or probe the user deeper on this\nmatter.\" And then what it detects that\nthis is happening, it says like share an\ninsight, you know, ask a question, issue\na statement that encourages the\nbehavior. And so basically like the\ntrigger here is not semantic similarity\nbut situational similarity. I see a lot\nof overlap here for people building\nagents where if you have a workflow that\nthe agent needs to perform, it can\nidentify that it encountered that\nsituation before and kind of pull up\nsome learning it had from the past\nrunning of the workflow. So this is kind\nof our way our retrieval pipeline worked\nin 2024 which is like parallelized\nretrieval across all of these systems.\nSo if here's a query which is very hard\nto read so maybe these slides will be\naccessible separately. um what\nrestaurant should I take my brother to\nfor his birthday and in this sense in\neach of our four systems we detect if a\nquery is necessary across this system\nfor holistic stuff we always load load\nthe whole theory of mind episodic is\nonly triggered if it's like what did we\ntalk about last week or what did we talk\nabout yesterday and then here there's\ntwo like different types of entity\nqueries detected like brother and\nrestaurants and then we would do kind of\na hybrid search thing where like We mix\ntogether BM25 semantic keyword basically\nlike no attachment to any particular\napproach just like whatever improved\nrecall for specific entities. Um and\nthen the procedural memory here if\nthere's a behavioral module loading like\nrestaurant selection or planning then\nthat would get loaded into the final\nprompt. So funny thing also is when we\nlaunched people tried to prompt inject\nus but because we have so many different\nbehavioral modules and different things\ngoing on we called it like Franken\nprompt and like if people did prompt\ninject us they'd be like wait I think\nthis prompt changes every time which it\ndid. Um okay so for the formation for\nthese again really distinct per system\nso holistic theory of mind you don't\nneed to update that frequently episodic\nis like periodic summarization so like\nif you want to have it be per week you\nmight update across daily summaries once\nper week per day once per day etc\nentities we did per line of conversation\nand then we would run kind of cron jobs\nthat we called dream sequences where\nthey'd identify possible duplic\nduplicates and potentially merge them\nand procedural memory also updated per\nline of conversation.\nSo um along the past year our product\ntrajectory has changed. We're now\nbuilding dots which is a hive mind. So\nit's like instead of remembering just\none person that it meets, it actually\nremembers um an entire group of people.\nAnd yeah, so it's like many dots stores\nthe relationships between everyone. Um\nyeah, so you basically some of the added\nchallenges we're dealing with now are\nrepresenting um different people's\nopinion of each other, how they're\nconnected, and how information should\nflow between them. in addition to\nunderstanding all of the systems I just\nmentioned above.\nSo one other thing I'll share that has\nevolved in terms of how like the world\nhas changed a lot since 2023. So we keep\nre-evaluating how we should be building\nthings constantly and now we have a\nmillion token input context window. We\nhave prompts that are really cheap and\nthey're also really really fast. So some\nof the things that we held true in terms\nof compressing knowledge and context, we\nno longer hold true. Here's an example.\nSo if you look back at this pipeline I\nshared before, um here's an updated\nversion that we're experimenting with\nnow, which is getting rid of episodic\nand entity level compression in favor of\nreal time Q A. So that means that like\ndepending on your system maybe you don't\nneed to be compressing context at all\nbecause again like I said at the\nbeginning the raw data is always the\nbest source of truth. So it's like why\nwould you create a secondary artifact as\na stepping point between you and what\nthe user's asking. Ideally you just want\nto examine the context. And so we do\nthat pretty frequently depending on how\nmuch data we're dealing with. We try\nbasically not to do to do the minimal\namount of engineering possible\nand our theory kind of going forward is\nlike this trend will only continue. So\nwe think the procedural memory and like\nbasically the insights the\ninterpretation and analysis that the\nthing does is the important part of\nmemory. It's like the record of its\nthoughts about you and kind of its notes\nto itself is the important part. You can\nalmost separate that from retrieval as a\nproblem. you can say like okay maybe\nthere'll be an infinite log of like my\ninteractions and model notes will be\ninterpolated in in the in the future and\nso maybe we don't even have to deal with\nretrieval and context compression at all\nso I guess if I want you guys to take\naway one thing it's like the perfect\nmemory architecture doesn't exist and\nstart with kind of what your product is\nsupposed to do and then think from first\nprinciples about how to make it work and\ndo that all the time because the world\nis changing and you might not need to\ninvest that much in memory\ninfrastructure.\nThat's it. So you can follow us at\nTwitter, a new computer. Thank you.\n[Applause]",
  "transcript_chars": 10602,
  "ingested_at": "2026-05-15T04:40:05.766896+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 4853,
    "like_count": 172,
    "channel_id": "UCyR2Ct3pDOeZSRyZH5hPO-Q",
    "categories": [
      "Science & Technology"
    ],
    "tags": []
  }
}