{
  "video_id": "FcnLiPyfRZM",
  "channel_slug": "machinelearningstreettalk",
  "channel_handle": "machinelearningstreettalk",
  "title": "29.4% ARC-AGI-2 🤯 (TOP SCORE!) - Jeremy Berman",
  "duration_seconds": 4108.0,
  "url": "https://www.youtube.com/watch?v=FcnLiPyfRZM",
  "upload_date": "",
  "transcript": "You can describe every single ARCV v2\ntask in 10 bullet points of plain\nEnglish, most of them in five bullet\npoints. And I think that this actually\ngets to the heart of ARC, right?\nEverything is quite simple. It's not\nvery hard. And I think this is also how\nwe do it too, right? Like we when we\nlook at these arc graphs, we're coming\nup with these bullet points in our head\nand we're, you know, checking them.\nOkay, uh this was right, this was right.\nAnd Python doesn't have these features.\nIt's just not as expressive as natural\nlanguage.\nMLST is sponsored by Cyber Fund. Link in\nthe description. I get actually even\nmore fundamentally like the ideal system\nwould be we have a set of data. Our\nlanguage model is bad at a certain\nthing. We can just give it this data and\nthen all of a sudden it keeps all of its\nknowledge and then also gets really good\nat this new thing. We we are not there\nyet. And that to me is like a\nfundamental uh missing part. Really what\nyou want is more expressive program. And\nso that's why I switched from Python to\nEnglish which is a much more expressive\nprogram. You can langu you can always\nteach a language model skill, right? But\nit's the meta skill. the skill to create\nthe skills that is AGI and to me that's\nreasoning like reasoning is that meta\nskill and so uh to put another way I\nthink if you fundamentally learn the\nskill of reasoning you should be able to\nthen uh apply that skill to learn all\nthe other skills that is the meta skill\nyou know kick whatever weights out you\nneed to uh align the model to reason and\nthen from there you have a foundation\nfrom which you can actually build\ngeneral intelligence okay folks hot off\nthe press. Many of you would have seen\nlast week that Jeremy Berman, who is a\nresearch scientist at Reflection AI, is\nnow the winner of the RKGI V2\nleaderboard, uh the public version of\nthe leaderboard. He's using an\nevolutionary approach. Now, remember\nlast year in December, he published a\nsimilar approach, um generating Python\nfunctions and then refining those\nfunctions in a kind of iterative loop.\nHis new architecture is generating\ndescriptions of algorithms rather than\ncode and iteratively in an evolutionary\nsense refining those ones and discarding\nthe ones that don't work. He's now at\nthe top of the leaderboard. It's a\nreally really cool and elegant\nalgorithm. And um by the way he works\nfor reflection AI. So he's doing\nreinforcement learning with verifiable\nfeedback and he's trying to address the\nbiggest gap in AI at the moment which is\nthat we want systems that can synthesize\nnew knowledge and new understanding.\nCurrent systems um just get trained with\na whole bunch of data and they only know\nwhat they've been trained on. They can't\nkind of think outside the box by\ncreatively synthesizing new knowledge.\nprolific are really focused on the\ncontributions of human data in AI. And\nthe reason this is important, it's\nactually the dirty secret of Silicon\nValley, the extent to which human data\nis used to evaluate and fine-tune um AI\nmodels. The reason for that, as we\ndiscuss in today's show, is that current\nAI does not understand the world in a\ngrounded way. It doesn't have a deep\nabstract understanding of the world.\nWhich is why the only way that we can\nmake AI work effectively is by grounding\nthe generation and supervising the\ntraining of AI models with human data.\nProlific are putting together a report\non how human data is being used in AI\nsystems and they need volunteers. Um you\ncan just go and fill out this form to\nhelp them produce this report and you\nwill get privileged access to see the\nreport before anyone else. The link is\nin the description. Oh, there was an\namazing part in I think it was in your\nfirst paper where you said, \"A parrot\nthat lives in a courthouse will\nregurgitate more correct statements than\na parrot that lives in a mad house.\"\nThank you. Thank you. My sister, who\ndoesn't know anything about language\nmodels or AI, she pointed that out and\nsaid that was a great line. So, at least\nI have that.\nI've already used it. I I credited you,\nbut I'll be using that quite a lot. Um\nwell Jeremy it's amazing to have you on\nMLST. I've wanted to to have you on ever\nsince you um released your first blog\npost. Um you know it was December last\nyear. I was at Nurips at the time and um\nat at the time you you actually got the\nhighest score on the um the public ARC\nv1 leaderboard just before the famous 03\nlaunched. Do you remember when when they\ndid this ridiculous $200 per task thing\nand and they knocked you off the board?\nUm Jeremy, can can you just tell the\naudience a little bit about yourself and\num and and and maybe we should start\nwith your with your first ARC solution?\nYeah, sure. So, I actually have only\nbeen working in research for about 8\nmonths. Before that, I had a company um\nright out of college. I was got into Y\nCominator and so I've been running a\ncompany uh for the last four and a half\nyears as CTO and I've always been very\ninterested in reasoning in the brain.\nUh, and I actually picked up Jeff\nHawkins book, A Thousand Brains, and I\nread that and at the same time I was\nkind of coming into language models and\nsomething just clicked inside of me and\nI just knew I had to be working on this.\nAnd you know, I believe that uh general\nintelligence, artificial general\nintelligence will be the most important\ninvention uh of hopefully my lifetime.\nAnd so I decided to drop everything. I\nstepped down as CTO. Company's still\ngoing well. So it was a difficult\ndecision. And um I actually had gotten\nin touch with Mike and Francois because\nI thought ARC AGI was such an elegant\nway of describing the problems with\ncurrent language models uh and their\ndifference between you know the human\nbrain and so I just kind of dug in that\nwas my first research project\nindependently and yeah I ended up\ngetting the top score on that uh and\nthat was um really great and after that\nI got recruited into Francois and Mike's\nAGI lab India where I was working on\nprogram synthesis this and as you\ndescribed earlier um over time I've\nbecome more convinced that language\nmodeling with reinforcement learning uh\nwill yield uh generalization um far\nbeyond what we see today and so um I\ndecided to uh move to a company that was\nfocused purely on language models and\nthat's where I am now so I'm uh\ncurrently working on reasoning and post-\ntrainining at reflection which is uh\nwe're building frontier uh foundation\nmodels.\nVery cool. Maybe we should save that bit\nfor a for a tiny bit later. But, you\nknow, one of the take-home messages in\nin your Well, no, I mean, it's it's\nsuper interesting and one of the take-h\nhome messages on your new approach is\nthat rather than producing um explicit\nprograms, you are evolving descriptions\nof programs. And France is a neurosyolic\nguy. He thinks that we need to have a\nsymbolic substrate where we you know\nrepresent the the kinds of problems that\nthat we can do and we need to do this\nkind of compositional form of of of\nintelligence. So we need to kind of be\nworking in the symbolic layer but\nperhaps guided by um you know deep\nlearning models but maybe we we should\nget to there um in a minute. So in your\nin your first solution it was an\nevolutionary approach. it was using set\n3.5 and you had um about four iterations\nI think and and essentially you you know\nyou were working on the arc challenge\nand you were producing these programs\nthrough evolution maybe just for folks\nthat don't know about the arc challenge\nas well could you introduce that and and\nget into your solution\nsure yeah so the arc challenge is kind\nof like an IQ test for machines it's a\nset of input output grids and the whole\npoint is to be able to figure out how to\ntransform input grids into output grids\ngiven a common transformation rule. And\nso uh what's interesting is these are\nreally easy for humans, right? The\naverage human gets around 75% accuracy\non ARC v1 and at the time the best\nlanguage models GPT4 sonnet 35 uh was\ngetting maybe 5%. And so yeah, basically\nthe idea is you have a few training\nexamples and then you're you're uh\nyou're trying to extrapolate the\ntransformation rule uh on the final test\nexample. And uh so I approached this I\nwas actually inspired by Ryan Greenblat\nwho had a solution earlier which was to\ngenerate a ton of Python programs that\nwould encapsulate the transformation\nrule and Python programs are great\nbecause they're deterministic and you\ncan pretty quickly check whether or not\nthe Python program works or not which is\nreally cheap. So it's cheap to verify\nand you can be relatively sure if the\nPython program works on all the training\nexamples that it'll work on the test\nexample. So uh I started with his\napproach but then I noticed that uh you\nknow the language models actually\nstruggled on first attempts. Even if you\nask the language model a thousand times\nto generate Python programs they were\nalways off by small amounts on easy\ntasks which I thought you know\npresumably it's in their distribution\nthey should be able to solve this. So uh\nwhat I found is that actually by taking\nthe top uh programs the top performing\nprograms and then running that in a\nrevision loop. So asking uh Sonnet 35,\nhey uh here's what you got wrong. Here\nare the cells you got wrong. Here's your\noriginal Python program. Improve it.\nThat started to really work well. And\nthen I thought, well, uh why not just\nincrease the depth, right? Uh why not\nask it 10 times uh you know, to revise\nuntil u I'm happy with the solution\npasses some sort of accuracy threshold.\nSo um that's kind of how I was inspired\nby it. And you know, I didn't think of\nit as evolutionary at first. I was just\nthinking about you know broadly what\nwould work and uh over time I I kind of\nunderstood you know there was something\na bit deeper going on here which is that\nyou know uh evolving solutions is a\npowerful technique generally and um I\nthink it's actually going to play a role\nyou know in future technologies but uh\nyeah that's generally uh guided by just\nintuition. Yeah, I I had Ryan Greenblat\non the show. I'm a huge fan of of his.\nHe's a very very smart guy. And I asked\nhim a similar question because he he did\nthis iteration, right, where where you\nhave a certain depth of iterations. And\nI guess one approach is that you have a\nlike a shallow method, right? So you\njust tried 200 different variations. And\nin your in your um blog post, you kind\nof said there's a there's a goldilock\nzone where you want to have a certain\nnumber of tries of you know different\nver you know different variations of\nthings but you also want to be able to\num refine your solution because that\nallows you to do this kind of\ncomposition and composition is very very\nimportant for things for problems that\nrequire iteration. And indeed the second\nversion of the arc challenge um I I\nthink the tasks were selected so that\nthey had at least a couple of iterations\nin them which meant that they needed to\nhave this depth. Can you talk about that\ntrade-off?\nFirst of all the second the ARCV2 is in\na sense fundamentally different than\nARCV1 uh because of what you're\ndescribing. They're compositional. There\nare many rules that you have to go\nthrough and this is partially why I\nfound that my solution on ARC v1 did not\nperform well. Um yeah, so there's a\nconstant trade-off uh between how deep\nyou go, how many revisions you take, and\nthen how how broad you start out. Uh the\nproblem with, you know, going deep and\nnot going so broad is there are some\nedge solutions where you'll never you'll\nnever get to, right? Um but then of\ncourse most solutions end up being uh\nsomewhere within the bounds of your\nfirst broad attempt. Um so that's\ngenerally the trade-off and uh the\ntrade-off is different for ARV1 or\nARCV2. Interestingly, I found that for\nARCV2, it was more important to be\nbroad. And I think this is surprising to\na lot of people. And partially this is\nbecause the models now think and that's\ngreat. So the models actually do a lot\nof the deep revision for you in their\nthinking block. Um, and this is a\nfundamental change from when ARCV1\nexisted and when I, you know, just\nstarted out in the field. I think I'm a\nbit embarrassed by a lot of the things\nthat I wrote in my first post because it\nwas 2 weeks before 01 was released and\nyou know everything about 01 changed how\nI think about these things. Uh which is\nbefore you could kind of um simulate\nemulate thinking to to the quote you you\nuh described with the the parrot um the\nstocastic parrot. I think uh before you\nhad reinforcement learning over um\nbefore you actually taught the machines,\nthe language models to think with\nreinforcement learning, you were almost\ndoing this stochastic guessing that was\nnot a very efficient uh uh revision\nloop, basically internal revision loop.\nAnd so you needed to artificially create\nthat revision loop with code. Um but in\nV2, I was able to use a very powerful\nthinking model which has a lot of the\nyeah the deep revisions for it. So I\nfound it was best to kind of increase\nentropy, let it explore the space\nitself, and then I'll add a revision\nloop on top of that. But the revision\nloop um is less important in V2.\nYeah. So the first one was Sonet 3.5. So\nthat didn't have this thinking thing\nbuilt into it. So in your prompt, you\ntold it to think step by step. And there\nI I think you were inspired by Ryan\nGreenbat's Greenblat's prompt, right? So\nyou had a a whole bunch of ways in there\nfor representing the board state. and\nyou said, \"I want you to think now and I\nwant you to, you know, that this is a an\nabstract reasoning challenge and and I\nwant you to think from first principles\nand it would kind of go through that and\nthen it would give you the answer.\" But\nyou're saying that on the um the RL\ntrained models, it was significantly\nbetter at doing that.\nRight. Exactly. You could think of uh RL\ntrained models as having in-built\nrevision loops. Um I you know they are\ntrained to explore the space in a deep\nway. um thinking generally for\nthemselves in a way that uh so you\nreally don't you know thinking uh you\ndon't need to prompt um thinking models\nto think step by step they already do it\nyeah I mean I wanted to challenge you on\nthis a tiny bit right so you kind of\nsaid in in your um I think it was in the\nsecond version of of the blog post that\nyou just released last week that at the\nmoment the models can do domain specific\nthinking so they can do math thinking\nand they can do code thinking and what\nwe want to do is is imbue like the core\nmachinations of thinking into these\nmodels and I'm a little bit skeptical. I\nfeel that these models because they're\nnot too incomplete because they're not\nsymbolic, you know, similar to what\nFrancois believes that I'm sure you read\nthat LLM biology paper as well. They\nwere talking about these circuits that\nwe can find in papers that do things\nlike multiplication and and addition.\nAnd what we what we saw was that they\nare quite patterned. They're quite\ntemplated. They're not thinking in a\nvery general sense. And my suspicion is\nit will always be that way because the\nmodels don't have semantics. They're\nnon- symbolic and and so on. Do you\nthink we could ever make them truly\nthink in a general way?\nYeah, I think fun I think fundamentally\ntaking a step back the fact that our\nbrains can do it and our brains are\ngenerally running similar algorithms\nuh to me this means that we will\neventually be able to um inject general\nreasoning into the language models. I\ndon't think there's a fundamental reason\nwhy uh neural networks can't behave like\nbiological neural networks. Um so that's\nI guess the higher the higher level\npoint. And then uh zooming in right now,\nyou know, the models are as bad as\nthey're ever going to be. There uh\nthere's generally more compute going\ninto pre-training than there is\nreinforcement learning. And uh of the\ncompute going into reinforcement\nlearning, you know, a subset is going\ninto specific general reasoning. And so\nI think that over time you're going to\nsee the models get better and better at\ngeneral reasoning. But I guess a\nquestion I would have for you is do you\nthink there's a fundamental difference\nbetween the way the brain works uh where\nthere's some sort of symbolic nature to\nthe brain and and it's not possible to\ninject that type of uh nature into an\nartificial network.\nUh yes. Yeah. I mean you mentioned Jeff\nHawkins. I I interviewed Jeff. He's\nabsolutely amazing. And of course his\nhtm algorithm is computationally\nstronger than a neural network. It's\nit's too incomplete and our brains even\nthough they are finite they run a tour\nincomplete algorithm which means our\nbrains know how to expand their memory\nright our memory we can go and write\nthings on a whiteboard and we can go and\nyou know get another notebook and that\nis a special type of algorithm which is\nnot traversible with stochastic gradient\ndescent. So you know the the rough\nargument is yes there is there is a\ndifference there and I also wanted to\ntouch on this um you know RL with\nverifiable rewards thing which is that\nwe do that at training time. I'm very\nexcited in the future about an active\ninference version of that like an\nagentic version where we're actually\ndoing this kind of um transductive\nactive fine-tuning in an agential way\nright so you know I I take an action I\nget some new information from the\nenvironment and I update my weights and\nthat would be truly adaptive that would\nbe intelligent but what we do now is we\ndo all of this stuff at training time\nand the resulting frozen artifact is\nstill an LLM it still has just a bunch\nof patterns in there And I think that\nwhile that can uplift reasoning in many\nways, I don't think it it has the\nintelligence and and according to\nShalet, intelligence is simply the\nability to search through the space of\ntouring programs, right? And I don't\nthink that's what's happening with these\nLLMs at the moment. I I think you're\ngenerally correct u that it's not\nhappening at the moment but I still\nthink uh like fundamentally I don't\nthink there is a like a fundamental\nblocker physically for why they won't be\nable to do it in the future and it's\npossible that SGD right like stocastic\ngradient descent is is an issue\nfundamentally and I think we're going to\novercome that. I guess what I would say\nis artificial neural networks I think\nhave the structure um capable of yeah uh\nbasically um being as smart in every way\nas a human brain and I subscribe to uh\nFrancois's definition of of general\nintelligence as well.\nYeah. I mean I I think we mostly agree I\nthink I mean you know let's look at\nAlpha Zero or Muse or something like\nthat. um they did this training loop\nwhere they were actually updating the um\nyou know like the the value network and\nthe policy network and then it was\nfrozen and they did some kind of you\nknow um Monte Carlo research. So they\nwere achieving adaptivity through\nexhaustive search during the actual\ngames and in an ideal world we would\nhave this adaptivity that's actually\nupdating the weights. Now, I believe the\nonly reason we can't do that at the\nmoment is just computational\ntractability, right? We have these huge\nmodels. We couldn't possibly have a um a\ndynamically updating model for every\nsingle person that's using chat GPT. It\nit would it would just be ridiculously\nslow. But I think you and I agree that\nif that were possible, that would be an\nentirely different kind of form of\nintelligence.\nI don't think that's so intractable.\nActually, I think um my guess is and uh\nyou know uh Nvidia just put in a hundred\nbillion dollars into open AI uh open Sam\nAlman's plan is to produce a gigawatt of\nwhat of of compute a week something like\nthat. I actually don't think with uh you\nknow ever efficient uh um algorithms\nthat that is like crazy far off. I mean,\nright now you could buy a GPU, you could\nhave it running in your house and it\ncould be running OSS 12B, right? And\nfine-tuning is relatively trivial\ncompared to, you know, the entire\nprocess for pre-training. I actually\nthink that is a that is totally within\nthe realm of possibilities in the next\n10 years. Um, and I think that actually\nis is potentially where this goes. Yeah,\nI mean you know far more about this than\nI do, but I think the reason why\nfine-tuning is so expensive is, you\nknow, we have this continual learning\nproblem. And when you fine-tune a model\non Open AI, they're not just fine-tuning\nit on the data you give them. They, you\nknow, just to stop this catastrophic\nforgetting problem, they presumably have\nto sample in a bunch of the original\ntraining data and maintain the\ndistribution and so on. And and if if\nthey did this for everyone, it would be\ninsane. But I am excited about it just\nlike you are because I I interviewed the\narchitects and um I I think they got\nfirst place on the on the private\nversion last year and they were doing\nthis transductive active fine-tuning.\nRight. So they and they actually said by\nthe way that this is a curious oddity\nwith transformers that if you start with\na um you know almost like a virgin 8\nbillion transformer it almost doesn't\nmatter what it knew about before. You\ncould just pretty much start training it\nfrom scratch on the arc challenges. So\nthey did a whole bunch of augmentation\nand active fine-tuning and they built an\nintelligent artifact. I mean\nintelligence is domain specific as per\nShetle and they actually built this\nsystem which was per task adapting and\nsolving the tasks and they were updating\nthe weights and it was beautiful. So\nthat was an existence proof if nothing\nelse that this thing could work\nand that was on a Kaggle notebook.\nYeah. U you know in 10 years this is\ngoing to be what like the you know\nApollo mission computer. Um, I think\nthat I think what you're describing, I'm\nactually not even totally convinced that\nuh continual learning is fundamentally\nthe blocker. But I think if it is the\nfundamental blocker, that's actually\nincredible because we will solve\ncontinual learning. Like that's\nsomething that's physically possible.\nUm, and and I actually think like uh\nit's not so far off. Now the the uh\nforgetful issue that is a much more\nfundamental issue in my mind and not\njust that the fact that you need to uh\nevery time you fine-tune you have to\nhave some sort of very elegant uh\nmixture of data that you know uh goes\ninto this fine-tuning process so that uh\nthere's there's no catastrophic\nforgetting. Um this is I think actually\na fundamental problem. So I the the um\nand and it's a fundamental problem that\nthat you know even open AI has not\nsolved right. Um, and I think Francois\nhas a great example and I think this is\nan important example. You know, if you\nhave the perfect weights for a certain\nproblem and then you fine-tune that\nmodel on more examples of that problem,\nthe weights will start to drift and you\nwill actually drift away from the um\nfrom the correct solution. His answer to\nthat is well, we could make these uh\nsystems composable, right? Like we can\nfreeze uh the correct solution and then\nwe can add on top of that. I think\nthere's something to that. Um I think\nactually it's possible that there's a\nresearch direction uh that where you\nknow maybe we freeze experts or may\nmaybe we freeze uh layers for a bunch of\nreasons that isn't possible right now or\nbut people are trying to do that but I\nyeah I think uh fundamentally compute is\nnot the issue. I think it's this\ncatastrophic forgetfulness.\nYeah. So I'm inclined to agree. I've\nI've long dreamed about there being a\ndocker for language models. Right. you\nknow in Docker you can kind of freeze\ndry a state of you know like a let's say\nLinux operating system with an\napplication with this security updates\nyou have these kind of immutable layers\nand the composability that we often talk\nabout could actually happen at the\narchitectural level and we could do\ndynamic model merging between different\nlayers and and whatnot that would be\nvery very exciting so you know but also\njust to come back to what you said\nbefore I've never really heard this\nbefore you're distinguishing like\nforgetting and learning right when when\nwe were about you know catastrophic\nforgetting and continual learning. Um\ncan you just sketch out that distinction\na bit more? So uh the way I think about\nit you know you have a neural network\nand it has all these weights inside of\nit right anytime you update those\nweights you are pushing some weights out\nand presumably you are pushing some\ncorrect answers that you've previously\nyou know trained and they are getting\npushed out and the benefit I think\nfundamentally of symbolic systems is\nthat doesn't happen right symbolic\nsystems are deterministic when you get\nthe right answer you can be sure you\nhave the right answer you stash it away\ninto your library of correct solutions\nuh this is this is the problem with\ncontinuous structures but this is also\nactually now why I think um you know\nit's important to draw from the brain\nwhich is this similar thing happens\nactually with the brain I believe the\nbrain is much more composable than uh\nneural networks biologically but I think\nthere's no reason why we can't uh you\nknow we we won't be able to figure this\nout right like again uh it could be it's\nit's as easy as we end up freezing\nexperts uh again like the the freezing\nof the layers I think this is an an\nunderexplored uh area and I think It's\nactually I think we're going to go\nthrough um basically this RL S-curve and\nthen I think this is the next Scurve is\nuh figuring out how to make language\nmodels composable uh like figuring out\nhow we to I guess actually even more\nfundamentally like the ideal system\nwould be we have a set of data our\nlanguage model is bad at a certain thing\nwe can just give it this data and then\nall of a sudden it keeps all of its\nknowledge and then also gets really good\nat this new thing. we we are not there\nyet and that to me is like a fundamental\nuh missing part of general intelligence.\nYeah, completely agree. So it it sounds\nlike we have very very similar\nintuitions and and Shal talks about this\nas well. I mean interestingly in his\nmeasure of intelligence paper it was\nactually about the measure of\nintelligence. He's never really spoken\nabout the machinations of intelligence.\nHe talks about it just um casually. He\nsays, you know, we need the spell key\nprize and we need to so those are like\nthe basis functions and we need to do\nlibrary learning and library transfer\nand we do some kind of like, you know,\num symbolic compositional process, you\nknow, to adapt to novelty. And he's kind\nof sketched out the the the mechanics of\nit, but he's never actually formally\nspoken about it. I assume that's what\nhe's building at at his company. But um\nthere was um a famous guy called Jerry\nFoder in 1988. He had this connectionism\ncritique. he had this beautiful paper\nand he was basically saying that\nsymbolic systems have um systematicity\nand productivity and systematicity is\nthis compositional thing. It's that, you\nknow, it's the ability to generalize\nbetween Mary loves John and Mary loves\nJane, right? So, so you have semantics,\nyou have these kind of like, you know,\num the these symbolic relations and they\nhave certain computational properties\nlike you can do variable binding and and\nquantification over potentially infinite\ndomains. Like we intuitively understand\nthat symbolic things have very\ninteresting properties. And then what\nwe're trying to do is like we know\nneural networks are really good and we\nwant to somehow graft this capability\nonto neural networks.\nYes. Yeah. And I actually think um\nneural networks are in some ways a\nsupererset of symbolic systems. I think\nuh you can generally\nyou should be able to encapsulate a\nsymbolic system with a neural network in\nthe same way I think that you can do the\nthe I think you can do the same thing\nwith the brain as well. Um I think\nthere's no there's nothing fundamentally\nblocking but of course once you have you\nknow you have this symbolic system in\nthe neural network it might\ncatastrophically forget when you\nfine-tune it. Like I guess that's uh\nthat might be where we we disagree a\nbit. I think everything you're\ndescribing is totally possible, but then\nwhen you're uh when you're coming to\ntrain it again, there's no guarantee\nthat it keeps the same structure.\nI think it's possible because a neural\nnetwork is is not too incomplete. So I I\nthink in principle it can't do many of\nthese things, but you can build a\ncontroller, right? So you you could just\nbuild a very simple um uh kind of um\nenvelope just as you did with your\nsolution. So you you you had a bunch of\ncode and it was doing this you know this\nbasically compositionality in code um on\ntop of the neural network substrate and\nthat gives you many of those things you\nknow for example we often talk about\nlibrary learning and library transfer\nand I'm not sure if you've seen Eric\nPang's solution um I'm speaking to him\nin in Hong Kong actually in a couple of\nweeks but but he um rather than the\ndream coder approach where they do this\nexplicit library learning um he was\ndoing it in a kind of implicit way using\nthe LLMs and I I think there's a whole\nspectrum between you know you don't have\nto do it explicitly. I think you can\nkind of use neural networks and you can\ndo some kind of implicit composition and\nyou can get many of these features.\nAlso I I want to say um generally when I\nspeak about language models I assume\nthat they have um basically a Python\nterminal that they can run.\nOkay.\nSo uh not so I guess two things. The\nfirst is I think uh if you have a large\nenough neural network um I think uh\ngenerally almost everything is you could\nyou can represent a symbolic system but\nof course it's not turning complete um\nbut given a neural network plus uh the\nability to write programs then um I\nthink we're we're basically at the we're\nat the um human brain equivalent.\nSo yes so that that is a hybrid system\nand that certainly is significantly more\npowerful. Um I'm just regurgitating my\nco-host Dr. Dugar because this is his\nlike favorite point he always likes to\nmake. But but he says that um that's\ntrue but stochcastic gradient descent\ndoes not find the algorithms that allow\nthe systems to behave as if they are\ntouring machines. God knows how it\nhappened in our brains. There is some\ndent of evolution or something where you\nknow we suddenly got the merge operator\nor god knows what happened and and we've\ngot this this incredible like touring\ncomplete algorithm in in our finite\nbrain. And so so we're getting into that\ntrainability thing. Yeah. You know, so\nyes, maybe there is an out, you know,\nthere there is a set of weights that we\nmight find one day and it can access\nlike a Python, you know, tool and it can\ndo all of those things and whe you know\nits capability now is it now effectively\nsearching the space of Turing machine\nprograms. I think it's not like there's\nlots of problems there like how how\nwould it know which ones halt and which\nones don't and how would it be able to\nefficiently search that space? It feels\nlike there's a gap now, but I agree with\nyou that it's significantly stronger\nthan not being able to use the tools.\nYeah, I think I I but but you think that\nthe human brain is running a touring\nsystem.\nYes, I think the algorithm that runs in\nour brain is a is a touring machine\nalgorithm. So you know like a a touring\nmachine has um a code book which is a\nfinite state automter and then it has\nthis like you know read write access to\nthese two potentially infinite tapes and\nyou know the the the algorithm that you\nput in that touring machine that is very\ndifficult to find. I I don't disagree\nwith that, but I don't why why wouldn't\nwe be able to find that algorithm for\nneural networks, right? Like why\nwouldn't uh you know we we train uh\nneural networks much bigger than the\nbrain. We put a lot of compute towards\nthem. Do you you just don't think that\nuh finding the same algorithm is\npossible with with SGD? I think I think\nwith SG because the fascinating thing is\nthat you know if you look at all of the\nFSA algorithms a a tiny sliver of those\nalgorithms are capable of controlling\nyou know a touring machine and expanding\ntheir memory and so on. it's in the\nspace and I don't know if you saw that\namazing paper by Kenneth Stanley the\nfractured entangled representations\npaper and he had this beautiful diagram\nand he said that um you know SGD finds\nthe algorithms over here and neuro\nevolution algorithms find the ones over\nhere and it just so happens that the\nneuroeolution algorithms find ones that\nhave these factored you know\nrepresentations which means they they\nfind representations that are about the\nworld that are grounded in the world\nthat carve the world up by the joints\nand if only we could find those things.\nYou know, when I spoke to Schmid Huber,\nhe said the same thing. He said like,\nyou know, it is actually possible to\nfind the right weights in a neural\nnetwork to make it, you know,\neffectively incomplete with some caveats\nand so on. But when we do SGD because\nthere are all of these shortcuts, right?\nIt's a bit like good hearting. It it\nwill always just find the wrong thing.\nI need to think about that a bit more.\nOkay. So on the first one as well, you\nwere generating Python programs\nexplicitly and because of all the things\nthat we were just talking about, I'm a\nbig fan of that because I feel\nintuitively and I think you did that\nthere's something special about Python\nprograms and and then you you did this\niterative updating of of those programs\nand and you converged on the right one.\nYou also had this amazing diagram in\nyour first blog post where you kind of\nvisualized the space of all the possible\nprograms and you kind of showed what was\nhappening in every iteration. In the\nfirst one, I used Python programs\nbecause Python programs are\ndeterministic and it's really easy to uh\nverify whether or not you know it's\ncorrect. Did it did it run and then uh\ndid it run on the training examples and\nproduce the correct outputs. So it's\nlike a perfect program, right? Like it\nis a program. Uh the problem is it uh\nyou know Python programs are brittle in\nthat uh you know there are many things\nthat are very difficult to describe with\nPython. Arc grids in V2 being one of\nthem. Right? So you have some grids that\nare very easily described by Python but\nthen almost the majority overwhelming\nmajority in ARV2 are very hard to\ndescribe in Python. The correct Python\nformulation is you know lines and lines\nand lines. And um really what you want\nis more expressive program. And so\nthat's why I switched from Python to\nEnglish which is a much more expressive\nprogram. You can you can describe every\nsingle ARCV2 task in 10 bullet points of\nplain English. Most of them in five\nbullet points. And I think that this\nactually gets to the heart of ARC,\nright? Everything is quite simple. It's\nnot very hard. And I think this is also\nhow we do it too, right? Like we when we\nlook at these art graphs, uh arc grids,\nwe're coming up with these bullet points\nin our head and we're, you know,\nchecking them. Okay. uh this was right,\nthis was right. And Python doesn't have\nthese features. It's just not as\nexpressive as natural language. And I\nthink another way to put it would be,\nyou know, you have this inductive uh\ntransductive trade-off, right? Um\nyou could think of language models as,\nyou know, being trained inductively and\nthen um they have an inductive bias and\nyou almost want to let that inductive\nbias express itself fully in a way. Um,\nand the way you do that is to give it\nthe full power of how it was trained.\nAnd I think this is the same thing with\nhumans too, right? If I told you to uh\nsolve with Python programs, you do a way\nworse job even if you were an expert at\nPython. And so uh I think fundamentally\nit's more general and uh it leads to to\ngeneral and better solutions. I mean the\naccuracy is much higher when you use um\nnatural lang. Now the problem is you\nactually have to then verify whether the\ninstructions are correct. You can't run\nnatural language on our grids. This was\nthe fundamental problem with the\nsolution this is what made iteration\nchallenging uh especially because uh you\nknow for each grid for each training\nexample you have to run the natural\nlanguage instructions and it takes a\nreally long time especially with this\nthinking model. So, I originally started\nwith a weak model. You know, it's the\nchecker model. It's the checker agent.\nUh, let's just use GPT5 mini, whatever,\nnano. And it did terribly. So, I ended\nup it was actually more important that\nthe checker was stronger than the actual\ninstruction creator, which I I think is\nuh interesting. Um, but yeah, that just\nhighlights, you know, the the trade-offs\nwith using natural language. It's it's\nuh you can express uh you know much more\nconcisely um programs that you want to\nrun but then they're not runnable\nprograms. You actually have to check\nthem inductively. Um so that was the\ntrade-off but it was worth it for ARCV2.\nYeah. So so fascinating. Um and just for\nthe audience we've been using\ntransduction and induction to\ndistinguish predicting the solution\nspace versus predicting a program. Um I\nhad this this discussion with with\nClement Bonet we need not detain us now\nbut I think in traditional machine\nlearning transduction means that the\ntest example is a function of your\nprediction. I had this discussion with\nthe architects as well that when you\nhave this natural language description\nnatural language is more expressive\nwhich simply means that there are more\ndegrees of freedom and this is the\nbeauty of of LLMs that there's this huge\nkind of space that you're traversing\naround and when you use natural language\nyou can just traverse to more places in\nthat space more easily. So it seems like\nit would be a win and I'm really\nfascinated to to find out whether that\nis just like a huge component of your\nsolution because on Eric's solution he's\nstill predicting programs and still\ndoing very well. So I'm I'm not sure\nabout that. And the other thing is um I\nwasn't entirely sure whether you are\nactually using a transductive method. So\nin your solution checker agent is it\ndirectly going to the solution space or\nis it generating a program and testing\nit\nin the checker? Yeah,\nin the checker, it takes in the natural\nlanguage and then it outputs uh a grid.\nThat's all it does. It just outputs a\ngrid.\nOkay, cool. So, so you've moved to a to\na transductive um modality. Did you see\nany errors in that? So, did it sometimes\nproduce the wrong grid or\nall the time?\nYeah.\nYes.\nAll the time. Um and it it's worth\nnoting actually uh you know the Python\nsolution was I obviously I tried my V1\nsolution on V2 right um and\nit wasn't so much worse but part of what\nI wanted to do with V2 is show that as\nlanguage models get more powerful and we\nget to use thinking models we can start\nusing more general solutions and I just\nthought there was something elegant\nabout using natural language and then it\nalso happened to be that there were\nproblems that I could tell the Python\nuh functions we're never going to get\nand there are basically no programs that\narc v2 my solution for arc v2 won't get\nso when gro 6 comes out or gpt 7 comes\nout uh you can use my v2 solution and it\nwill win it will beat arc that is not\nthe case for my v1 solution the the\nother important thing is you're now\nusing gro 4 which is very very powerful\nI I assume you you chose grock 4 because\nit just happened to be the the winner on\non the leaderboard for for the base arc\ntoo and how much uplift is coming from\nthat so for example if you used gro 4 on\nyour on your solution last year how much\nbetter would it be\nso uh I actually don't think it would be\nso much better for some reason and this\nis what I talk about in my blog post\nthese language models are very spiky in\ncertain things where they were trained\nheavily on right so I think what\nhappened with grock is there was a\ndistribution of similar uh shape tasks\nright grid tasks just reasoning in the\ntype of general direction that allowed\nGrock to have a special capability in\nthis area. And I actually like uh tested\neach model. So I tested Grock versus\nGPT, you know, I didn't just go by the\nleaderboard. Um and Grock definitely\noutperformed. Uh the problem is, you\nknow, for my V1 solution, uh you also\nhave to generate code and Sonnet 35 is\nreally good at thinking about code and\ngenerating code and I I prefer Sonnet to\nGrock uh for code generation. So my\nguess actually would be that if you use\nmy V1 solution, it's highly possible uh\nyou know Opus 4.1 would be the best. I\nhaven't tested that. That would be very\nexpensive on Opus 41, but maybe it's\nworth testing. Um I but I think that the\ngeneral idea is that uh these networks\nare are very spiky when you get into\nspecific domains. The uh it it actually\nvery much matters which model you use.\nAnd the ARC is a great example of this,\nright? Like the leaderboard is super\nspiky uh in ways that uh other\nbenchmarks are not. I I did an\ninteresting interview at Europe's last\nyear um with the Google guys and and\nthey were talking about this adaptive\ntemperature in language models for\nreasoning because you know there's this\nconstant trade-off between reasoning we\nwant to be quite constrained right so so\nwe actually want to kind of like go go a\nparticular pathway we want to be\nconstrained by our knowledge and when\nwe're being quite creative and flexible\nwe want to we want to be able to go in\nin different places and I'm I'm really\ninterested in creativity for example and\nand and I think creativity is like you\nyou you it's very similar to reasoning\nas chalet talks about you know you're\ncomposing together these constraints\nthere's this fogyny of of knowledge and\nyou need to respect it as much as\npossible because if you don't respect it\nyou're not grounded anymore so it kind\nof feels to me that intuitively code is\ngreat because it means that I'm actually\nrespecting the constraints and and the\nsemantics are correct and it's grounded\nin in the real world. Do you feel in any\nway that by using these natural language\ndescriptions that you're kind of\ncreating something which might by dent\nof chance or search find the right\nsolution but is isn't correct and\nverifiable?\nYes. Okay. Yes. Tell me tell me more.\nUm yes I I for sure um I think generally\nuh when models think in natural language\nand they output a natural language they\nuh they are higher entropy right. uh I\nthink the when you the second you start\nprompting with code they go into code\nmode and this is you know there are a\nlot of papers that show right just by uh\nprompting it in a certain direction it\nactivates certain weights that are you\nknow just naturally lower entropy um but\nthat was part of the thing that I wanted\nI actually wanted to introduce entropy\nbecause uh you know still most ARC tasks\num for v2 the models don't get close\nright uh you know my solution was the\ntop and it's at 30%. So, I actually\nwanted to inject as much entropy as\npossible. Uh, which is partially why my\nuh prompts are so broad. You know, I\ncould definitely improve my accuracy on\na few tasks by making the prompts more\nspecific, but I wanted to just\nconstantly berate it like more entropy,\nmore entropy. Um, so I actually found\nthat to be a a positive, not a negative.\nInteresting. Interesting. On the\nefficiency of the solution, so the 03\nmodel from OpenAI, that was about $200\nper task, and that was I think it was\nDid we ever find out? I think it was\nsampling, right? So they just sampled it\na bunch of time so they had a basic\nverifier. Is is that correct?\nUm I don't think we ever figured that\nout.\nSo you think it could be? So because\nwhen I interviewed Chole, he was he was\nbeing quite kind of not ky, but it it\nseemed like he was suggesting they were\nactually performing a search algorithm\nand and I think the OpenAI guys said on\nTwitter, no, they were just doing very\nbasic sampling. And I think they even\npublished the code that they used. Um,\nyeah, I'm not quite sure.\nYes, I I also spoke to the Open AI guys\nabout this and I I I'm not sure after\nall of this what they were doing. I I\nthink it's probable that they were doing\nsampling.\nI mean, it took a very, you know, uh,\nhard to imagine they weren't doing they\nwere certainly doing sampling. I I'm not\nsure what else they were doing, whether\nthe model was fine-tuned. My guess is my\nmy best guess is that they were sampling\nand it actually was not a fine-tuned\nmodel.\nOh, ve very interesting. Yeah, I\nremember there was that big hoo-ha at\nthe time that they, you know, it was\nscandalous that they were training on\nthe training set. But um anyway, that's\none side. There's also like the thought\noccurs that if they did did if they did\ndo something like what you are doing, so\nyou know like this um approach of\niterative refinement with verification\nat every single step, would they have\ndone even better?\nFor sure. Yeah, for sure. I think OpenAI\ngenerally they want to do the right\nthing and they want their solutions to\nbe very general and broad. And this is\nthe sense I get. I think it's to their\nculture. And I spoke so I spoke to the\nOpenAI uh guys. They did include the\ntraining data in that 03 model. But I\nthink that's fair game, right? Uh they\nalso I don't think they uh fine-tune on\nit, right? So it's just part of the\ncorpus uh that went into pre-training,\nwhich to me is fair game, right? This is\nthis is totally fine.\nOkay, very cool. So So on ARV1, their\nefficiency was $200 per task. What was\nyour efficiency? Oh, on ARCV1 I maybe\n$10, something like that. Um, I need to\nsomething like that. Okay. In order of\nmagnet, may maybe I need I need to\ncheck.\nYeah. So, talk to me more about this.\nSo, and and Eric Pang's solution, he he\nwas slightly came in slightly worse than\nyou, but I think he was a fair bit more\nefficient. I think his one I'm going off\nmy memory now. Was it about $8 per task?\nWas yours about $30 per task on Arc V2?\nOh, this is the V2. So my yeah my latest\nsolution right my latest solution was\naround $30 on V2 and $8 on V1.\nOh okay. And I think Eric's one was\nslightly more efficient and and I and he\nwas indicating that it was because he\nwas doing the library learning and\ntransfer and and even that I I was left\nkind of thinking first of all it's\ninteresting that you got better results\nand is that because there isn't much\ntransfer you know where where does the\nlibrary transfer come into this? Because\nyou know like maybe like the broader\nquestion is if you were to make your\nsolution significantly more efficient\nwhat would you do? I had a version that\ndoes do library transfer basically I\nwould uh save the traces from training\nand try and basically pull those in\nduring test time and um I actually just\nout of simplicity sake because I was\ngetting such high scores with the simple\nsolution I wanted to just push the\nsimple solution and I might actually uh\nit's we'll see um if someone's going to\nbeat my score I might bring that back in\num I it's for sure that will improve the\nscore and it's useful there is a lot of\ntransfer efficiency I just found what I\nwas doing very um elegant and so I I\nactually like like to keep it. Um but um\nyou know no third party dependencies or\nanything like that. Uh but that for sure\nhelps accuracy. Um I think the\nfundamental reason why I got higher and\nalso I could match his efficiency and I\nwould still get higher um because I was\nusing natural language. Natural language\nis a much more efficient um area to play\nin.\nYeah.\nThat's at least what what I found.\nYeah. I I just wonder how close do you\nthink are we getting to the kind of\nparto optimal of of this approach? I\nmean just to give you a few examples um\nwe interviewed the alpha evolve team\nthat was fascinating and maybe you can\ncontrast with with those guys. Sakana AI\nyesterday, Robert Lange was the first\nauthor. they released this um I think it\nwas called shrinker and that was a kind\nof similar kind of evolutionary you know\num program thing and and they had some\ncool features in there like you know\nusing bandits and using UCB and I I\nguess like are we getting to the point\nwhere we're going to really figure out\nwhat is the most optimal way to do this\nby the way they were also switching\nbetween different foundation models\nI think uh improvements will be log\nright uh will be will be logarithmic so\nI wouldn't expect um using these jet\nbasically using the language models we\nhave today. I would not expect anyone to\nbreak let's say 40%. But you could\nprobably make my solution twice as\nefficient I would say. Um you wouldn't\nget such so much uh yeah you wouldn't\nget uh you wouldn't get more than a few\npercentage points more accurate is my\nguess but you could make it a lot more\nefficient. There's a ton of efficiency\ngains to be made. We've been dancing\naround this a little bit that you know\nShal's measure of intelligence was all\nabout um resisting memorization and\nthere is this question now you know\nwhich is to what extent are we actually\nbuilding systems that we might call\nintelligent and he says that\nintelligence is simply like the\nefficiency of knowledge acquisition and\nand I'm really on board with that and\nand I think it's fair to say at the\nmoment that um let's say like you know\nyour solution in Greenbat solution. It's\nquite ephemeral and stateless, which is\nto say that when you have a new task\ncome along, you kind of start again from\nscratch, which means it's not really\nlike um adapting and acquiring new\nknowledge and transferring that\nknowledge. So maybe you would agree that\nin the spirit of Shalet's measure of\nintelligence at at at the moment, it's\nmore of a kind of searching approach,\nbut what do you think we would need to\ndo to kind of, you know, make it more\nadaptable?\nRight. So I think uh test time uh\nfine-tuning would be the way to like\nfundamentally make it adaptable. But I\nalso think you know Shallay hits it uh a\ncore problem with language models which\nis they um their reasoning is domain\nspecific right I kind of in my blog post\nI described that um when you train a\nlanguage model to reason about math for\nsome reason most of the reasoning\ncircuits it just gained live in the math\nweights\nand then you try and train it on science\nand uh it gets some generalization but\nnot as much as you would want and I\nthink uh not nearly as much as what what\nhumans get. Humans have like this this\num you know generalization engine that\nis our reasoning capability. And this is\nthis is uh the fundamental hole in\nlanguage models today. And I think um in\nfact actually I would say I I actually I\nyou know generally agree with Francois\nand he says you know um langu you can\nalways teach a language model a skill\nright but it's the meta skill. the skill\nto create the skills that is AGI and to\nme that's reasoning like reasoning is\nthat meta skill and so uh to put another\nway I think if you fundamentally learn\nthe skill of reasoning you should be\nable to then uh apply that skill to\nlearn all the other skills that is the\nmeta skill and we need to figure out and\nso that is the fundamental problem and\nyou need to uh do whatever you can you\nknow kick whatever weights out you need\nto uh align the model to reason And then\nfrom there you have a foundation from\nwhich you can actually build general\nintelligence. So I guess I don't know if\nthat was a that was maybe a higher level\nanswer to your question but I think you\nknow how how we and what I'm what I'm\nfocused on is uh really just fitting uh\nall of reasoning into these models and I\ndon't really care what else is left. I\njust want all of reasoning in. Yes, I I\nI pretty much agree. And uh I mean, you\nprobably know that I'm Charle's biggest\nfan, so I've obviously, you know, been a\nhuge fan of his for years. But um by the\nway, he's just released um revision\nthree of his deep learning with Python\nbook. And I recommend folks to read\nchapter 19. You can actually read it\nonline for free. And he sketches out\nthis entire vision. You know, it's ab\nit's so exciting. And I I think that\njust to see it so beautifully\narticulated. Um because there is a bit\nof an elephant in the room in in the\nscene at the moment. I think so many\npeople just just don't have such a crisp\nunderstanding. But the only departure\nthat I make with Shalet and and and\nyourself Jeremy is that I think it's um\nyou know Shallay really focuses on\nbehavioral tests of intelligence you\nknow like so so it's reasoning if it can\npass the test and it can actually like\nget the right answer and I think we need\nto go this is where I was kind of\ntalking about the systematicity and the\nsymbolic AI. I think how you got there\nis important right so I think it's\npossible to get the right answer for the\nwrong reasons and I think that if we\nhave a system which has semantics so we\nactually know what these symbols mean\nand we've composed them together in a\nprincipled way not only to get the right\nanswer for the right reasons but to be\nevolvable so to have like an efficient\num epistemic base that allows us to go\non to acquire new knowledge in the\nfuture and that to me points to this\nneed to have a mechanistic like you know\nlike how are we acquiring this knowledge\nview? Would you agree with that?\nYes. And I I think about it a bit\ndifferently. So let me know if what I\nsay is aligned with what you think.\nOkay.\nSo to me uh pre-training is kind of the\nopposite of what you described. I view\nuh there's two two types of knowledge.\nThere's knowledge that is memorized like\nthe capital of New York or the Spanish\nlanguage and then there's knowledge that\nis deduced. So that's uh physics,\nspecial relativity, general relativity.\nRight? from axioms you can deduce these\nthings and it's a causal tree and then\nthere's another type of knowledge what\nis the capital of North Dakota that is a\nknowledge network it's not deductive\nit's not a tree and I think pre-training\ntreats all knowledge as a knowledge web\nit's embeddings that are connected but\nthere's no uh there's no guarantee that\nyou have the correct causal relationship\nbetween things and this is where uh the\nmemorization comes in and I think this\nis actually where compression fits into\nintelligence so My view is actually uh\nintelligence is compression in that you\nshould be able to deduce um you should\nbe able to build a knowledge tree uh\nbased on almost you can build a\nknowledge tree based on almost nothing\nright you can deduce uh so much of math\nyou can deduce special relativity from\nthe very you know uh the the very roots\nof physics and uh Einstein was extremely\nintelligent because the hints that he\nneeded to come up with special\nrelativity or zero right he could start\nfrom almost nothing and build up this\ndeductive tree And I think it's almost\nlike reinforcement learning and\nreasoning is the process of pruning our\nknowledge network and replacing it with\nthis tree. And um and until we have\nweights that represent the actual\ndeductive nature of knowledge, we won't\nactually get generalization. I don't\nknow if this this fits in, but this is\nkind of how I think about reinforcement\nlearning, which is replacing uh\nknowledge web with a with a knowledge\ntree.\nYes. Yes. This is this is brilliant.\nWe're getting to the center of the\nbullseye here. Absolutely. And I I\nremember I read in in the first version\nof your blog post that you were talking\nabout we need to do this kind of\ndeduction where we synthesize hypotheses\nand then we we test them and we do this\nkind of generate test loop and and that\nis what creativity is. It's what\nreasoning is because you know when I\nfirst read Shet's paper you know years\nago I was kind of like I didn't\nunderstand whether he was talking about\nacquisition or synthesis and I now\nunderstand he's talking about synthesis.\nSo reasoning is like you just like Lego\nyou you build you build this kind of\ntree this epistemic tree and actually\nthis is what we do. So there's a\ndifference between knowing and\nunderstanding. So knowing is is kind of\nlike at the high level and understanding\nis is actually like there's this whole\nyou know just imagine this big block of\nthis big Lego structure and you're\ntracing down the structure with all the\nbuilding blocks of how you got there.\nAnd I actually think even when you teach\nkids at university, what you're really\ndoing obviously like you teach you teach\nthem facts, but then they kind of\nsynthesize their understanding over\ntime. So they're doing this composition\nand they're kind of getting there the\nway that they get there. And and we need\nto build systems to do this. And there's\nthe perennial problem that you were\nsaying that in deep learning what we do\nis we kind of start with this big\npattern network and we kind of sparify\nit. And I think reasoning should be more\nabout synthesizing from building blocks.\nAnd I I think that when you synthesize,\nyou can actually um do types of\nreasoning which are not in the training\ndata, right? You can build things that\nsimply are not there. You can just think\nabout things and and and figure things\nout. So do you think of that as a gap?\nYes, I think that's exactly right. And\nthen the question is can you build the\nsystem with language models or not?\nYeah.\nAnd I I think you can build them with\nlanguage models. And I think the fact\nthat we're slowly climbing uh our\nability to synthesize new information is\na testament to kind of what I'm saying\nwhich is when reinforcement learning and\nreinforcement learning you know with\nverifiable rewards is is fundamentally\nensuring whatever circuits led to the\nright answer they must be consistent\nwith the deductive tree. So it's\nbasically like can you replace all of\nyour pre-trained weights with weights\nthat are uh coherent from the\nenvironment. The problem is all you know\nthere's so many weights there's so many\nweights from pre-training that it's very\ndifficult. So I actually think part of\nyou know one of my I guess hot takes is\nthat pre-training in many ways slows\ndown uh reasoning. It makes it harder to\nreason because I I think I think the\nanalogy I draw is you know you have\nconsultants that know the names for\nthings but uh couldn't deduce the thing\nand then you have finemen right who uh\ncan deduce anything and reinforcement\nlearning is is turning your consultant\ninto finemen and um this is yeah this is\nwhat I'm most interested in um and I\nthink it's it's interesting because you\nget to play at both sides you get to\nplay with the pre-training of okay well\nmaybe we shouldn't include these things\nand let the model figure it out in\nreinforcement learning um well there's\nno guarantee if we pre-train it like\nthis, you know, it's going to have the\nproper deductive uh circuitry. So maybe\nthis is best uh you know left for post-\ntraining. And I think this is uh not\nthis is this is a hot t this is not what\npeople currently think. I think people\nthink let's jam as much information as\nwe can in pre-training and then we'll\nreinforcement uh learn when need when we\nneed. Um but I think this this could be\nincorrect.\nYeah. And I I pretty much agree with you\njust with the caveats we discussed\npreviously that if we could do that on\nthe basis of representations that are\nactually like grounded in the world um\nrather than things that just you know\nhappen to give you the right answer for\nthe wrong reasons. That's that's\nabsolutely true. Um just a little bit of\na a curveball. So I think in the first\nversion of the article you said you were\ninspired by Yan Lun's Jeep these joint\num embedding prediction architectures\nand he's also um a big advocate of\nenergy based models which are really\ncool. I don't know if you've seen the\nrecent couple of papers that are\napplying it to transformers, you know,\nwhere essentially um it's it's kind of a\nstep towards um probabilistic models\nwhere you actually have this uncertainty\nquantification, you can do\ncounterfactuals and you actually have to\nsolve an optimization problem at\ninference time. So you can do adaptive\ncomputation and it's all very exciting.\nBut um I still have some reservations.\nBut do you think like architectures like\nthat are exciting?\nI think they are exciting. I'm a bit\nless excited about them. not because of\ntheir merits but because I think I was\nunderrating transformers when I wrote\nthat. So this was really before I\nstarted uh doing being an actual\nresearcher and uh you know building\ntransformers\num actually coding with them. Uh since\nthen I've had a new appreciation for\nlanguage models. I think where I was\ncoming from it was uh it seems like\nlanguage models are overfitting to the\nnext token, right? Like uh and Jeepa is\nso interesting because oh all of a\nsudden you have them predicting concepts\nand you know fundamentally it's we care\nabout concepts the words don't really\nmatter it's the concepts that matter but\nI think that language models do operate\non the conceptual level in the hidden\nlayers and um so that was yeah something\nthat I I slowly came to the realization\nof. So I think there's a lot of\npotential in Japa frameworks. I think\nthey're really cool. I hope people keep\npulling on them. But I think mo most of\nthe benefits I thought that came from\nJeepa exist in language models. I just\ndidn't see it at the time.\nYou know, a lot of people get an earworm\nwhere they just get obsessed about an\nidea and you're kind of thinking about\nit all the time. Like what is that thing\nfor you?\nIf we do have these uh language models\nthat have their weights that are aligned\nin this uh I guess the tree of\ndeduction, um it seems like we're\nmissing still one more thing which is\ncreativity which we touched on, right?\nSo you can have the correct deductive\ntree, but then how do you search through\nall of the possible premises you can add\nto this tree? How do you find the right\nones? And I think there are a few\nexperiments that I'm looking forward to\ndoing. Uh one of them is uh a blading at\nwill pre-training data and then uh\nbasically reinforcement uh building an\nenvironment to have the model um\nregenerate that information. So for\nexample, let's say you had the ability\nto ablate special relativity and all of\nthe physics that came from special\nrelativity from your pre-training data.\nThat is a gold mine of an environment,\nright? Uh because now you can uh prompt\nthe model, you can do everything. You\ncan really try and get it to to to\ndeduce special relativity. And my hunch\nis that part of the reason why models\nare not yet great at coming up with uh\nnovel solutions and information is\nbecause they don't have that uh that um\nthat the circuitry of invention and I\nthink that is actually a circuit that\nneeds to be developed and we don't have\nthe environments to develop that circuit\nyet. And I actually just saw today I\nthink OpenAI released this math uh I I\nskimmed it this math paper right where\nthey're coming up with uh they almost\ncame up with novel uh conjecture or\nsomething like that. And I think that's\nexactly what was in my head, which is I\nI want to be able to build environments\nwhere uh the model's never seen\nsomething and basically it it uh tries\nto deduce these new things uh that that\nare you know outside of the distribution\nand uh over time it it learns and\npractices and builds this invention\ncircuit. Um so I think it's like it's\ntwo things. It's uh reinforcement\nlearning to make sure that the knowledge\ntree is consistent and then making sure\nthat the circuitry to be able to pull\nfrom you know its entire corpus of of\nunderstanding of the world uh and and\nbring bring that in to kind of uh to\nfuel the innovation engine.\nYeah, I I think we're so close. I think\nthe only like slight disagreement is\nwhether there could be such a thing as\nan invention circuit. seems to me just\nlike that LLM biology paper and and and\nwhatnot that it would be very kind of um\npatented in the weights. It would be\ntemplated. It would be kind of you know\nspecific to certain um domains and I\nfeel that we can do it but we would need\nto build a controller um on the top. I I\nalso feel by the way that um creativity\nis very domain specific and what I mean\nby that is just with this Kenneth\nStanley view that there's this big\nfogyny of knowledge and I've noted that\nwhen I hire creative professionals they\nlike an editor for example they can't\nedit my show and even if they're really\ngood at editing other people's shows and\nit's because they simply don't know\nanything about machine learning. So, you\nknow, like I' I've just discovered the\ncrazy degree to which creativity is\ndomain specific. And I just wonder\nwhether there is an algorithm for\ncreativity because I've got this\nepistemic lens of creativity, which is\nthat, you know, it's simply about, okay,\nwell, I can access all of the ancestors\nin my tree and I can do some\ncomposition. It's like building Lego and\nso on. And and if I want to use\nsomething from the tree over here, this\nother branch, you know, maybe it's\ncompatible. maybe I can bring it in or\nmaybe I just need to start a new branch\nor maybe I need to jump over to the\nother branch and I'm not sure whether\nI'm applying the same algorithm when I'm\ndoing that.\nI think you I think you actually are and\nthat it's um you it's not sufficient to\njust be creative in this case. You need\nto be creative and you need to be\nknowledgeable otherwise your creativ you\ncan't build the tree. I think your\neditors can't build the the tree because\nthey don't have the deductive footing.\nUm that that would be my that would be\nmy best guess. But that's an interesting\nperspective. I want to think about that.\nI mean, fundamentally, right, creativity\nis knowing which axioms to include in\nthe next uh uh branch of the tree,\nright? So, you're at level five of the\ntree. How do you get to level six? It's\nknowing which assumptions to pile in to\nget to level six, right?\nIt's it's so beautiful that we're\nwriting an article about creativity at\nthe moment. And um I believe that in\norder to be creative, the the depth of\nunderstanding of the tree is very\nimportant. And you know um just per\nchole you know intelligence is is the\nefficiency of how much of the history\nyou can acquire. So like a university\nprofessor understands the tree very\ndeeply and that actually makes them\nunintelligible to a normal person. Like\nwhen Steven Wolf talks about the rouad\nall the time people have got no idea\nwhat he's talking about. He's actually\nbeing very expressive. He's talking\nabout things at a level of abstraction\nwhich you know can refer to anything but\nit's beyond most people's cognitive\nhorizon. But there is something to be\nsaid for that when when you can respect\nthe history deep down into the epistemic\ntree. The creative stepping stones you\ntake because they respect the history,\nthey actually have more evolvability,\nright? Because you're you're you're\nstill grounded to the real world. You're\nnot becoming incoherent. So there's\nthere's something there about really\nknowing things deeply is important.\nYes. Yes. I I'm for the record, I would\ndefine understanding as being able to I\nmean I think understanding is a\nspectrum. I think on one end it's\nmemorization and then which is zero uh\nunderstanding and then on the other end\nit's ability to deduce and uh and also\nto deduce correctly. I agree with what\nyou said. It's not enough to just you\nknow have the right proof. It's you\nactually have to have uh got you know\nunderstood the tree and that is like\nthat this is understanding and then\nintelligence is just how many things you\nunderstand. And so it's really just how\nhow um how wide and and high is your\ngarden of trees.\nYeah, we're pretty similar. I I would\nsay intelligence is the efficiency that\nyou can acquire the tree and reasoning\nis building the tree. Discursive\nreasoning is executing the tree and and\nand understanding is simply just\npossession of the tree.\nSo intelligence to you is the speed at\nwhich you can build the tree, not how\nmany trees you have.\nYeah.\nOr not how large your tree is.\nYeah. Yeah. Yeah. I think understanding\nis how much of the tree do I have.\nI think that's correct. Yes. Yes. cuz\nyou like a you could have a very\nintelligent child who doesn't know a lot\nabout the world but has the ability it's\nlike potential to build a tree.\nYes,\nit's uh yes I I think that's correct.\nThere's a spectrum of understanding. So\nlanguage models famously they don't\nunderstand the tree very deeply. So they\nthey only understand the tree a few\nlevels down. And so when language models\nare doing autonomous generation, the\nreason why we have to do so many\ndifferent generations and select the\nbest one is because obviously like it's\nnot grounded. It doesn't understand the\ntree very deeply. We can overcome that\nby because we understand the tree\ndeeply. So we can put a prompt in there\nthat constrains their generation and now\nwe can make them act as if they\nunderstood the tree when they didn't.\nBut you know we just need to build the\nmodels that do understand the tree\ndeeply and then we can just trust them\nto generate autonomously.\nYes, that is that is a very good way of\nof uh of saying what what I think and\nthat is what I'm focused on. I that's a\nreally good way of putting it right\nwhich is um forcing the language models\nto develop these deep trees from the\nground up. it you can only develop it\nfrom the ground up I think and so you\nneed to be a we need to come up with new\ntechniques new environments to uh to\ngrow the trees instead of pre-training\nwhich is pre-filling random you know\nit's not random but uh it's a web it's\nnot a it's not a tree\nyes there's also the vexed issue of what\nhappens at the bottom of the tree so um\nchole argues that this spelky core\nknowledge you know these are knowledge\nprimitives that are so fundamental that\ninside the deductive closure\nof those primitives we can talk about\nanything and you know you can still get\nlost in different parts of the tree I\nsuppose and there's some issues of\nintelligibility going between them but\nif you understood the tree deeply enough\nyou can go anywhere but maybe there are\ndifferent trees maybe there you know\nlike in physics there are different\nlevels of you know levels of description\nof understanding the universe so do do\nyou think it's like one big tree or lots\nof trees\nI think um the the you have it's kind of\nuh gated in the laws of whatever you're\ndoing and I know you know I I looked\ninto this a few times. I know there are\nsome axioms that you need to take for\ngranted and let's say deducing some\nforms of math, right? And they would be\ntheir own tree because you can't get to\none part without the other. You know, if\nyou take this for granted then you can't\nuh you can't deduce this but you can\ndeduce this. And so I'd say basically\nthey have to you know they have to be\nlogical chain and so of course there\nmight be there might be multiple chains\nand maybe everything is grounded in\nlogic right like logic is the\nfundamental I guess that is true\nactually logic must be the fundamental\nblock of a tree right like the\neverything comes from logic because if\nwe didn't have logic we couldn't have\ntrees so I guess that yeah I guess it\nwould be all one tree and it's a logic\ntree and uh I guess there are people\nthat subscribe to logic trees and then\nthere are people that are illogical and\nthey they don't have their trees at\nJeremy, it's been an absolute honor to\nhave you on the show. Just before we go,\nare you hiring or anything you want to\nsay to the audience?\nFor sure. Yeah. So, at Reflection, we're\nbuilding uh open\nuh intelligence models. So, uh we're\nhiring across the stack, pre-training,\npost-training, uh large language models.\nWe have a lot of GPU. So, uh if you're\ninterested in pre-training, post-\ntraining, we're in SF, we're in New\nYork, and we're in London. So definitely\nuh apply on our site or you can just hit\nme up on Twitter.\nAmazing. Um Jeremy, I've really enjoyed\nthis. Thank you so much.\nAwesome. Thank you.\nJust keep doing what you're doing, man.\nI I um I just I really really think that\nyou that that you you're on to something\nhere. I mean, obviously like with the\nminor, you know, um discussion about,\nyou know, where the how we're actually\ngoing to do this. But I think the the\ndirection is is quite clearly set. I\nknow it's such a vexed issue, though. I\nmean um I'm I'm interviewing a bunch of\ncognitive scientists in Japan next week\nand like you can really go down the\nrabbit hole on this. So for example, I'm\na big fan of you know um ex like um\nexternalism you know like inactive\ncognition and there are also all these\nphilosophical views where um you know\nlike consciousness is basically a\nproperty of certain types of physical\nmaterial. So because you know what we're\ntalking about here with understanding\nlike if you actually abstract it into\nphysics we're talking about certain\ntypes of causal graph and you can argue\nthat certain types of cognition actually\nrequire certain types of physical\ninstantiation you know where in the\nmiddle of that graph you have material\nwhich is capable of producing\nconsciousness. So some components of\nunderstanding are phenomenal. they're\nconscious,\nright? And like when when you start\ntaking it to this philosophical level,\nthere's almost like no there's no end to\nit, you know, because you'll always have\npeople that argue against functionalism\nthat that say that cognition must be\nphysically instantiated in a certain\nway. And I'm not sure where I am on that\nbecause even the cognitive scientists\nare saying, \"Okay, well guys, we have to\nadmit that, you know, even though we can\nmake all of these arguments, I mean,\n[ __ ] me. These LLMs are they're doing so\nwell.\" But um yeah, I think it' be it'd\nbe a fascinating discussion because a\nlot of neuroscientists are internalists.\nThey basically like Jeff Hawkins, they\nthink that all of the [ __ ] happens in\nthe brain and you know, we have these\nsensory motor circuits and we've got\nthis master algorithm in our neoortex\nand that does all of the things. And I\nthink there's there's something to be\nsaid for that, but you know, I think it\ndoes ignore quite a lot of the other\nfield. Yeah. Well, it all comes down to\nif we can build machines that actually\nhave a deep grounded understanding of\nthe world. And let's assume that that\ndoesn't actually have to be physically\ngrounded, right? If if the\nrepresentation is like, you know,\ngrounded in the sense that it's a\nfaithful description of what is\nhappening out there and we can do this\ncreative reasoning on that\nunderstanding, then what's to stop us\nfrom inventing new things? I mean,\nthat's basically the thesis, right?\nYeah. Beautiful stuff, Jeremy. Thank you\nso much, man. I really really appreciate",
  "transcript_chars": 74425,
  "ingested_at": "2026-05-12T00:42:26.994647+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 17622,
    "like_count": 502,
    "channel_id": "UCMLtBahI5DMrt0NPvDSoIRQ",
    "categories": [
      "Science & Technology"
    ],
    "tags": []
  }
}