{
  "video_id": "4APMGvicmxY",
  "channel_slug": "machinelearningstreettalk",
  "channel_handle": "machinelearningstreettalk",
  "title": "Tensor Logic \"Unifies\" AI Paradigms [Pedro Domingos]",
  "duration_seconds": 5269.0,
  "url": "https://www.youtube.com/watch?v=4APMGvicmxY",
  "upload_date": "",
  "transcript": "TensorLogic unifies not just symbolic AI\nand and and deep learning, it also\nunifies things like kernel machines and\ngraphical models.\nI'm Pedro Domingos. I'm a professor of\ncomputer science at the University of\nWashington in Seattle and a long-time\nmachine learning researcher. And my\ndream from my PhD onwards has always\nbeen to unify all the different\nparadigms of AI into a single one. My\nPhD unified two of them. My best known\nresearch unifies a couple others of\nthem. I wrote this book that turned into\na big bestseller surprisingly called The\nMaster Algorithm that is precisely about\nthis goal and where we are towards that\ngoal. And my latest work which this this\nthis podcast will talk about is a new\nlanguage called TensorLogic that I would\nsay for the first time brings this dream\nof a unified representation, a unified\nsolution to AI [music] within reach. So,\nif you want to find out, you know, how\nhow we're going to do that, watch this\npodcast. You know, I can set the\ntemperature to GPT to zero and it still\nhallucinates. And I can I can have a I\ncan have a poor deductive system that\nhallucinates all kinds of things. So, to\nme like those are separate separate\nproblems.\nNo, very good. So, precisely the problem\nor one of the problems with GPT is that\nit it hallucinates even when you set the\ntemperature to zero. What the hell,\nright? I want to have a mode Yeah, yeah,\nyeah. Right? Not I, but like every\nFortune 500 company, if it's going to\nuse AI, needs to have a mode where the\nthe logic of the business is just obey.\nThe security isn't violated, the the the\ncustomer doesn't get lied to, etc. We\ngot to have that or you have it that\nthey will not take off, right? And\nTransformers can do that. TensorLogic\ncan do that precisely because\nin this in you know, reasoning in a\nmating space, you know, mode that I just\ndescribed, if you set the temperature to\nzero, it does purely deductive\nreasoning. And by the way, the\ntemperature can be different for each\nrule.\nTensorLogic is just based on this to me\ngob-smacking observation that an\nEinstein under rule in in a logic\nprogramming are the same thing.\nThere is this thing called predicate\ninvention, which is discovering new\npredicates, discovering new relations\nthat are not in the data, but they\nexplain it better.\nI would say that, you know, in some\nsense, discovering representation like\nthat is the key problem in AI, is the\nholy grail. What was Turing's\nachievement that we now take for\ngranted? Turing's achievement was for\nwhich he is deservedly famous, right? Is\nto postulate this notion of a universal\nmachine.\nThe amazing thing about computers is\nthat they're universal machine, which in\nhis time was an was a completely\ncounterintuitive notion. What do you\nmean a machine that can do everything?\nThe typewriter can type, you know, you\nknow, like the sewing machine can sew.\nYou're telling me there's a machine that\ncan type with one hand and sew with the\nother. What are you talking about? So,\nlike this is the genius, right? So,\nfirst step, you want to have this\nproperty of having a machine that can do\nanything. What we're missing to be able\nto do what the universe does and\nevolution does is universal induction.\nWhat is the Turing machine equivalent\nfor induction, for learning? That's what\nI'm after.\nMLST is supported by CyberFund. Link in\nthe description.\nHey folks, I'm Amar, writing design lead\nat Google DeepMind. We just launched a\nrevamped vibe coding experience in AI\nStudio that lets you mix and match AI\ncapabilities to turn your ideas into\nreality faster than ever. Just describe\nyour app and Gemini will automatically\nwire up the right models and APIs for\nyou. And if you need a spark, hit I'm\nfeeling lucky and we'll help you get\nstarted. Head to ai.studio/build\nto create your first app. The idea of\nhaving to traffic [music] in squishy\npeople in order to make our systems go\nis not immediately [music] appealing,\nlet's put it that way. This episode is\nsponsored by Prolific. Let's get few\nquality examples in. Let's get the right\nhumans in to get the right quality of\nhuman feedback in. [music] So, so what\nwe're trying to make human data or human\nfeedback, we treat it as an\ninfrastructure problem. We try to make\nit accessible. We make it cheaper. We\neffectively democratize access to this\ndata. I'm a long-time fan of of Machine\nLearning Street Talk. In fact, I was a\nfan of it before it was [music] big.\nJust like I was doing deep learning\nbefore it was big. So, very close\nanalogy. So, you should definitely watch\nMachine Learning Street Talk. It's one\nof the best ways to not only learn about\nmachine learning, but find out about\nwhat's going on at a deeper level than\nyou see everywhere and that is very\nimportant. So, you should definitely\nsubscribe to Machine Learning Street\nTalk.\nProfessor Pedro Domingos, it's amazing\nto have you back on MLST. I've lost\ncount of how many times we've had had\nyou on the show now. So, it's it's\namazing to have you back. The main\nreason that we've invited you today is\nyou've just released a brand new paper,\na very exciting paper called\nTensorLogic, the language of AI.\nAnd fields you said take off when they\nfind their language. So, you gave the\nexample of calculus in physics and\nyou know, Boolean logic when designing\ncircuits. What's the idea behind this\npaper? Well, TensorLogic in many ways is\nthe goal that I've been working towards\nmy entire professional life because I\nreally do strongly believe that a field\ncannot take off until it has really\nfound its language. And and TensorLogic,\nI believe, is the first language that\nreally has all the key properties that\nyou need in the language of AI. For\nexample, it has automated reasoning\nright out of the box like for example\nProlog has, right? The classical\nlanguages had a number of things that we\njust took for granted. The transparent\nand reliable reasoning, you didn't even\nhave to worry about. It was just already\navailable, right? At the same that you\ndon't have that in in PyTorch at all,\nright? You have all these hacks to try\nand do reasoning on top of it. At the\nsame time, the Lisps and the Prologs,\nthey never had the auto differentiation.\nThe ability to learn, right? One of the\nbeauties of the current moment in many\nways is that you barely have you look at\nmost papers, people barely talk about\nthe learning because it's already\nimplemented under the hood. So, you want\nthat as well, right? And you want the\nscalability on GPUs is the other thing\nthat things like like, you know, PyTorch\nand TensorFlow and and whatnot give you.\nThere was no language before that have\nall of these and and there's a number of\nothers, but these maybe are are some of\nthe key ones. So, TensorLogic is is\nbasically a language which as the name\nimplies is a marriage, a very deep\nunification, not just some superficial\ncombination of the tensor algebra that\ndeep networks are all built out of and\nthe logic programming that symbolic AI\nis built out of. There's only one\nconstruct in TensorLogic and it's the\ntensor equation. You can do everything\nwith with tensor equations. Are you\nsaying that there's only one language of\nAI? Because certainly in in some fields\nlike physics, you gave the example of\ncalculus.\nI mean, yeah, like you know, almost all\nof calculus I mean, almost all of\nphysics, you know, involves quite a bit\nof calculus.\nThere are other fields where actually\nthere are kind of multiple\nmultiple languages\nthat that play, you know, almost equal\nroles. So, I'm wondering if you think\nthat\nis TensorLogic\ngoing to be 85, 90 plus percent of the\nway that we should be talking about and\nthinking about AI or will be a will be\nkind of a mixture of different different\nlanguages?\nThat is a very good question. And in\nfact, we know very well in computer\nscience that there's no one programming\nlanguage that is better for everything.\nThere's just people who think it is.\nRight? Everybody has their favorite\nlanguage that they believe, you know, is\nthe universal solvent, but it never\nreally is. So, we and we know also for\nfundamental reasons like, you know,\ngoing back to, you know, Shannon and\nwhatnot that that there is no language\nthat is the most, you know, pithy for\nanything you might want to say, right?\nHaving said that, right? Physics is a\ngood example because calculus is so\nfundamental, you know? Feynman famously\nsaid that he thought in calculus. Right?\nAnd the thing that I found with\nTensorLogic is that, you know, I don't\nknow how much of AI it's going to be\nor how much it should be. But what I\nhave found in many ways to my surprise\nis that in some ways TensorLogic is more\nthan just a programming language. It\nreally, I think, captures the\nfundamentals of what you need in AI in a\nway that going in I didn't even think\nwas possible. All of tensor algebra can\nbe reduced to this operation which, you\nknow, going back to physics is called\nthe the Einstein summation.\nRight? Einstein summation was something\nthat was introduced by Einstein when he\nwas working on relativity and and got\ntired of writing summation signs. It was\nall about tensors, right? General\nrelativity is all about tensors. And he\njokingly called it his great\ncontribution to mathematics. But the\nbottom line and you know, there's this\ngreat paper by Tim Rocktaschel or or,\nyou know, blog post saying Einstein is\nall you need. And truly, you can do all\nof deep learning with just einsum. All\nof the matrix multiplications and and\nand, you know, tensor products, all of\nthat are instances of einsum. On the one\nhand, on the other hand, in symbolic AI,\nit's all about rules, right? And and and\nTensorLogic is just based on this to me\ngob-smacking observation that an einsum\nunder rule in in a logic programming are\nthe same thing.\nThey are actually the same thing. The\nonly difference is that one is operating\non real numbers and the other one is\noperating on Booleans, but you know,\nthat's just a different, you know,\natomic data type.\nRight? And then on top of that so so so\nto summarize, at this point I think that\nit would be I look at all different\nthings that I and others have done in AI\nand I think it would be crazy to not do\nthese things with TensorLogic. There may\nbe other better things coming, you know,\nafter, but at this point I would say\nTensorLogic probably better for what\npeople are doing across the board. But\nhey, that's me. I may be a little\nbiased. First of all, shout out to Tim\nRocktäschel. I I read that blog post\nfrom 2018 earlier that that you were\nreferring to. But, I suppose the thought\noccurs that if it is mostly about\nEinstein and, you know,\nyou might make the argument, why do we\nneed an abstraction when we already have\na great abstraction in Einstein? So,\nfolks now can use uh PyTorch and, you\nknow, uh JAX. W- What What exactly does\nyour abstraction allow them to do that\nthey can't do with PyTorch?\nNo, very good. It does several things.\nSo, first of all, and I this is going to\nbe in increasing order of importance,\nthe syntax of Einstein in these\nlanguages, there's also this package\ncalled I nots, is incredibly clunky.\nSo, at a very basic level, TensorLogic\nis just a much pithier, more compact,\neasier to write and understand way to\nwrite Einsteins.\nAnd, you know, physics and\nmathematicians are famous to like to say\nthat a good notation is half the battle.\nSo, this might not seem like a big deal,\nbut my experience is that you can just\nthink better and faster once you have\nthis notation that like this funky\nprocedure call with these indices and\nthese arrows and these arguments. It's a\nnightmare, and, you know, it the syntax\nof TensorLogic especially like you write\na a Einstein like you would write a\nrule. There's a tensor equation with a\ntensor on the left-hand side, and and\nand and this join of tensors on on on on\non on the on the right on the right-hand\nside, right? So, this is one aspect.\nAnother very important aspect uh uh and\none that I think could prove decisive is\nthat people don't use Einstein much\nbecause it's not very efficient. Under\nthe hood, it's not as efficient as you\ntip you know, sometime I you know, this\ncould be done so much better, right?\nBut, I've done some of programming of\nthis, and I wound up even I wound up not\nusing Einstein because it's so slow and\nclunky. And all of that can be fixed\nonce you have this one abstraction of\nthe tensor equation, and you implement\nit on CUDA, for example, you can\noptimize the heck out of it, and you'll\njust be able to, you know, Einstein will\nfinally be able to reach its potential,\nright? But, actually none of these\nthings that should be the most important\npart. The most important part is that\nthe the Einstein as we know it is only\ngood for for for tensor algebra.\nTensorLogic is a language where the same\nconstruct does all the symbolic and all\nthe the numeric part and and any mix and\nvariation between them, including\nlearning the symbolic part and whatnot.\nThese are all things that in Einstein\nworld just didn't exist, right? You talk\nabout the people who knew Einstein,\nwhether in AI or mathematics or physics,\nand they just had no idea that any of\nthis had anything to do with reasoning.\nYou look at all the ways that people are\ntrying to do reasoning today and just\nwant to pull out your hair. Let me ask a\nvery concrete, you know, in some sense\nI'm a simple simple man. I need like a\nvery concrete example, cuz I completely\nagree with you, which is that the\nsymbols we use, the language we use are\njust\nsimplicity is so fundamental to our\nability to like reason at higher and\nhigher levels. So, let's take one\nexample, you know, from your paper,\nwhich is um\num a logical or logical or of a bunch of\nvalues is equivalent to\nan Einstein within a Heaviside, you\nknow, function applied to it. Like you\ngive this example, right? The what just\nto So, to be precise, what I did in in\nyou know, maybe this is an important\npiece of context. So, uh if you look at\nSo, the simplest form of logic\nprogramming is Datalog, right? Which is\nthe foundation of databases, right? You\nknow, most SQL queries are variations on\non Datalog rules, and Datalog rules are\ncomposed of two things, joins and\nprojections.\nAll right, this is like databases 101.\nAnd what I have done is I have\ngeneralized join and projection to\nnumeric values.\nThere's this thing which I define called\nthe tensor join and the tensor\nprojection, which when the tensors are\nBoolean becomes the regular symbolic\ndatabase one, but now the the the\nnumeric version has all these things as\nspecial cases. And by the way, it's also\nmore general than the Einstein, right?\nSo, another benefit of this is that it\nactually goes goes beyond the Einstein.\nNow, an or, right? The way you get an or\nis is is by having more is is just as in\nSo, how do you get an or in Prolog or\nDatalog is by having, you know, multiple\nrules with the same head.\nAnd then those rules the those then then\nthey implicitly being disjoint. All\nright? So, if I have A if B C and A if D\nE, then that means A if B C or D E. And\nthe same thing happens here. And you\ncould also, of course, just put them all\nin the same equation, cuz like, you\nknow, it might be more convenient just\nto say like, well, AB plus CD, right?\nSo, so doing an or is a completely, you\nknow, straightforward thing, but it's\nreally not where the main action is.\nIt's in the tensor joins and tensor\nprojections. Thank you like for laying\nall that out. Completely agreed, makes\nsense. I I just wanted the very example\nI was giving is that an Einstein over a\nparticular index of Boolean values\nthen with a Heaviside function applied\nto it, which is just that, you know,\nzero if it's less than zero or one if\nit's greater than zero is equivalent to\na logical or of the over that same\nindex. Oh, yeah, sorry. I I I see I\nunderstand your question.\nSo, so again,\nit's more than that. It's It's It's a\nDNF.\nRight? It's not\nSo, a DNF is a disjunction of\nconjunctions.\nYeah, exactly. And so, what happens is\nthat like, you know, like the Einstein\nis So, in numeric land, right? The dot\nThink of a dot product, right? The dot\nproduct, right? Is just the sum of\nproducts, right? Which in Boolean land\nwill be a disjunction of conjunctions.\nIf more than one is true, you get a\nnumber that's greater than one, which is\nyou need to pass it through a step\nfunction to reduce all the values\ngreater than one back to one. Yes. But,\nmy question was more like, okay, I have\nthese two different representations of\nthe same the same operation at least at\nthe element level. Like an an Einstein\nover an index followed by a Heaviside on\nthat element is equivalent to an or over\nall the same Boolean values of that of\nthat index. And I guess and my question\nor per element. So, my question to you\nis\nI I give up one thing, which is instead\nof having a single symbol, which is kind\nof like an or, I've now got two\noperations, you know, Einstein\nHeaviside.\nAnd there are many examples of that,\nright? Like I can build every single\ncircuit out of NAND gates. I think we\ndiscussed this like once actually. Or or\nI can have like other kinds of gates,\nand and it's useful to have other kinds\nof gates. So, in your in your language,\ndo you foresee people not having\nsyntactic sugar like an or operator,\nwhich under the hood is Einstein\nHeaviside, or would they still retain\nthose? It's just that the fundamental,\nyou know, the most basic um you know, uh\nconstructs of the language are\nTensorLogic. We can do everything with\nNANDs, so why do we need high-level\nprogramming languages at all, right?\nThe The point of So, there's two things\nthat you want the language to be. First\nof all, you want it to be universal, so\nyou can\nFor some things you don't, but in\ngeneral, right? For AI, surely you want\na universal language. You want something\nTuring-complete and Turing-large a- a-\na- a- and TensorLogic is that. But, then\nthis is actually the most important and\nmost difficult part. You want something\nthat is at the right level of\nabstraction for the things that you want\nto do. And NAND definitely is not. And I\ncan show with a lot of examples that I\nhave in the paper, for example, you can,\nyou know, code a transformer in a dozen\ntensor equations\nas opposed to a vast mass of code,\nright? And then what happens when people\nhave a language that suits their needs\nis that then they just get used to that.\nThey often wind up using it for even for\nthings that it wasn't the perfect thing\nfor,\nbut at that point it's what they're\ncomfortable with. So, my guess is that\nat the end of the day people are just\ngoing to do everything in TensorLogic,\nand and and they know, you know, in the\nback of their heads that yes, there are\nors going on here, and you can think of\nthem as ors, but\nthey just think of them as joins and\nprojects and tensor equations. Very\ngood. And And by the way, you can\nimplement transformers and anything else\nin TensorLogic. It's so easy, in fact,\nthat I fed your paper into Claude Code,\nand I got it to implement the whole lot\nuh this afternoon, and maybe I'll\npublish that on GitHub if folks want to\nhave a look, but um it it it's quite\nstraightforward. But, just to get the\ntrajectory a little bit here, Pedro, uh\nyou're famous for writing this master\nalgorithm book, and in that book you\nspoke about all of these different\ntribes of machine learning, you know,\nlike Bayesian folks and logic folks and\nkernel methods and neural networks and\nand all of this. And I guess do you see\nthis as as a step towards unifying these\nthings together? Because now in\nTensorLogic, you can actually create a\ncomposition of different modalities of\nAI, and and it just works, but this\nmight seem a bit weird to people. I\nmean, can you explain what that might\nactually look like?\nAbsolutely. So, uh in a way, the master\nalgorithm was laying out my agenda,\nright? Was asking the question, what is\nthe master algorithm? I did say at the\noutset, I'm not going to give you the\nmaster algorithm in this book. I'm just\ngoing to tell you where we are and why I\nthink this is the central goal of AI. I\nwould say that TensorLogic is that\nanswer.\nTensorLogic, we haven't talked about\nthat yet, but in TensorLogic unifies not\njust symbolic AI and and and deep\nlearning, it also unifies things like\nkernel machines and graphical models.\nThe you know, the the the thing the\nthings that graphical models, for\nexample, are built out of, and then you\ncan compute probabilities with them, are\nthey are a direct I didn't do this on\npurpose, but it just fell out. Uh you\nknow, the the fact the the the factors\nthat graphical models are made of, those\nare just tensors.\nAnd then the marginalization and and and\nsummation and and and Sorry, the\nmarginalization and and the the\npointwise products that are what\nprobabilistic inference is made of, they\nare just tensor joins and projections on\nthose tensors that represent\npotentials and in the case of Bayesian\nnetworks, conditional distributions. So,\nat this point, we do have this very\nsimple language where you can do the\nentire gamut of AI, which honestly, I\ndidn't think this was going to be\npossible going in. I thought the answer\nwould be much more complicated. Now, is\nthis the master algorithm?\nTensor Logic per se is not the master\nalgorithm because it's just a language.\nI would say that it's the scaffolding\non top of which you can build a master\nalgorithm. Now, Tensor Logic is not just\na language, it's also the learning and\nreasoning facilities under the hood. So,\nfor example, one of the best things\nabout Tensor Logic is that the auto\ngraph is incredibly simple. Because\nthere's just one construct, is the\ntensor equation, and and and the the\ngradient of a Tensor Logic program is\njust another Tensor Logic program.\nSo, this is all there. So, the learning\nand the reasoning are all there.\nHowever, you know, what I would say is\nthat this is not the the the the master\nalgorithm per se, but it's what we need\nto producing and I intend to produce it\non short order. You've described a\nlanguage and certainly if components of\nthat language are Turing complete,\nthat's a big vexed issue. We'll come\nback to that a little bit later. But, um\nbecause of computational equivalence, we\ncan, you know, from an expressibility\npoint of view, we can describe anything\nin the universe. So, we've got this\nframework, but to me, the challenge in\nAI is structure learning. Right? So, as\nwell as being able to express stuff,\nit's being able to adapt to novelty and\ncreate perhaps from building blocks that\nwe already have a new structure to allow\nus to do something useful in that\ndomain. And I can't quite make that leap\nwith with your technology yet. So, how\nhow do we do the meta thing where we\nactually build the the Tensor Logic\nconstructions to represent the kind of\nworld that we're seeing?\nOh, very good. So, I actually go into\nthat in the paper, you know, but\nbriefly. The paper is just, you know, an\ninformal introduction to these ideas.\nInductive Logic Programming, right, is\nthe field that deals with discovering\nrules from data.\nBut, it does this by things like greedy\nsearch or beam search and it's a very\nlarge, you know, search space and it's\nextremely inefficient, right?\nWhich is actually one of the things that\nkilled it even though it could do all\nthese things that people know in deep\nlearning are just painfully\nrediscovering. In Tensor Logic, this is\none of the best parts of it, the\nstructure learning falls out of the\ngradient descent. The gradient descent\nactually does structure learning. And\nthen on top of that, this is actually\nthe the the best part as as far as the\nlearning is concerned, there is this\nthing called predicate invention, which\nis discovering new predicates,\ndiscovering new relations that are not\nin the data, but that explain it better.\nI would say that, you know, in some\nsense discovering representation like\nthat is the key problem in AI, is the\nHoly Grail. Everything that we know, you\nknow, like when you look at it at the\nworld, right, you don't see pixels. You\ndon't see photons hitting your retina,\nright? You see objects. The objects are\ninvented predicates. All the way up to\nin science, right, the most like\nNewton's genius was to introduce a new a\nnew quantity, which is force, and\nenergy, and entropy, and all these etc.\netc., right? So.\nIn in Tensor Logic, that also just\nhappens by gradient descent, right? It's\nit's you know, it's hard to believe, but\nwhy Let me just give you a hint as as to\nwhy this is the case. There's this other\nthing that is folded into Tensor Logic,\nwhich is tensor decompositions.\nRight? And tensor decompositions are a\ngeneralization of matrix decompositions.\nAnd and if and if you think about matrix\ndecompositions, to take that simple\ncase, what a matrix decomposition does\nis it takes a matrix and decomposes into\ntwo new matrices that together are more\ncompact, but essentially reproduce the\nsame data, right? And there's a\ngeneralization of that to tensors called\nthe Tucker decomposition. There's\nothers, but the Tucker one is is the\nmost relevant one here. And so, if you\nwrite in Tensor To answer your questions\nvery directly, if you write in Tensor\nLogic a rule schema\nyou know, including, you know, a a data\ntensor on the left-hand side. And by the\nway, your entire data can just be\nreduced to one tensor embedded as one\ntensor. We can touch on that later. But,\nyou write a rule expressing that as a\nfunction of a few other tensors, and the\ngradient descent, just as in matrix\nfactorization, will discover the best\nvalues for those. And then if you want\nto, for example, then discretize it, say\nlike I'm going to threshold this and\nmake it Boolean again, you will see what\nis the concept that that learned, or you\ncan leave it in in in in numeric form.\nSo, the learning is actually\nextraordinarily powerful. I've always\nthought, and you know, I think, you\nknow, a lot of people in deep learning\nreally believe this, that\nyou know, gradient descent can do\namazing things provided you give it the\nright architecture to operate on.\nAnd in a way, what all these million\npapers are about is about finding the\nright architecture for gradient descent\nto operate on. And of course,\ntransformers are a great leap forward,\nbut I think transform I think, you know,\nTensor Logic is an even greater leap\nforward. How so? Because, for example, I\ncan picture Suppose we we we want to get\nrid of Python. So, like I'm over here in\nyou know, PyTorch and and I've described\nall my my layers kind of in the clunky\nsyntax and and said I'm like, \"No, now I\nhave, you know, the Tensor Logic, you\nknow, GitHub programming language. Let\nme go do there.\"\nI'm still going to construct my layers,\nright? So, because like, for example,\nyou do, of course, you allow for the the\nyou know,\nnon-linearities, right? So, after every\nein sum, I can apply whatever kind of\nnon-linear function I want, a ReLU or\nsigmoid or whatever else, right? That's\nstill going to be described in my\nprogram. It's like I'm going to have\nthis shape\nein sum followed by this non-linearity\nfeeds into this shape followed by So,\nI'm still going to have to do that kind\nof like,\nyou know, structuring of the network, if\nyou will, except now in Tensor Logic.\nAnd in my opinion,\nthat's one of the biggest limitations\nright now is these are all just divine\nincantation structures that people have\ncome up with. Like, let's put in a\ndropout layer here and this kind of\nlayer there and there. We don't actually\nallow the machines to learn the overall\ntopological structure. We only allow\nthem to to find weights within that\nstructure. No, but okay. I understand\nyour question, but Tensor Logic does\nallow that. You know, step one, um\nyou can you can encode a multi-layer\nperceptron, the entire multi-layer\nperceptron, and I do that in the paper\nwith a single tensor equation.\nAll the layers, provided that they all\nuse the same non-linearity, can be\nencoded in one equation.\nOkay? Number one. You can also have\ndifferent equations for different layers\nor typically sets of layers, you know,\nto your however way you please. But,\nfrom the point of view of structure\ndiscovery, the thing to realize is that\nif you create if you set up one of these\nvery general equations that you can in\nTensor Logic, that in some sense it can\nit's a very broad classes of\narchitecture, then what the learning\ndoes is it discovers the architecture\nwithin that space.\nRight?\nWhich if you think about it at some\nlevel is what a neural network, when you\ncompare an ordinary multi-layer\nperceptron with a set of rules, right?\nA multi-layer perceptron is is you can\ntake In fact, there was a system called\nKBAND in the early days that did this\nvery clever. It initialized\na multi-layer perceptron with a set of\nrules because each neuron is a rule,\nright? But, it's also more flexible\nbecause now you can have weights, right?\nBut but but, you know, a neuron, a\nsingle neuron can represent a\nconjunction and therefore a layer can\nrepresent a disjunction and so forth.\nSo, when you're learning weights in an\nordinary neural network, you can\nactually see it as learning the\nstructure of a set of rules. What Tensor\nLogic is doing this is at a more\npowerful level. Like like that was just\npropositional, and now this this is at\nthe full level of of of generality of of\nfirst-order logic. But, you can learn\nthe structure and then of course, then\nthere's more than one way to do that.\nAnd you can also decide how black and\nwhite you want the structure to be, what\nyou want to leave as weights, what you\nwant to discretize. But, the the\nstructure itself can be learned by\ntaking a tensor equation. A tensor\nequation is a very general thing, right?\nWhen when you learn the weights of those\ntensors, that the that materializes to a\nspecific network structure. Yeah, so I I\nI understand that. Let me bring this\nback to like to to the folks who are\nfamiliar with,\nyou know, PyTorch or traditional\ntechniques. What you described is\nyeah, I can just create a fully\nconnected, you know, network with\nhowever many layers I want and then let\nSGD, you know, find all the weights.\nThat doesn't work. Like, it doesn't work\nin practice and it's not going to work\nwith Tensor Logic. You know, it's just a\ndifferent representation of the same\nfundamental problem, which is there's\ntoo many degrees of freedom. It's not\ngoing to learn anything useful. This is\nwhy so much alchemy goes into\nstructuring, you know, constrained\nnetworks to have certain, you know,\nbuilt-in, you know, inductive bias,\nright?\nNo, absolutely. So, to take another\nexample, you can also do an entire\nConvNet in just one tensor equation. And\nand, you know, like the quintessential\nexample of like, yes, complete\nconnections don't work is a multi-layer\nperceptron for for vision, right? Which\nyou replace with a ConvNet that actually\nhas the local structure. That is also a\nTensor Logic equation. Now, you're\nsaying, \"Well, how do you choose between\nthe ConvNet and then an MLP?\" Right?\nVery good question. And now, there's a\nrange of things you can do. You can\nactually these days start out with a\nvery general structure because I mean,\nGPUs and and and large, you know, server\nfarms are an amazing amount of power for\nsomething like this, right? So, you can\nalmost, I would say, brute force that\nsearch provided you have the data. I'm\nnot actually recommending you do that,\nright? You can also, however, and more\ninterestingly, you can And this is\nactually one of the key benefits of\nTensor Logic is that you can write down\nwhat you believe are properties of the\nsay\nRight now, what happens in in in in, you\nknow, when you, for example, program a\nnetwork in Python is like, you have to\ncommit. You say like, \"Here's the\nstructure.\" And now, the you know, the\nthe learning of The only thing that\nhappens is the learning of the weights.\nIn Tensor Logic, you don't have to do\nthat. You can you You just you can set\nup one of these very general structures\nand then you say, \"Let me give you a\nbunch of equations that are things that\nI believe to be true about the structure\nbut do not completely determine it.\"\nAnd those just work like priors and\nindeed like soft priors, right? Those\ncan and then again you can turn up the\ntemperature on this or down and say\nlike, \"You got to obey this equation and\nthat one, you know, sure you can\noverride.\" And then and then this in my\nexperience this is actually what is\nimportant is that the gradient descent\ninstead of starting from a tabular rasa\nhave this kind of soft knowledge. And\nthen most importantly like you the\ndeveloper, you the AI researcher, you\nyou get to\nthis is really the essence is that it's\nnot this you\nevery every, you know, deep learning\nresearcher or data scientist know this\nis like you don't do this a priori and\nthen push the button and hope for the\nbest, right? It's like it's there's an\niterative loop\nof you set up the structure and then you\nlearn you get the results and then you\nrefine the structure. And what this does\nis it makes that more efficient much\nmore because you just have to really in\nyour interpreter you write one more\nequation or you modify an existing\nequation and also the entire stack\nis is\nof what you learn is much more\ninterpretable than it was before.\nThat's actually in some ways one of the\nmost important properties of tensor\nlogic is that you can understand what's\ngoing on much better than you could in\ntwo ways. One is that the code is much\nmore transparent than the whole pile of\nthings that you have sitting under a\nbunch of, you know, PyTorch procedure\ncalls but also the result of learning at\nleast if you do it in in certain ways\nthat I discussed in the paper, the\nresult of learning is transparent in a\nway that a transformer just, you know,\ncan't hope to be. So we've covered some\ninteresting topics on MLST before and of\ncourse there's geometric deep learning\nwhich is this idea that symmetries are\nfundamental.\nUm we've spoken with Andrew Wilson from\nNYU recently about soft inductive priors\nand I've just spoken with Yi Ma about\nhis crate series of architectures. And I\nguess the the prevalent idea here is\nalmost Platonistic that there are real\nnatural patterns and if we kind of bias\nthe model as you're just alluding to\nthat it will converge on really good\nrepresentations that describe reality.\nNow the the alternative view is that\nreality is is constructive and gnarly\nand and that won't work but you were\ntalking about your Tucker decomposition\nearlier and that's this idea that, you\nknow, we might have a large sparse\nmatrix, we might want to densify it, we\nmight want to factorize it and the\nfactorization will kind of pull out some\nof these natural orderings, you know, of\nthe universe perhaps. And I guess I was\nthinking isn't it a bit like a Gzip\nalgorithm? I mean, what if these\nfactorizations are just semantically\nmeaningless, you know, how do you know\nthat you've got a good one? You know,\ngreat question and you've touched on\nseveral things there. Let me start with,\nyou know, the geometric deep learning,\nright? I'm a big fan of this. In fact,\nyou know, I I gave a keynote at the\nsecond ICLR on something that I called\nsymmetry-based learning\nwhich is in some ways an ancestor of\ngeometric deep learning. I really do\nthink that the universe possesses these\nfundamental Actually, I don't think\nthat. This is known, right? In physics,\nright? The standard model is basically a\nbunch of symmetries.\nAnd this is extraordinarily powerful,\nright? That such simple things could be\nsuch universal regularities that you\nthen basically can build everything else\nout of, right? And and if you think\nabout it in machine learning the problem\nis like what is the learning bias that\nyou should start from?\nRight? Should you pull in a lot of\nknowledge? Should you have a very, you\nknow, very vague architecture? The thing\nabout machine and there's the no free\nlunch theorem, right? That says you, you\nknow, if you don't assume anything you\ncan't ever learn anything. The thing\nthat's amazing about machine learning is\nthat with very weak biases you can get\nvery far, right? And I would submit that\nthose weak biases fundamentally at the\nend of the day the most important ones\nare these symmetries. And tensor logic\nis precisely, you know, I think the\nperfect language for expressing those\nsymmetries as the physicists will tell\nyou, right? It's what it's what they use\nin in like in not the logical, you know,\nversion but but the but the numeric\nversion, right? So I I think we can\ndiscover those regularities. I have some\nsuspicions as to what they might be but\nI think, you know, we're not quite there\nyet but I think once we have those\nregularities in some sense,\nyou know, a a um\nthey will play in AI the role that the\nstandard model plays in physics, right?\nNow of course as you say\nyou know, there are people who say like,\n\"Oh, forget that, right? You know, going\nback to Marvin Minsky, right? There's\nlike there is no small set of AI laws or\nanything. It's just one damn thing after\nanother, blah blah blah blah, right?\nLike you're dreaming, right?\" And I\nrespect that point of view, right? And\nand you know, we will find out\nempirically but I what I if I had to\nguess how this is going to play out at\nthe end of the day it's going to be like\nthis. This stuff that I'm talking about\ngives you, you know, the 80/20. You\nknow, it gets you 80% of the way.\nAnd then the other 20% of the way you\nhave to do a lot of these things, you\nhave to do a lot of hacks, etc. etc. But\nsomething like since tensor logic still\nmakes it much easier and faster to do\nthose hacks than than if you didn't have\nit. So you it actually gives you a\nbenefit both in the 80% part and in the\n20% part. There are folks so, you know,\num complexity science there's this guy\ncalled David Krakauer\nand in his book on the first page\nactually, the very first sentence, the\nscientific and social implications of\ndifferences between A closed,\nreversible, symmetry-dominated, and\npredictable classical domains. I think\nthat's what you're talking about the\nkind of the Roger Penrose type world.\nAnd B open, self-organizing,\ndissipative, uncertain, and adaptive\ndomains. Now I think the latter is where\nall the interesting stuff in the\nuniverse is. It's where life and\nintelligence and all the other stuff we\nwant to model. And\ncould it be the case that those things\nare not reducible in the way that you're\narguing they are? I'm glad you asked\nthat question because this really is the\ncrux of the matter. Also, you're\nprobably familiar I know you're familiar\nbecause we've talked about it before\nSteve Wolfram's notion of\ncomputationally reducibility, right?\nYes. And and of course the whole notion\nthat we now understand very well that\nsystems are, you know, many systems are\nchaotic and therefore inherently\nunpredictable, right? And you know,\ncomplex systems and all of that. But so\nwhere does and you know, the whole\nnotion that, you know, more is\ndifferent, right? Like very famous, you\nknow, notion in in in condensed Exactly.\nWhich I'm a very strong believer in. So\ndoesn't that contradict what I just\nsaid? Actually, no, right? I I would say\nthe following is and you know, from\nphysics all the way to AI with biology\nin the middle\nthe universe is basically composed of\ntwo things,\nsymmetries and spontaneous symmetry\nbreakings,\nright? God made the symmetries. The\nsymmetries are the laws.\nAs far as we can tell none of these\nsystems at any level violate the laws,\nright? Those symmetries are there. I\nmean, you can go into that. There's a\nlot to be said there but essentially,\nyou know,\nmost people the great majority of people\nmaybe accepting, you know, some there\nare some exceptions\nbut they believe that the laws of\nphysics apply to everything. Like my\nbrain obeys the laws of physics. Society\nobeys the laws of physics. The problem\nis that the laws of physics are useless\nat some point in understanding, you\nknow, even biology let alone psychology\nor sociology or AI. Why are they\nuseless? Because we have inherited from\nthe beginning of the universe a series\nof spontaneous symmetry breakings,\nright? And my brain is doing spontaneous\nsymmetry breakings one after another\ncontinuously. And those spontaneous like\nthose then some of them die out, right?\nOr become irrelevant, stay the same. But\nothers balloon into very big things. And\nthat's actually what evolution is. It's\none of these things after another. And\nonce you have that so\nso the the computational reducibility\nproblem is that at some level\nit is true that in although in principle\nthis is all predictable and reducible in\npractice it isn't, right? But now here's\nhere's the point like how do we handle\nthat?\nOur brains know how to handle this in a\nway that AI doesn't. And the way they\nhandle is this like you predict you\ncomputationally reduce everything you\ncan to begin with. And I'm actually I've\ntalked with Steve, you know, at some\nlength about this and I'm actually more\nmuch more optimistic about how much is\nreducible than he is. And the thing is\nthat like your overall\nuniverse is not reducible but it's full\nof these irreducible of these reducible\npieces.\nAnd in a way evolution is a cumulate our\nbrain is an accumulation of these\nreducible pieces. So you do that, you\nwant the machine learning to discover\nit, you want the inference to exploit it\nbut then after that you I have to I have\nno choice but to just keep gathering\ndata and using that to inform your\npredictions, right? In a way the physics\ngoal of like I give you the initial\nconditions and then I just predict like\nthe, you know, the the Laplace's demon\ndream. It is a dream\nbut but the I think that the problem\nthat, you know, some of the complex\nsystems people have not realized is that\nwe don't have to do that. Ask any, you\nknow, engineer, any aerospace engineer\nusing a Kalman filter. What you do is\nyou predict just what's you know, or\nreinforcement learning, right? It's like\nyou want to have a sense of where you're\ngoing but at at every step of time\nyou you you you recalibrate your\npredictions with the new data that comes\nin. So you actually only need to predict\nthings well enough to control them to\nmake them predictable, right? We humans\nare always controlling the world to make\nit more predictable and this is what\nrobots need to do as well. And this is\nall like what I'm trying to, you know,\nsupport with a language like like tensor\nlogic. Increasingly more of a believer\nin kind of Hofstadter's, you know,\nconcepts, right? That there are multiple\nlevels of description and even within a\nlevel of description there may be\nmultiple languages, you know, to\ndescribe things at that level. And I\nthink part of the lesson is not only do\nwe observe, like not only do we kind of\nobserve a particular level and sure we\ntry to reduce things and come up with\ntheories at finer grain levels, higher\nresolution theories or whatever. But we\nalso observe a certain layer and we're\nable to by whatever sort of miraculous\nmechanism\nto almost pull out of thin air or\nadduct.\nUm\na theory at this level. Like here's\nthermodynamics. Somehow we came up with\nthat, right? And even if we learn\ntheories at lower levels or higher\nresolution theories, actually most of\nthe time you don't replace those older\nones. It's like within their domain of\noperation, you know, Newtonian mechanics\nis still extremely useful for all lots\nof things that have to do with our\nscale, right? Our scale of activity. GR\nis useful to different scale, quantum\nmechanics at different scale. So we we\nretain all these languages. Um\nand\nI'm hearing that tensor tensor logic is\nis a great language for a certain, you\nknow, layer of description and for for\nactivities of of AI, but you're not\narguing that\nit's the language to sort of replace all\nother layers, right? Like you still buy\ninto the idea that there are other\nlanguages at different I'm glad you\nasked that question. I am absolutely\narguing that tensor logic is the\nlanguage to use in all these layers and\nlet me give you some evidence towards\nthat. Express relativity in tensor\nlogic. It's tensors and, you know,\ndifferentials of tensors and whatnot.\nThat's, you know, that tensor logic does\nthat out of the box. Do the same thing\nwith quantum mechanics. Do the same\nthing with all these others. With all\nthe different pieces of AI that I know.\nAnd why why is that possible and why\ndoes tensor logic do that? Again, I\nthink this gets at a at a very deep fact\nabout the universe which, you know,\ncomplex systems, people and physicists\nhave, you know, suspected as well, which\nis that the universe has this amazing\nproperty without which it would not be\ncomprehensible, that you can have a lot\nof complexity at one level that then\norganizes itself into a new level at\nwhich now a different set of, you know,\nlaws applies, right? And and you know,\nin a way what we do with computers is do\nthat by design, right? But here's the\nkey. What you want is a is a language in\nwhich to express this process, right?\nThe whole process by which multiple\nlevels get created, by which multiple\nrepresentations get created, including\ndifferent representations at the same\nlevel, right? For example, right? The\nyou know, going back to Herb Simon, you\nknow, people many people at least in AI\nhave believed that the essence of in\nhuman intelligence is your ability to\nswitch between representations as the\nproblem dictates. And as long as you\npick one representation, you stuck\nyourself in the box. But but at that\nlevel, tensor logic is a meta\nrepresentation. It's the way to\nconstruct representations. And yeah, you\nknow, a large language model, you know,\nto take a very salient example, what has\nthat transformer learned, right? When it\nlooks at all that text? Precisely, I\nwould say, where a lot of its part come\nfrom is that it has looked, you know,\nit's it's like, you know, say Bill Blake\nsays, like it has learned these super\nalgorithms. There's all these different\npieces and different ways of doing\nthings that it has gathered from\ndifferent places and it doesn't choose\nbetween them. It's the prompting and the\nfine-tuning and all that that then pull\nout the parts that are better for one\nthing or another. So we absolutely have\nto do this in AI. I think it also\nreflects a deeper truth about the\nuniverse. I think there are going to be\nlaws of this. Uh you know, we're not\nthen describing laws of the universe and\nand I think tensor logic at least is my\nbest attempt at having a language in\norder to do both this AI and this type\nof scientific discovery. I also believe\nand I, you know, I I I discussed that\nbriefly in the paper that tensor logic\nis not going to be just a good language\nfor AI. It's going to be a good language\nfor science in general for several\nreasons. One of them is this, but the\nother one is that if you look at the\ndifference between the equations on the\npage and and and the resulting program\nfrom implementing them, often there's a\nlot of complication. In tensor logic,\nit's almost, you know, they quite the\ntensor equation is an almost symbol for\nsymbol translation of the equation on\nthe page. So now you can just do, you\nknow, science, you know, on a different\nlevel. Also, the logic If you look at\nscientific computing curing, right? It's\nusually these tensor operations with\nsome logic wrapped around it. Tensor\nlogic does the tensor operations and the\nlogic in one language, but more\nimportantly, the logic now becomes\nlearnable.\nYou can now learn the logic as well.\nLet me just challenge you on on this\nbecause um for example, like in your\npaper when you got to the RNN section,\nright? Like, you know, tensor logics can\ncan represent RNNs, but then you hacked\nin start T. You're like, oh, I need this\nlittle start T here. What's start T?\nWell, start T is a virtual index that\ndoesn't create new memory. That's not\ntensor logic. You hacked in start T\nbecause you needed that in order to\nexpress RNNs, right?\nNo. No, no, no, no, no, no. Look, great\nquestion. So there's two very important\nthings to distinguish here. One is,\nwhich start T is not, but let me mention\nthat first, the RNNs also illustrate\nthat, is syntactic sugar, right? You\nalways have syntactic sugar because, for\nexample, in an RNN, you want to express\nX of T plus one, right? And I could, you\nknow, tensor logic is Turing complete,\nbut I don't have the T plus one. I can\nthis is a very simple syntactic piece of\nsyntactic sugar to add. Why wouldn't I\ndo that, right? Again, there's an 80/20\nrule of like which of these contracts\nyou want to have. But the start T is\nactually completely different thing. The\nstart T is there for computational\nefficiency purposes.\nStart start T is a hint about how to\nimplement that tensor that saves a ton\nof memory, right? And you know, you know\nthis notion of a leaky abstraction, all\nabstractions are leaky famously in\ncomputer science. Tensor logic is no\nexception. For the most part, when you\nwrite tensor logic, you don't have to\nworry about what goes on under the hood,\nbut sometimes you want to. And this is\nprecisely one of those things. This The\nidea of the start T is that some is like\nI don't we don't have for loops anymore,\nright? Which is great. Forget all of\nthat. But sometimes I don't want to be\ncomputing a new tensor or even just a\nnew vector for every new thing that I do\nbecause that would be a waste of memory.\nThe start T is just saying, you know, re\nyou have one vector and you reuse it at\nevery iteration. So you have the initial\nX zero and then X one is over it\noverwrites that, right? So this this is\na piece of the language, right? You can\ndo everything without it, but it would\nbe silly to not use it. All right. So\nlet me let me push back on something cuz\nyou mentioned it twice now, which is\nlike the term incompleteness. So your\npaper relies on like Siegelmann's, you\nknow, 1995\nsort of paper. She herself now, like\ndecades later, has admitted that that\nthing is a total toy that has no\npractical relevance whatsoever, okay?\nBecause it requires like infinite\nprecision rational um registers that\nencode in a fractal way, etc. And by the\nway, in her paper, all she demonstrated\nwas that under these infinite\nassumptions that she could build a\nparticular RNN that was a universal\nTuring machine. The problem with you\nusing that for your tensor logic is two\nthings. One, that restricts the field\nover which you can have your tensors. It\nmust be one of these fields that has\nlike infinite precision. So infinite\nprecision rationals or whatever. I can't\nuse any other fields. Like no modular\narithmetic, which is actually what runs\non, you know, GPUs, for example. And\nsecondly is it would restrict the actual\nstructure of the weights to her\nuniversal Turing machine. Therefore, it\nwouldn't be a general purpose tensor\nlogic.\nUh well,\nDo you realize this problem?\nNo, no, no. So actually, there is no\nproblem there. Let me tell you exactly\nwhy, right? And let's just do this in\nthree steps. First of all, uh Turing\ncompleteness doesn't matter at all\nwhatsoever because the only difference\nbetween a Turing machine and a finite\nstate machine is the infinite tape. And\nin the real world, there is no infinite\ntape. So if you can implement a finite\nwhy do you keep mentioning it?\nI I'm that's part two. That is part two,\nright? Uh uh\nThis is actually a very interesting set\nof questions. So let's let's let's set\nit let me get a part of it, but like So\ntensor So Turing completeness doesn't\nmatter. What matters is that you want to\nbe able to express any computation that\nyou might want. That's what matters,\nright? You might choose a specific\nlanguage for specific purposes. For\nsomething like tensor logic, you want\nthat generality. You have that\ngenerality irrespective of Turing\ncompleteness. So this is part one,\nright? We can debate, but let's let's\nset that aside for just a second. Now,\nbut\nyou know, the way I don't get to change\nthe way computer science is and Turing\ncompleteness is a shorthand for\nuniversality. I just want to show people\nthat tensor logic is universal. And now,\nI have a proof that tensor logic is\ncomputationally universal that does not\nrely on the on the Siegelmann construct,\nright? I chose to not publish in this\npaper because it would take too long,\nright? The beauty of that is that in one\nparagraph, I can just say, look, the the\nequation in the Siegelmann paper, you\ncan implement it here and we're done,\nright? I can also So you know, there's\nthere's so many ways to prove that\nthings are Turing complete. So I only\nyou I completely agree with you and her\nthat that construct is ridiculous,\nright? It's it's silly, right? It's it\nit has no practical significance, but\nthe reason I used it is like it's just\nmy way of telling people in one sentence\nthat and why, you know, tensor logic is\nTuring complete, right? But the real\naction is I'd love for you to share\nI'd love to see the other proof. Uh\nOh, I can I can I mean, the other So\n[clears throat] so\nactually uh\nthere's even more than one other type of\nproof that is possible. Let me Let me\ntell you what what that one is and what\nSo here's two, you know, not just So\nthree ways. There's the Siegelmann way,\nright? Another one is you have a finite\ncontrol with access to an infinite\nexternal tape, right? That is a much\nmore reasonable thing in my view, right?\nYou have a memory\nYeah. The memory is infinite, but but\nall that you have to do in in the tensor\nlogic is know how to access that memory.\nSo like it gets back Remember, a Turing\nmachine is a finite control and an\ninfinite tape, right? So if the if the\ntensor logic can realize the finite\ncontrol, which obviously it can, and you\ngive it an infinite tape, then we're\ndone, right? And then on that note, you\ncan even just do it the following way,\nright? Which for example, like, you\nknow, Dell Sherman has a great paper\nabout this is there you know, people\nhave come up with various very simple\nways to set up a Turing universal, you\nknow, computer. And one of them is\nthere's a set of rules, right? That, you\nknow,\nsets up that machine, right? Without\ngoing into details. And that set of\nrules, you know, you can just write in\ntensor logic without even, you know,\nhaving to wake up from your sleep. So,\nthere you go. I totally agree with you.\nLike and I I often say to people I'm\nlike, a Turing machine is just and I\nreally hate to use the word just cuz it\njust doesn't do justice to Turing, to\nAlan Turing and like the genius of his,\nyou know, creation, the theory of\ncomputation, right? But it's just a\nfinite control with an unbounded read\nwrite, you know, external memory.\nTotally on board with that. Absolutely,\ntensor logic is a finite control, but\nthen you need to add to it these\noperations to, you know,\nmanipulate external memory, right? So,\nit's kind of tensor logic plus some\noperations to deal with external read\nwrite memory, no?\nI mean, so those operations are just\nread, write, move left and move right.\nThat's all there is. I know, but but\nthat's an extension of I mean, at least\nin my view, I mean,\nI don't know if before you there was\nsuch a thing as tensor logic. I'm not\nsure. I know that a lot of people have\ntalked about tensors for like a decade\nor or more, but um, you know,\nI it seems like some kind of an\nextension to the typical It's certainly\nan extension to the way tensors are used\nin GR. There's no read write to external\nmemory in that.\nOf course, but that is why tensor logic\nis more than tensor in math tensors in\nmathematics, right? The the tensors that\npeople mathematics just don't do this,\nright? But tensor logic does because of\nthe of the logic programming side,\nright? If tensor logic can do logic\nprogramming, then it can do everything\nthat a computer can.\nHave you specified fully like all the\noperators in tensor logic somewhere like\non a website or something?\nThere's only There's only two. Tensor\nproject or three, right? There's tensor\nprojection, right? There's tensor join\nand there's univariate linearities. And\nthe linearities are crucial.\nRight? Tensor algebra is multilinear.\nAlgebra is linear, tensor is\nmultilinear, right?\nTotally agree. Where where does the\nWhere do the memory operations fit in\nthere? Are they projections? Are they\njoins? Are they\nOh, no, I mean, like it's they're not\nThey're not even projections or joins,\nright? I mean, think of a trivial\nprojection where you're not summing\nthings, you only have one, right? That's\nwhat a write is, right? Actually,\nlet's let's let's not even worry about\ntensor joins and projects. Let's just\nthink about, you know, propositional\nrules, which of course are what you If\nyou want to implement propositional\nrules in tensor logic, all that you need\nis tensors with no with no indices, with\nzero indices, right? So, we're only\ndealing with with scalars. And the\nright, right, is just, you know, you\nknow, a rule that says, you know, the\ntarget of the writing is on the\nleft-hand side and what you want to\nwrite is on the right-hand side. Now, to\nget but to get very quickly to the issue\nof an infinite memory, right? What is an\ninfinite memory? An infinite memory is\njust an infinite vector, right? Indexed\nby the memory address. That's all it is,\nright? And so, how do you write this\ninfinite memory in tensor logic? You\njust have the memory as your tensor on\nthe left-hand side. It it's kind of so,\nyou know, it it it's so simple it almost\nthere's nothing to think about.\nI'll have to I'll have to work through\nsome examples.\nand and so, just to finish that thought,\nhow do you advance the tape? Well, you\njust increment the index. And how do\nyou, you know, move it left? You\ndecrement the index, right? It's like\nit's done.\nWell, could we come up with a solid\nexample cuz we I don't think we\nsufficiently described the start T\nfunction. So,\num, roughly as I understand it, rather\nthan it becoming um, a dimension, it\nbecomes a transition function. So, we\ndon't need to model the the full\ntrajectory. But just just to give an\nexample, if I wanted to compute, you\nknow, let's say I want to write a\nfunction\num, to compute the nth digit of pi or to\napproximate it, would I not need to fix\nthe size of the tensors before, right?\nSo,\ndid you see what I The way I understand\nit is that these things have a fixed\nsize. So, how could it possibly solve\nunbounded problems?\nNo, very good. So, to to clarify, start\nT is not a function. Start T is a\nnotation about an index. So, for\nexample, if I have a vector, you know,\nlike\nyou know, X of I,\nright?\nOr or\nbetter example, a matrix M of IJ, right?\nThis occupies, you know, if if I and J\nare each, you know, 100, this occupies\n10,000, you know, positions in memory,\nright? But if what I do is MIJ star,\nright? On the left-hand side of my\ntensor equation, then this is just, you\nknow, instead of being whatever 100 by\n100, it's just 100. Because what this is\nsaying is like, if I put the star in the\nJ, what I'm saying is like, run through\nthe I, right? And for every J, you you\nyou\nuh um,\nyou re you overwrite the result, right?\nYou can do this in either, you know, in\neither\ndimension, but, you know, so pick one,\nwhichever one. It just says like, keep\nkeep overwriting the results, right? So,\nyou lose your old one. So, it So, let me\nput this way. MIJ star is actually a\nvector.\nIs a vector where the only dimension is\nI. J is actually just an iterator for a\nfor loop. You see what I'm saying? And\nconcretely, for example, in an RNN, this\nis what you want because X, right? XI is\nyour vector and the J, let's call it T,\nright? XIT. At every new step in time,\nwhen the state evolves, you don't want\nto I mean, you could, but in general,\nyou just want to overwrite the old state\nwith the new one as in any state\ntransition system. Okay? Now, you're\nyou know,\ndoes this make sense?\nIt It does, but you're describing um, an\naccumulator.\nAnd\nIs it Do you lose something by losing\nthe history? So, because if you think\nabout it, you're you're overwriting what\nwent before with new information and\nyou're just, you know, unrolling in\ntime. Do you lose anything doing that?\nOh, of course you lose. So, if you don't\nwant to overwrite it, then then then\ndon't put the star in, right? But now to\nanswer your question about pi, right?\nWhat How would I, you know, compute all\nthe digits of pi, right? In, you know,\ninfinite Turing machine land, right? I\nhave a vector of the digits of pi that\nhas a start but not an end, right? And\nwhat the computation in tensor logic\ndoes, it computes every successive So,\nlike, we didn't talk about this, but\nthere's, you know, how is inference done\nin tensor logic? Forward chaining or\nbackward chaining. Forward chaining is a\ngeneral They are both generalizations of\nthe corresponding operations in in\nsymbolic AI.\nIf you applied forward chaining to a set\nof rules that computes the digits of pi,\nactually, just one rule cuz it's very\nsimple. What it will do is in each\niteration, it will fill in the next\ndigit of pi, right? Now, if your vector\nis infinite, this will go on forever as\nit should. If your vector is finite,\nwell, at some point you run out of\nmemory and you're satisfied with the\nnumber of digits, which is what we do\nwith any real computer in the real\nworld. I don't want to I always get us\nbogged down into Turing issues. So, I\nthink we should move move on, but I\nthink it'd be fun to talk about it,\nyou know, more at another time or just\nto work through some examples. I think\nI'll probably work through some\nexamples, but I think this was an\ninteresting one. There's um, there's a\nstrange attractor with Turing\nconversations and normally it goes the\nthe Schmidhuber direction where, you\nknow, the universe is finite, there's no\ndifference between an FSA and a Turing\nand I felt I felt that we actually had\nsome information gain in this\nconversation. Well, actually, you know,\nso so on that point and and this is a\nbit of an aside. It doesn't actually\nhave anything to do with tensor logic,\nso I hope you don't mind me asking, but\nsince we have a computer science\nprofessor, like I want to just run\nsomething by with you, you know, so I\nalways get this kind of\npushback from people. We're all say, for\nexample, you know,\nautoregressive transformers and I and I\nmean, classic autoregression, not\nextended autoregression, not generalized\nauto just autoregressive transformers\nare not Turing complete. Like DeepMind\nadmits this and they write a paper\nshowing how you can extend them to\nbecome, you know, Turing complete. So, I\nI'll say something like that and\nsomebody will be like um,\noh, yeah, but, you know, if uh, if if I\ncan't do 100-digit multiplication with\nwith this context size, all I got to do\nis just have more context and then and\nthen then I'll be able to do it. And I\nkeep I keep making them point, here's\nthe crucial difference, right? Between\nSo, and you brought this up beautifully\nwhen you said, look, a Turing machine is\na finite control\nwith an unbounded rewrite memory. And\nhere's the really cool thing about those\nthose Turing machines is they can run in\na way where they're churning, churning,\nchurning, churning and then they say,\nout of memory. And all you got to do is\njust give them more memory and hit\ncontinue. You don't have to reprogram\nthem. You don't have to retrain them\nwhen you've like\nincreased their context size, right?\nThat's the whole difference is that with\na neural network, a traditional\ntransformer, if you increase its context\nsize, go back to the training board, you\ngot to retrain it, right? Cuz you've run\nout of memory. Is that a fair point that\nI'm making? This So, um, this is\nactually extraordinarily simple and it's\nto me incredibly frustrating that\nthere's so much confusion about it,\nstarting with computer science and\ntheoretical computer science and now\nplaying out in AI and transformer land.\nAnd and and and you know, it\nit just boils down to this, right? You\nsaid earlier and I violently agree that\nand correct me if I misinterpreted it,\nbut you said like, Turing completeness\nis not important, but that shouldn't\ncause us to underrate Turing's\nachievement. Absolutely. What was\nTuring's achievement that we now take\nfor granted? Turing's achievement was\nfor which he is deservedly famous,\nright? Is to postulate this notion of a\nuniversal machine.\nThe amazing thing about computers is\nthat they're universal machine, which in\nhis time was an completely\ncounterintuitive notion. What do you\nmean a machine that can do everything?\nThe typewriter can type, you know, you\nknow, like the sewing machine can sew.\nYou're telling me there's a machine that\ncan type with one hand and sew with the\nother. What are you talking about? So,\nlike, this is the genius, right? So,\nfirst step,\nyou want to have this property of having\na machine that can do anything. This is\nthe foundation of computer science as of\ncomputers as a revolutionary technology,\nright? So, point one. But, point two I'm\ngetting to the transformer part, right?\nI don't know unfortunately these\nconfusions and they build on each other\nand and never get it's one of those\nsymmetry breakings, right? We went down\nthis road of defining things a certain\nway and worrying about infinity and and\nand now we're stuck there, right?\nNP-completeness is another example, but\nbut ignoring that. So, the problem with\ntransformers, so like the the real\nproblem is the following is is people\nsay like, \"Oh, but if you only have this\nmany, you know, blocks then you can\nThe the thing for example that inductive\nlogic programming has and we want is\nthat you can learn things from very\nsmall examples\nlike children do in elementary school.\nYou learn to do addition on tiny\nexamples, but then if needed you can do\naddition on numbers of any length. Of\ncourse, your life is finite. You will\nnever add infinite numbers, but that's\nnot the point. Infinity is just a\nshorthand for something that's so large\nit doesn't matter how large it is. And\nwhat I want in machine learning is to\nprecisely be able to learn to do to\nhandle problems, graph structures,\nknowledge bases, inference problems,\nwhatever of any size from very small\nones.\nThat's the limitation that a lot of\nthese transformers have and that's the\none that you want to fix and can fix and\ntensor logic helps you do that.\nAnd I just just to cap off the\ndiscussion about Alan Turing cuz I think\nI think he deserves, you know, us\nmention this. You know, you mentioned\nthat this was the real achievement this\nuniversality. And I mean, it wasn't just\na machine to do it typing can't do this\nand that. It was even within\ncomputation, right? In his time people\ndidn't know this. They're like, \"Well,\nwhat if I have a machine that just has a\nseparate read tape and a separate write\ntape? I don't know. Well, how about if\nwe add two write tapes? Does that make\nit more powerful? What if it's\nread-write? What if it's just a stack?\nWhat if it's lambda calculus? What if\nit's There were so many myriad of, you\nknow, lag systems, blah blah blah. All\nthese different computational models,\nright? And nobody knew that they were\nall equivalent. And that was the real,\nyou know, remarkable No, very good. And\nI mean and to be fair, you know, Turing\nwasn't the only one doing things like\nthis and and precisely now we know that\nthere are all these things that are\nequivalent and then extensions on that\npower. But, here's actually a really\nimportant point, right? The question\nthat has been on my mind for decades is\nthis. A Turing machine is a model of\ndeduction.\nIt's universal deduction.\nWhat we're missing to be able to do what\nthe universe does and evolution does is\nuniversal induction.\nWhat is the Turing machine equivalent\nfor induction, for learning? That's what\nI'm after.\nRight? That's what the master algorithm\nis. And I know it exists and again, just\nas you can have a million different\nversions of Turing machines that are all\nequivalent, you can have a million\ndifferent versions of the master\nalgorithm that are all equivalent and\nthat's okay. The point is that first we\nhave to realize that there is one. We\nhave to prove what it does and then we\ncan refine it with the syntactic sugars\nand whatnot and that's all good, but the\nmain point is having, you know, gotten\nthe universal induction machine, which I\nthink we are we are pretty close to. But\nPe- Pedro, I know the answer. It's\nBayesian tensor logic. No, I'm just\nkidding. No, if you're Bayesian it is\nBayesian tensor\n[laughter]\nThis is This is a good segue because we\nare talking about reasoning and\ndeduction. And transformers, they don't\nreally reason.\nRight? Um I I think of them as a kind of\ncollection of fractured bits of\nknowledge maybe with a little bit of\nunderstanding two levels down, but we\nunderstand many levels down. And when we\ndo reasoning, what we're doing is we are\nrespecting all of the constraints of\nthis epistemic understanding phylogeny\nthing that we have and that allows us to\nbuild new knowledge, right? Because you\ncan build new knowledge, you can create\nnew things when you respect all of the\nunderstanding that you already have. And\ntransformers don't do that, but let's\ntalk about how this works in in tensor\nuh logic. So,\nyou have this um temperature parameter.\nSo, for example, you could do something\nakin to deduction even in an embedding\nspace, right? And and certainly with an\nMLP. And this is where I was a bit\nconfused because I can I can appreciate\nthat if we have um a logical model which\nis in the domain of certainty, we can do\ndeduction, right? And then if we have\nsomething like an MLP and and we learn\nthe weights and we turn this temperature\nparameter up, right? So, it's it's\nactually introducing some degree of\nrandomness.\nWhy would that be anything like the kind\nof logical deductive reasoning we do?\nWould that not just do what neural\nnetworks do now, which is they just look\nfor similarity in some embedding space\nand the type of reasoning it's doing\nisn't actually semantically meaningful\nat all?\nI actually say that like of all things\nin that paper, this is the most exciting\nand important one is that you can do\nsound and transparent reasoning in\nembedding space with tensor logic. And\nhow come, right? Why is that possible?\nAnd to to just sort of like give the\ngist of it, here here's here's the key,\nright? Is you know, think of kernel\nlet's go to kernel machines for just a\nsecond and like the gram matrix, right?\nThe similarity matrix, what is it,\nright? You're in feature space and it's\nfor every pair of objects IJ the dot\nproduct of their feature\nrepresentations, right? And now, if you\nembed all your objects, we already know\nwho they are like, you know, you know,\nthere's a matrix with the embedding\nvector for every object whether it's a\nword or a token or anything else, right?\nNow, I can do the dot product of the\nembeddings of two objects, right? And\nlet's suppose they're all unit vectors\nto keep things simple, right? And now\nwhat happens is that if I\nyou know,\nand let's let's for the moment let's say\nyou're not even learning the embeddings\nyet, right? Let's say you just have\nrandom vectors, right? Your embeddings\nare random, right? That's actually\nalready useful for a lot of things, but\nof course it's not where the action is,\nright? And now there's the following\nvery interesting property, which is the\ndot product of a of a vector with itself\nis one.\nBut, the dot product of two random\nvectors in a high-dimensional space is\napproximately zero.\nSo, your gram matrix, your similarity\nmatrix will be approximately the\nidentity matrix.\nOkay?\nAnd now what happens like if I have a\ntensor logical that operates in this way\nand then it has something like a sigmoid\nnonlinearity, right? Then then then\nwhat's going to happen is that it's\ngoing to clean out that noise and it\nturns into the identity matrix, right?\nAnd now I have all these rules that are\njust operating in a purely logical mode,\nright? They Boolean it's Boolean tensors\ngoing in meaning relations, right? And\nit's Boolean tensors going out, right?\nSo, that way you can do pure deduction\nin embedding space with these embedding\nrandom embedding vectors, right? That's\nalready something interesting, but now\nlet's say you learn the embeddings,\nwhich of course is the whole point,\nright? When you learn the embeddings,\nwhat's going to happen by trying to\nminimize the loss function is that the\nthe the embedding vectors of of objects\nabout which you tend to make the same\ninferences will will get closer,\nright? Because I mean if I'm saying\nsomething about one object and that one\nis similar like this like to to, you\nknow, the gradient descent to minimize\nthe loss is going to is going to make\nthem, you know, is going to increase\ntheir dot product. So, you're going to\nwind up with a similarity matrix that\nhas high values for objects that are\nquite similar, right? In the limit one\nin the diagonal, and has low values for\nobjects that are quite dissimilar. And\nnow uh if if you turn the temperature\nparameter meaning the stiffness of the\nsigmoid, right? At at one extreme at\nzero temperature, you have a step\nfunction.\nAnd and and and and and the the the\nsimilarity matrix is discretized back to\nzero one.\nSo, so at the zero temperature extreme,\nyou have pure deduction.\nBut, it This is This is very right? You\nYou see where I'm going with this?\nI do, but could I could I challenge a a\ntiny bit? So, when when we train neural\nnetworks, um we we think reasoning is\ngood when we are building, you know,\nlet's say we'll use the LEGO analogy.\nSo, we're building these blocks and the\nnew understanding tree that we've\ncreated is a good one if it represents\nthe world in an abstract causal way. So,\nI can see how you've\nyou know, framed this as deduction in\nthe sense that, you know, you you've got\nthis Boolean operation and and you can\nyou can build from it. But, what if\nyou're building on a sand castle? What\nif the What if the component, let's say\nit's an MLP component, what if it just\ndoesn't represent the way the world\nworks? No, no, so very good. So, like\nagain, uh\nthere's more than one thing you can do\nwith tensor logic. One of them is you\ncan just re-implement existing things\nlike MLPs and transformers and whatnot.\nAnd if all that you did was re-implement\nthem, it will have all their pros and\ncons, right? It's the same thing just\nimplemented much more elegantly, blah\nblah, right? What I'm talking about here\nand talk about in in in you know, in\nthat section of that paper is doing\nsomething different. It's not an MLP,\nit's not a transformer. It's actually\ndoing these things of like you embed\nobjects, you embed relations in a\ncertain way that follows from the\nobject, you embed the rules, you embed\nthe reasoning, right? So, this is a\ndifferent process. What this different\nprocess allows you to do is that when\nyou raise the temperature, you get to do\nanalogical reasoning.\nYou know, we we you know, Douglas\nHofstadter came up before. Uh Douglas\nHofstadter I think would like this\nbecause it's it's an analogical like he\nhas this whole 500-page book arguing\nthat all of cognition is just analogy,\nright? And again, this is one of the\nschools of thought like this is one of\nthe tribes in you know, in the master\nalgorithm is reasoning by analogy. You\ndo reasoning by analogy because what\nhappens is you generalize from from from\none object to an object that has a high\ndot product with it. So, now now I'm I\nget to borrow inferences from similar\nobjects. And the higher the temperature,\nthe the looser the inferences, the more\nanalogical inferences can be. But, for\nexample, and again, Douglas goes goes\ninto this in some of his book. And any\nmathematician like I you know, Turnstall\nthe other day I just heard him say this,\nright? He's like, \"Mathematicians reason\nby analogy.\nThey notice similarities between things.\nBut, at the end of the day you need to\nhave a proof.\"\nIn tensor logic, in this scheme, in this\nparticular scheme of of embedding in a\nreasoning in embedding space, this is\njust simulated annealing. You start out\nwith a high temperature being very\nanalogical and then you lower it. At the\nend of the day, you have a proof. It's a\ndeductive proof that is guaranteed to be\ncorrect, but you couldn't have gotten to\nit because the search space is so large\nwithout the analogical part, right?\nOkay, but I I understand what you're\nsaying. So, you you can generalize\nreasoning out, you know, outside the\ndomain of certainty. But, the question\nI'm asking, the reason why we have\nmetaphor and analogy is there's this\nincredible process of evolution and\nintelligence and it's led to the\ncoarse-graining of all of these concepts\nthat we use in our language and there's\nthis rich, beautiful phylogeny that kind\nof represents the causal reality of of\nwhat's happened. And why is statistical\nsimilarity the same thing as analogy?\nOh, it's not. So, again, I skipped over\nsome steps here. It isn't, right? So,\nanalogy So, the the most powerful type\nof So, kernel machines in some sense are\nthe least powerful type of analogy. It's\njust oh, here's a similarity or nearest\nneighbor, right? I have a distance\nfunction. That's not really where the\nthe action is. The action isn't what is\ncalled structure mapping, right?\nStructure mapping was this thing\nproposed by Dedre Gentner which where\nyou solve a problem by mapping its\nstructure to the structure of problems\nthat you know, right? And the canonical\nexample is Niels Bohr's, you know, model\nof the atom which he came up with by an\nanalogy between an atom and the solar\nsystem. The nucleus is the sun, the the\nplanets are the electrons. Turns out to\nbe a bad analogy, but it was crucial in\nthe development of physics, right? And\nstructure And there's also this whole\nsubfield of AI called case-based\nreasoning where I might help desk you\ncome up with a problem and I don't try\nto solve it from scratch cuz I don't\nneed to. That would be a waste of I go\nto my database of similar cases and I\nfind one and then I tweak it.\nSo, structure mapping is an\nextraordinarily powerful thing, but it's\nthe combination of similarity and\ncompositionality which kernel machines\nper se don't have, but tensor logic\ndoes. The point in tensor logic is that\nyou do have all the power of the kernel\nmachines, but all the compositionality\nof the symbolic AI. So, again, this the\nstructure mapping just, you know, just\ncomes out of the box. You don't need to\ndo anything more to have structure\nmapping and all the power of analogical\nreasoning that comes with that. Yeah,\ncan I suggest um a good analogy is to\nad-lib? Do you think that's fair? It's\nlike you've got the general structure\nthere and you can plug in parts into the\nblank spaces and you get, you know, a\nsolution, right? That's one mode in\nwhich things can function, right? You\ncan also\nYeah.\nThe the whole process of structure\nmapping or of case-based reasoning can\nactually be very rich. I can combine,\nfor example, two big pieces, but like\nthat's that's one example, yeah.\nOh, yeah, yeah, yeah. No, that that's\nfair. I mean, yeah, it has this this\nnice nested structure, you know,\nproperty.\nUm since while we're on this topic,\nlet me ask you about something that I\nwas confused about in the paper. So, I\ndon't understand your connection\nbetween hallucination and and deduction\nor determinate, you know, determinism\nbecause in my mind, you know, I can set\nthe temperature to GPT to zero and it\nstill hallucinates and I can I can have\na I can have a poor deductive system\nthat hallucinates all kinds of things.\nSo, to me like those are separate\nseparate problems. Like where what was I\njust kind of misunderstood\nVery good. So, precisely the problem or\none of the problems with GPT is that it\nit hallucinates even when you set the\ntemperature to zero. What the hell,\nright? I want to have a mode\nright? Not I, but like every Fortune 500\ncompany, if it's going to use AI, needs\nto have a mode where the the logic of\nthe business is just to behave. The\nsecurity isn't violated, the\ncustomer doesn't get lied to, etc., etc.\nWe got to have that or you have in the\nday will not take off, right? And\ntransformers can do that. Tensor logic\ncan do that precisely because\nthe in this, you know, reasoning in\nembedding space, you know, mode that I\njust described, if you set the\ntemperature to zero, it does purely\ndeductive reasoning. And by the way, the\ntemperature can be different for each\nrule.\nAnd I think this is what almost all\napplications are going to have is like\nthere are some rules that are either\nmathematical truths or logic that you\nmust guarantee will not be violated.\nThey are the laws, right? And those have\ninfinite temperature. And then there's\nall these others that are more\nqualitative reasoning and like more\naccumulating evidence, maybe stuff that\nyou mine from the web. And those, you\nknow, those will have lower those will\nhave higher temperature. And that\ntemperature parameter, you know, can be\nlearned in some rules and in others,\nright? So, now you have this whole\nspectrum between the deductive and the\nand the more, you know, or even\nfantasizing, like truly hallucinating at\nthe far end of the high temperature,\nright? But, precisely the point that I'm\nmaking in the paper is that, you know,\nat with LLMs, the best thing you can get\nat zero temperature is still a lot of\nhallucinations. Then there's things like\nrag, but all they do is retrieve and\neven then you still hallucinate, right?\nCompare, you know, tensor logic in this\nmode with rag, right? And it doesn't\njust retrieve things. It computes the\ndeductive closure of your knowledge,\nwhich is an exponentially more powerful\nthing to have, right? And with zero\nhallucinations. Well, it it it is if if\nthe model\nrepresents the world. You know, you\nknow, because um what what does\nhallucination mean? Or actually, what\ndoes slop mean? My definition of slop is\nwhen a creative artifact is produced um\nby something that doesn't understand.\nSo, if I'm if if I understand the domain\ndeeply, that artifact looks incoherent\nto me because it was generated by a\nprocess that doesn't understand the\nworld. And it isn't it even the same\nwith tensor logic that, you know,\ndeduction is great, but if the model\nisn't a good one, then wouldn't that\njust be a hallucination as well?\nOh, absolutely, but let's let's make\nsome distinctions here, right?\nThe only claim I'm making here because\nit's the only one you can make is that\ntensor logic at zero temperature in this\nmode will give you the soundness\nproperties that logic has, right?\nSoundness in the technical sense of\nsoundness. All that means is that you\nonly reach conclusions that truly\nlogically follow from the premises. You\ndon't say anything about whether the\npremises are valid or not. If the\npremises were hallucinated, so will the\nconclusions be, right? There's like\nthere's no there's no magic there,\nright? But, that is a very important\nproperty to have. Again, if I give to a\ntransformer a bunch of, you know, true\nfacts, it still hallucinates. And that's\nwhat I can guarantee will not happen in\ntensor logic. Now, coming up with the\ntrue facts, well, that's a different\npart of the game. You can write them\ndown, you can learn them, you can refine\nthem. You never know for sure if you\nhave the perfect model. And of course,\nthat's more the machine learning and\nknowledge acquisition part, right? So, I\ndo I think have a very important\nguarantee here of non-hallucination, but\nit's not a guarantee that the, you know,\nthat the model that you're working on\nand, you know, came from the real world.\nThat's a whole other, you know, neck of\nthe woods. Who's going to adopt this\nfirst? How are we going to bootstrap\nthis as as a community? How how do you\nsee this progressing?\ngood. So, you know, the last section in\nthe paper is is discussing adoption and\nwhat needs to happen and things like\nthat. Let's suppose that everybody\nagrees tensor logic is a beautiful,\nperfect language and what we need for\nAI.\nJust for that reason it that would not\nbe enough to make it take off, sadly,\nright? Because, you know, there's a\nvery, you know, things get, you know,\npeople are still using COBOL these days,\nright? I rest my case, right? So, legacy\nit there's this irony in in in computer\nscience or in the information technology\nindustry, like it moves faster than\nanything else, but at the same time, you\nknow, things never die, right? You can't\nkill them. You can't kill COBOL, right?\nAnd and and I really do believe, you\nknow, I like Python, right? I I program\nin Python. It's very nice in many ways,\nbetter than Fortran for some things,\netc., right?\nEven though it was never or NumPy, if\nyou will, but but you get the point,\nright? Like it's like for AI, it's just\na terrible thing. But like I'm a Python\nprogrammer, like I, you know, general as\nlike like, okay, your tensor logic is\nnice. I'm not going to rewrite all my\ncode, right? Forget that, right? So,\nwhat is going to make it happen, right?\nBut now, we can look at what has made\nthis happen in the past, right? And it's\nseveral things. One is that, for\nexample, look at how Java took off,\nright? Java took off at the time\n[snorts] of the internet because it was\nthe language of networking, allegedly.\nYou could debate that, but like people\nwanted to do things that it was, you\nknow, very hard to do with, you know,\nthings like C and blah blah and C,\nright? And so, Java took off, right? And\nwe are in exactly It was the language of\nembedded programs and web browsers. Like\nthat was that was the only option we\nhad, right? Exactly, right? So, and you\nknow, there's big arguments about this,\nbut not relevant to us here. The the\npoint I'm trying to make here is we are\nprecisely at a also very relevant. Why\ndid languages like Lisp and Prolog, you\nknow, fall out, right? Because they were\nbetter for AI than, you know, Fortran or\nC or or or whatever, right? Or or Java.\nIs that like they were niche languages.\nAnd the network effects of the more\nwidely used languages and all their\naspects just completely overrode that,\nright? We understand that very well now,\npeople didn't in the '80s. But now,\nwe're in a different ballgame now. Now,\nthe big technology, the center of\neverything is AI, right? If you have a\nbetter language for AI, that is the one\nthat is going to, you know, have the\nbiggest users. And moreover, if you have\na language that solves the big pains,\nright? To adopt a new language or a new\nanything, right? You know, like a new\napp, right? It needs to solve some big\npain, right? Is there a big pain that\ntensor logic solves? Well, hell yeah. It\nsolves hallucit potentially, okay? It\nsolves All this is subject to empirical\nverification, but it potentially solves\nhallucination. It solves the opacity,\nright? Like we're in this world right\nnow where there's like multi-billion\ncorporations and systems that are like\nthey're they're driven by this black\nbox. And nobody I know I've talked with\nCEOs of big tech companies that say\nlike, you know, I can't sleep at night\ncuz I don't know what this thing is\ngoing to do and the people who trained\nit have left the company and who knows\nit, right? So, if we can make a dent in\nthat, people will convert to it very\nvery quickly. Also, I think when people\nhave the experience of how easy it is to\nuse TensorLogic compared to the big pile\nof stuff that is lies under PyTorch and\nwhatnot, I think that they they will\nactually be very very motivated to\nto migrate very quickly. And then, you\nknow, like and there's several things.\nThere's like developing the open source\ncommunity and vendor competition and\nwhatnot. But but but you know, like\nthere's there's a couple of other\nimportant things here, one of which is\nthe following.\nUh TensorLogic is ideally suited for AI\neducation.\nIt's one language in which you it which\nhas very little, you know, extraneous\nstuff, and you can and you and you can\nteach the entire gamut of AI very well\nand do the exercise. It'll be a language\nthat the professors, the TAs, and the\nstudents will like, right? And and\nhistory shows, you know, going back to\nthings like Unix, that if you have\nsomething like that that takes off in\ncomputer science education, then people\ngo to industry and say like, \"I want to\nuse this cuz it's what's good it's what\nI like.\" And and a generation later,\nit's it's it's what everybody is is\nusing. And and one more thing is the\nfollowing. There\nyou the transition to\nTensorLogic from from from Python\ndoesn't have to happen all at once,\nright? You can have, for example, and I\nalready have actually,\nanother's have. Like again, because it's\nvery easy to do, right? You read the\npaper and you're like and you do that in\nthe next whatever 30 minutes. You can\nwrite a preprocessor\ninto Python. And again, all it does is a\none-to-one mapping between the syntax of\nof\nTensorLogic and einsum, right? Then\nmaking things efficiently, as we\ndiscussed, is another matter, but from\nthis point of view of of developer, you\nknow, uptake, right? All it there's a\nlong long history of people doing this\nwith different languages, right? It's\nlike you have a preprocessor that that\nthat lets you write tensor equations,\nbut then it converts you it converts\nthose equations into PyTorch or Python\nor just NumPy, let's say Python, right?\nAnd then you do everything else in\nPython that you did before. You don't\nlose anything, you don't lose any\nexisting code. It's just that a set of\nthings, and in particular, reasoning\nhave now become much easier than they\nwere before. And then once you have this\nlittle lollipop like, \"Oh, but I can do\nthis, and I'll let me have that piece of\nsyntactic sugar.\" And before you know\nit, people like, \"Well, I don't need all\nthat, you know, Python stuff anymore. I\njust rather live in TensorLogic world.\"\nSo, you said about AI for education, and\nand TensorLogic it's a declarative\nlanguage, which means it's it's the\nwhat, not the how. It's this incredible\ncoarse-graining that screens off a lot\nof unnecessary detail. But is it is it\nunnecessary? I guess is the question.\nLike, do you think that people learning\nabout AI should know about how the\nunderlying things work? And certainly\nfolks working at Google, they might need\nto do some domain-specific optimizations\nfor certain components of the machine\nbehind the scenes. Do you think that we\ncan screen off all that detail? Great\nquestion, but actually, let me start by\ncorrecting something.\nTensorLogic, like like Prolog and\nDatalog, has actually both declarative\nand procedural semantics.\nYou can look at a tensor equation.\nThat's actually the whole beauty of\nlogic programming in some sense is that\nlike you can look at a TensorLogic\nequation is like it's an equation.\nIt's like Einstein's equation. It's a\nstatement about the world. But you can\nalso look at look at it and treat it as\na function call.\nThe left-hand side is the call, and the\nright-hand side is the body, which is a\nbunch of other calls and the way to\ncombine them, right? So so you can And\nin fact, you know, most of the time, in\nin my experience that I've used\nTensorLogic so far, I tend to use it in\nprocedural mode, right? It's it's a set\nof equations. It's a bunch of\nstatements, like you would have in in\nany imperative language, right? So so\nyou know, very very important to to have\nto bear that in mind. Now, but but but\nto the heart of your question, right,\nwhich I think is very important.\nWhen you're teaching people something, I\nmean, I would actually say this is the\ntragedy of computer science education.\nFrom high school to intro courses to the\nmost advanced things is that you want to\nteach them the beauty of what you can do\nand the essence of the algorithms and so\non, but then you and particularly they,\nthe students, they spend all their time\nbogged down in all this crap. All these\ndetails where you get the semicolon\nwrong and the program doesn't work\nanymore, and they hate it.\nAnd they decide that computer science is\nnot for them, or at best, they waste 10\ntimes more time than they should, right?\nSo, precisely the whole point of having\nthe right abstraction is to avoid that.\nNow, so I would say this is one of the\nbest features of TensorLogic is to do\nthat for AI. Now, but you also say,\ncorrectly, that like, well, a lot of the\ntime you need to go beyond that level of\nabstraction, and for example,\nfrom a point of view of efficiency and\nso on and on a lot of things, right? But\nI would say, and again, you know, like\nwe won't know until TensorLogic is used\nwidely and we see what happens, but\nTensorLogic is a language that at some\nlevel it's it's like C, right? It's very\nlow-level. The beauty in in my mind,\nagain, this gets back to the multiple is\nthat like you can use it to say very\nhigh-level things. You can also to\nexpress you can also to express the\nlowest-level possible computations,\nright? Like a tensor equation is\nsomething that you can map onto a GPU\nwith almost no change, right? And and\nthen optimize the heck out of it, right?\nLike in fact, you know, I've joked like\nwith folks at Nvidia that, you know,\nCUDA is a nice moat, but TensorLogic\ncould be the end of that moat.\n[snorts]\nI sometimes feel like, and I I'm not\nsure exactly how much money was spent on\nbigger and bigger transformers, you\nknow, deeper and deeper, you know, wider\nand more data and whatever transform\nmore parameter transformers, but it's\ngot to be a lot, like a trillion dollars\nor or something like that. And I feel\nlike sometimes we spent a trillion\ndollars to learn, yet again, lessons\nthat people could have learned if they'd\nhave taken certain, you know, basic\ncourses in computer science.\nLike, I'm wondering if you sometimes\nfeel like that, and what lessons, if\nany, you think people should have known\nbefore spending a trillion dollars.\nI violently agree with that. In fact,\nthe the paradox of the current moment in\nAI is that on the one hand, this is\nsuper exciting, right? This is what\nwe've worked all our lives towards,\nright? It's like the dream is happening.\nI used to tell people, you know, when I\nwent into grad school that like one day\nmachine learning is going to take over\nthe world and be like, \"Uh what?\" And\nI'm like, \"See, it is taking over the\nworld. There, take that.\" So, on the one\nI you know, on a more serious note, like\ntransformers are a great leap forward,\nand you know, anybody who's used a\nchatbot is like, \"Wow, look at the\nthings this can do. This is great.\" But\nat the same time,\nthe sheer amount of like wastefulness\nand stupidity and ignorance going on is\njust unbelievable. It's like, \"Why are\nyou reinventing Why I'm For example,\nI've talked with people at for example,\nOpenAI that do the reasoning. And many\nof them are very good people, so I'm not\ntrying to, you know, pick on anybody,\nbut it's like, \"Oh, what is reasoning?\nWe need to figure that out.\" And like\nand then they say a bunch of stuff that\nis completely wrong. And I'm thinking to\nmyself, \"Why don't you spend an\nafternoon reading a couple of chapters\nof Russell and Norvig and save a hundred\nbillion dollars in wasted compute?\nPlease, just do that, right?\"\nAnd you know, way, you know, part of\nwhat I'm trying to do with TensorLogic\nis make things go in that direction,\nbecause the current direction is just\ntoo damn painful. And you know, it's not\njust that it's painful. This is going to\nend badly, right? Uh people are spending\nall right,\nOh, yeah. in a way like, you know,\nspending all this money on data centers\nis not wasted, because it's not like the\nfiber that went dark, right? We in AI\nhave an appetite for unlimited compute,\nright? But they're spending all this\nmoney prematurely\non stuff that isn't ready for that yet,\nright? The demand is probably not going\nto be there, and and we're going to look\nback onto then go like, \"Wow, 99.9%\nof that compute was completely wasted.\"\nBecause of a lot of the reasons that\nwe've been talking about, including like\nyou didn't know how to do reasoning, so\nyou brute-forced it, etc. etc. So like,\nyou know, we we got to change the\ndirection of this ship.\nIt's like that that well-known quote\nfrom, you know, Matt Damon in in\nGoodwill Hunting, right? Like, you know,\nto to paraphrase it, you know, you've\nwasted a trillion dollars on an\neducation you could have got for a buck\nfifty in late fees at the library.\nExactly. Exactly.\nWell, Professor Pedro Domingos, it's an\nabsolute honor to have you on the show.\nThank you so much for joining us.\nThanks for having me. Thank you. Always\na pleasure.",
  "transcript_chars": 96094,
  "ingested_at": "2026-05-12T00:41:35.174319+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 20095,
    "like_count": 687,
    "channel_id": "UCMLtBahI5DMrt0NPvDSoIRQ",
    "categories": [
      "Science & Technology"
    ],
    "tags": []
  }
}