{
  "video_id": "TiW96H5HmAw",
  "channel_slug": "openai",
  "channel_handle": "openai",
  "title": "Why AI needs a new kind of supercomputer network — the OpenAI Podcast Ep. 18",
  "duration_seconds": 2259.0,
  "url": "https://www.youtube.com/watch?v=TiW96H5HmAw",
  "upload_date": "",
  "transcript": "Hello, I'm Andrew Mayne, and this is the\nOpenAI Podcast. On today's episode, we're\ndiscussing how\nto make supercomputers better at training\nmodels. Joining me are Mark Handley from\nthe core networking\nteam and Greg Steinbrecher from workload\nsystems. They'll discuss how a\nbreakthrough has made\ntraining more efficient so everyone gets\nsmarter models faster.\nThis has really allowed us to remove one\nof the key barriers to continuing to\nscale.\nWe're talking about a lot of the world's\nfastest GPUs and making them all work\ntogether on a single task.\nWe know we've won when researchers stop\nneeding to know what network protocol this\nparticular cluster is using.\nSo tell me a bit about your background.\nI started out doing physics and math in\nundergrad, wanting to basically understand\nhow complex systems work. I always liked\nthe part of physics\nthat's about how do you take this thing\nthat is unknowably complicated and build a\nsimple model\nthat is a complete and utter lie but tells\nyou something about that system. And then\nbuild your\nintuition on that and kind of build more\ncomplex models. I ended up doing a PhD\ntrying to build quantum computers.\nAmbitious PhD.\nYou know, little things, little things.\nUnfortunately, what I liked is big\ncomplicated systems. And you'll note that\nquantum computers\ndon't work and therefore they don't scale\nyet. They will someday, but they don't yet\nwork. And so\nI kind of took a look at the chips we were\ndesigning to control light for quantum\ncomputers.\nAnd I went, huh, that kind of looks like a\nnetwork switch.\nWhat if we use this as a network switch?\nAnd what I found out pretty quickly was\nthat academia does not know a whole lot\nabout what\nreal data center workloads look like.\nYou get a whole bunch of kind of very toy\nmodels, but they're not very informative.\nAnd so I ended up kind of pitching an\nindustry company to get a fellowship.\nThey paid for the last two years of my\nPhD.\nI ended up working there for a while,\nbuilding out, kind of doing initial\nnetwork hardware\njust to try to understand what is it that\nwe actually need from data center\nnetworks.\nWhat I found out was that there's a huge\namount of headroom on just conventional\ndata center\nnetworking hardware and lots of room for\noptimization. We did not need my little\noptical\nchip. We did not need to do anything fancy\nlike that. But there's all sorts of really\nfun problems.\nAnd then around that time, the whole AI\nboom started to kick off. We decided we\nneeded to\nbuild big GPU clusters. And in particular,\nwe needed to build networks for those GPU\nclusters.\nAnd so I got roped in on trying to build\nkind of simulations of those so that we\ncan figure out what to build.\nAnd then in the process of trying to build\na simulation of these systems, you learn a\nlot about how they have to work.\nAnd at some point I said, well, why don't\nI just go build the actual thing?\nAnd so I transitioned from writing\nsoftware to build simulations to just\nwriting the software that allows GPUs to\ncommunicate with each other.\nAnd then a little over a year ago, ended\nup coming here to OpenAI to do some of the\nsame stuff, but to get even closer to the\nactual model training.\nSo the team I'm on is responsible for more\nor less making sure that we'd use the GPUs\nefficiently.\nSo are the models training as quickly as\nthey can be?\nAre we not bottlenecked on the network?\nWhat do we do when something fails?\nAre we restarting efficiently?\nHow do we kind of work around quirks in\nthe hardware?\nand yeah now I get to play with some of\nthe most fun hardware in the world and\nlike try to make it —\ntry to kind of squeeze every last ounce of\nperformance out of it.\nMark you've had a considerable amount of\nexperience trying to get computers to talk\nto each other and do meaningful\nwork tell me more\nabout your background.\nSo when I'm not at OpenAI I'm a professor\nat the University College London and I've\nbeen doing networking research for more\ndecades than I care\nto think about. Originally, I started\nworking out on trying to make the internet\ndo video conferencing.\nAnd back then, that was a really difficult\nthing because the computers were so slow.\nAnd then we came up with a way for doing\nthat that suddenly the rest of the world\ngot interested in.\nAnd so the standards we wrote for doing\nthat are now the things that your phone\nuses to communicate\nwith 4G and 5G networks. And so that was\nthe first part of my life, was trying to\nstandardize all of\nthat sort of stuff, that everybody could\nactually take advantage of that.\nAnd the problem with standardizing is\neverybody needs to agree.\nSo it just, it takes a long time to\nactually get everybody to agree on\nanything.\nA while back, I got interested in what was\nhappening in the data center world.\nAnd it had the big advantage that you\ncould actually do something different\nbecause you\nonly needed to agree with it while\nbuilding, not the whole world.\nAnd so that was how I got into thinking,\nwell, this data center networking stuff is\na really interesting place to be.\nWell, it's an interesting problem because\nI think that just the idea that we'd be\nscaling as many GPUs as we are now, it's\nhappened so fast, happened so quickly.\nAnd often, we're still using GPUs, which\nare graphical processing units.\nWe're just now starting to use next\ngeneration chips to do this.\nSo how much has been the work regarding\njust to update the way we think about it\nand update the sort of the tools we're\nusing for this, which is what we're here\nto talk about with.\nMultipath Reliable Connection.\nSo from a network point of view,\nthe data centers that we traditionally\nused to build,\nthey kind of got derived from what we used\nto do in the internet.\nAnd when you do communication in the\ninternet,\nyou have lots and lots of people\ncommunicate.\nIt's like there's a lot of data moving\naround,\nbut they're all doing their own separate\nconversations.\nAnd so generally speaking,\nif the more communications you add onto\nthe same shared network,\nthe more things smooth out and become\neven.\nAnd so that's great. You can take\nadvantage of the statistics of large\nnumbers.\nUnfortunately, when you look at what we're\ndoing when we're training things, it's\nexactly the opposite from that.\nWe're talking about a lot of the world's\nfastest GPUs and making them all work\ntogether on a single task, which is why\nthis stuff gets hard.\nThat process of basically ingesting all of\nthis data and allowing the AI model to\nlearn from that data in parallel is what\nallows us to build smarter, better models.\nBut let's say one GPU goes a little slow.\nWell, now all the other GPUs have to wait\nfor it. That's all wasted time.\nOr one of the GPUs, you know, a cosmic ray\nhits it and some bits flip and it stops.\nWell, OK, now that whole step is maybe not\nuseful and we have to maybe roll back or\nkind\nof stop and take stock of what has\nhappened.\nAnd while we stop and take stock, all the\nGPUs are not doing useful work.\nYeah, a key thing here is that the\ncommunication between the GPUs is actually\npart of the computation.\nIt's like they're doing one big\ncomputation across all of them.\nIt's not they're doing different things.\nWe have to actually have them communicate\nwith each other in order to agree on what\nthe result of that step of computation is.\nAnd that's just about the worst possible\nworkload you could think to put onto a\nnetwork.\nAnd so the way the industry has evolved\nover the last couple of decades has been\nslowly coming up with improvements on how\nwe actually do that.\nBut until recently, the scale wasn't\nenough that it really mattered.\nYou could get away with doing the same\nsort of things we did on the internet,\njust bigger.\nBut you can't get away with that anymore.\nAnd that's why we tried to think of\ndifferent ways to actually solve these\nproblems,\nto cope with these very synchronized\nworkloads as we scale things up.\nAI has upended a lot of things in the\nworld.\nIt has definitely upended the way that\ntechnology companies have to think about\nthe data centers that they're building.\nconventional hyperscalers for kind of the\nweb era, the teams that built this were\nvery kind of\ndisconnected from the individual\nworkloads. The goal was to basically just\nprovide an ocean of\ncompute. AI has forced us to think very\ndifferently. And OpenAI in particular has\nbeen kind of at the\nforefront of realizing that the systems\nand the design of the systems is integral\nto the training\nof the models and that you can't just have\nyour infrastructure team sit over here in\none building\nand kind of just deliver an ocean of\ncompute and your model team sit over here\nand like try to make\nthe best you know model that they can on\nthat compute you really have to do kind of\na co-design\nacross these whole things um and so you\nknow we on my team sit literally next to\nthe researchers\nand talk with them every day about you\nknow how they can best you know how their\nworkloads fit\nbest onto the existing servers. And in\ndoing that, we learn a lot about where the\npain points are.\nAnd we are on call for the big training\nruns. We get woken up in the middle of the\nnight if\nsomething is broken and can't be fixed.\nAnd so through that process, and you start\nto go,\noh, well, what if in the next generation\nwe fix these problems? What if we didn't\nbuild data\ncenters with the same properties as we did\nfor web scale workloads? What if we\ninstead fix this\npiece or this piece or this piece. And I\nthink the network has been a real source\nof pain for us.\nYeah. So when you're building a data\ncenter network, all of these GPUs, because\nthey do\nthis computation and they all need to talk\nat the same time, you need a lot of\nbandwidth.\nAnd the problem with that is you can't\nbuild that with a single switch or even a\nhierarchy of\nswitches. You have to build hierarchies of\nhierarchies of switches. And so that means\nthat\nwhen you communicate from one GPU to a\ndifferent GPU,\nthere are many different paths your\ntraffic could take through that network.\nThousands of different paths your traffic\ncould take\nthrough the network because we build\nso many different switches in there.\nThere's several thousand switches in one\nof these buildings.\nNow that gives us an interesting problem,\nwhich is which path do you take from one\nplace to the other?\nAnd if you basically have the requirement\nthat I want to be able to send from one\nGPU\nto another GPU as fast as possible,\nand I choose, say, a random path through\nthe network,\nif I get lucky and nobody else chooses the\nsame path, then great.\nBut if I get unlucky and two people choose\nthe same path, we go slow.\nAnd if 10 people choose the same path, we\ngo really slow.\nAnd so this statistical multiplexing that\nwe used to have when we designed the\ninternet\njust doesn't work out very well when we're\ntrying to build these networks for our\ndata centers.\nAnd so that's where we came in to try and\ndesign things somewhat differently.\nMaybe to put a fine point on that,\nthe synchronous nature of the workload\nthat we talked about earlier\nis why this becomes such a problem.\nIt is not about how fast can the average\npair of GPUs talk to each other.\nIt's always what is the absolute worst\ncase that occurs.\nSo if you think about you've got thousands\nof GPUs that are trying to talk to each\nother.\nSo there's tens of thousands or hundreds\nof thousands of network flows on tens of\nthousands of links.\nWhat you have to do is you have to look at\nthat entire network and go, what is the\nlink that got most bottlenecked here?\nThat one link is going to set how fast all\nof your GPUs are able to work and how much\ntime it takes for you to move data through\nthis.\nbecause everything is proceeding in\nlockstep. And so whereas previously we\nmight have been\nsubject to or kind of taken advantage of\naverage statistics, we don't have that\nluxury anymore.\nWe instead are subject to like the tail of\nthe tail, we call P100, the 100th\npercentile statistics.\nAnd that leads to very different systems\nrequirements than when you can kind of\nrely\non the law of large numbers to take care\nof you.\nAnd so the other problem with this is that\nwhen you build your networks, we build the\nbest possible networks we can. We go to\nthe best\nequipment vendors, we use the best optics\nand so forth. When you really scale things\nreally big,\nthings are always going to be failing.\nLinks will fail, switches will get\nconfused and will have to\nbe rebooted and so forth. Any one of these\nfailures is going to affect the traffic\nrunning\nover the network. And so if you've got\nthis problem where we only care about the\n100th\npercentile and then a link fails in the\nnetwork, what happens? Well, stuff will\nfail. We may have\ntime before the routing reconverges and\nthen we move the traffic onto a different\npath. We take a\nglitch there. That can be quite a long\nglitch that can cost us. Or worse, we can\nactually cause one\nof these communication transfers to fail.\nA single transfer fails and you could end\nup with the whole\njob crashing. And so we really want to\navoid that kind of problem. We want to\nbuild a way of using\nthese networks that is resilient to not\nonly the potential transient congestion we\nmight build in\nthe network, but also when things fail, we\njust want to be able to carry on and\nbasically not notice.\nBut that requires that we actually design\nthe network protocols differently from the\nstart.\nYou can't retrofit this onto existing\nnetwork protocols.\nYeah, it's a very interesting problem as\nyou describe it,\nbecause I can see that where you could say\nlike,\noh, well, if we have a thousand GPUs,\nthere's only one chance out of 10 that\nthey're going to fail or whatever.\nAnd now I have a hundred thousand GPUs.\nWell, guess what? I'm going to have a\nfailure all the time.\nAnd that's what you have to sort of solve\nfor each time you scale it.\nSo where does this break? Everywhere.\nThere we go.\nIf you think about the meantime between\nfailure of the equipment,\nfor any particular range of equipment,\nYou've got some time between which\nsomething will fail somewhere in your\nbuilding.\nAnd of course, the bigger we get, with the\nsame cost of equipment, that time comes\ndown further and further.\nAnd eventually you get to a point where\nactually something is failing sufficiently\noften that you don't get any work done on\na large synchronous workload.\nAnd we can't have that happen.\nSo we have to do things differently to\nmake that work. Yeah.\nSo the very simple math here is you can\nbasically assume that if failures are\nindependent and you double the size of\nyour system,\nyou're going to have half the time between\nfailures, right?\nYour mean time to failure goes down by\nhalf.\nThe important thing to think about here\nfor the network is for every GPU,\nwe have tens if not hundreds of network\ncomponents.\nSo even just like, say you've got one GPU\nconnected to one network adapter.\nIn that network adapter, if it has an\noptical transceiver in it,\nmaybe you'll have four lasers.\nOn the other end of that transceiver,\nyou'll have another four lasers.\nAnd so already just connecting that one\nGPU just to its first hop switch, you've\ngot an order of magnitude more lasers than\nyou have GPUs.\nNow add in multiple layers of switching\nand you start to get into several orders\nof magnitude more components in the\nnetwork than you have kind of at the edge\nof your network.\nBecause we need to have so much bandwidth\nhere.\nSo we have to build these networks.\nWe can't kind of taper them down and only\nhave a couple of components in the network\nbecause then we'd be starving the GPUs.\nThey wouldn't be able to actually kind of\nuse their full capability to do math as\nquickly as possible.\nInstead, we would just be letting them sit\nidle and wasting time, money, energy.\nWe would get trained models more slowly.\nIt would be bad.\nAnd so we built a very big network.\nBut now you have many, many, many more\ncomponents in that network than you have\nmaybe at the edge of your network.\nYou have literally millions of optical\nlinks within the same building.\nSo it's a huge scale. You mentioned the\ndata centers originally were.\nI go do something to search the cloud, to\nget my email, whatever.\nI may be talking to one server, there\nmight be a backup there.\nThe idea that we would be having more\ncomputers inside a data center talking to\neach other than we did just a few years\nago, people trying to connect to it.\nAnd so how have these protocols evolved?\nOne of the things that got me into\nnetworking in the first place, I have a\nvery distinct memory of, I was at a\nconference, OFC, it's an optical\ncommunications conference back in 2017.\nAnd they had a presenter from Facebook.\nand he put a chart up that had a stacked\nline chart of the traffic that they serve\nto their\nend users and the traffic that goes inside\nof their data centers. And the traffic\ninside of\ntheir data centers was just exploding,\neven while the kind of amount of traffic\nthat they were\nsending to end users was staying constant.\nAnd this is way before GPU clusters and\nAI. So this is,\nwhat AI does is it takes all of the\nsystems challenges that people were having\npreviously\nand it cranks them up to 11.\nSo to address this,\nyou've been working on a method,\nMultipath Reliable Connection.\nThe insight was basically that you have,\nwe try to manage congestion in networks.\nAnd so there are several pieces\nthat you can pull together and do this.\nAnd the first part of the insight was\nif you spray the packets across many\npaths,\nyou can low balance those paths\nthrough the network really equally. And if\nyou do that\nand you build a network topology that has\nenough capacity,\nthen you don't cause hotspots in the\nnetwork.\nIt leaves you with just one place where\nyou have congestion,\nwhich is if multiple people try to send to\nthe same destination at the same time.\nBut it also leaves you some problems\nbecause the packets can get jumbled up in\ntransit\nbecause they're taking different paths.\nAnd so if you do manage to cause\ncongestion and cause loss,\nit's a little bit difficult to figure out\nwhether you got lost\nor whether you should still be waiting for\npackets\nbecause they got jumbled up in transit.\nAnd so the second piece of this is a\ntechnique we call packet trimming,\nwhich is if you're causing congestion in\nthe network\nand you would overflow a queue,\nnormally we'd just drop the packet.\nAnd then we've got ambiguity. Did it get\nlost? Did it get lost?\nHow long do I need to wait for it?\nBut what we do instead is we will trim off\nthe payload of the packet\nand just forward the little tiny packet\nheader to the destination,\nwhich can immediately request a\nretransmission\nand we can retransmit that packet.\nAnd that totally removes the sense of\nambiguity\nas to whether we lost packets due to\ncongestion\nor whether we should still be waiting for\nthem because they got reordered.\nInteresting. So just making sure that the\npart saying, are you there, goes through,\nand then you can figure that out and then\nsend the rest.\nYes, you really need to know\nthat you should still be waiting for it\nor you should not still be waiting for it.\nWhat does this mean for the end user?\nThe biggest thing that this means\nis that you're going to get better models,\nmore intelligent models, faster from\nOpenAI.\nSo MRC allows us to accelerate\nevery part of our research and deployment\npipeline.\nIt allows individual users to not worry\nabout their jobs failing, not worry about\nhow their job has gotten scheduled and\nwhether the performance of it is going to\nbe different because they're placed on the\nsame rack as someone else's job.\nIt allows us to train frontier models much\nfaster, more reliably, and really just to\nturn the entire crank of that pipeline\nmuch faster and much more reliably.\nSo you should expect to see an ever\nincreasingly exciting pipeline of releases\nfrom us. The vibes are good?\nThe vibes are good. The vibes are very\ngood.\nThe idea came out of a lot of research\nwork that we've had over the last few\ndecades.\nWe're not fundamentally inventing anything\nnew.\nWe're just doing things that other people\nhave invented, but pulling the combination\ntogether into a set of features.\nSo we formed this group of people who are\nall interested in doing this.\nAnd last year, we finally got to the point\nwhere we were able to deploy this.\nAnd we went from, in a few months, from\nthe first hardware available to actually\nrunning\nand training models and it actually all\noperating.\nSo this has the result that we don't cause\nthat congestion that we talked about\nbefore.\nThe second really nice property is that if\nsomething fails in the network, every\nsingle\none of these flows that go through there\nwill probably be affected by the failure.\nBut it'll only be affected a little bit.\nAnd within a few round trip times across\nthe network, we stopped using a failed\nlink.\nAnd so this problem of links failing,\nbringing down the network just goes away.\nAll of the flows themselves from the\nnetwork interface at one side network\ninterface are\njust avoiding those failures as we go\nthrough the network.\nIt's like self-annealing. Exactly. Yes.\nSo I think Mark has mildly undersold this,\nbut maybe we should- Mark, come on, man.\nSo conventionally, when a link goes down\non a network, what happens is one side of\nthat link,\nthe switch at one side of that link, or\nmaybe both sides of that link notices.\nBut then it has to tell all of its\nneighbors that that link went down.\nAnd then they have to tell all of their\nneighbors that that link went down.\nAnd so you have a distributed systems\nproblem.\nIt's conventionally solved with a\ntechnology called BGP, Border Gateway\nProtocol,\nwhich is basically just like a gossip\nprotocol that allows one link over here to\neventually tell\nthis switch all the way over here, maybe\nthrough five or seven hops through the\nnetwork, then,\nhey, you can't get to this destination if\nyou take this link. You have to use these\nother links.\nThat's a distributed systems problem that\nhas a convergence time.\nWhat MRC has done is it has taken that and\nit has broken the need to coordinate.\nEvery endpoint\nindependently very quickly detects, hey, I\nshouldn't use that path and just stops\nusing it.\nAnd this is maybe counterintuitive because\nyou would think, oh, it's easier if I just\nhave some central authority that tells me\nthat this link is down and that central\nauthority can distribute that information.\nAnybody who's waiting for a website to\nupdate knows that. It's not going to work.\nRight. Central authorities are, generally\nspeaking, also known as single points of\nfailure.\nAnd so instead, what we've done here is we\nno longer have to wait for this whole\nconvergence process to occur, which can\ntake seconds or in the tail, tens of\nseconds.\nInstead, everyone within, generally\nspeaking, milliseconds notices and just\nstops using that link.\nSo this is a very big deal because\npreviously, you know, the link goes down\nand the whole job stops for a few seconds\nas we wait for kind of the network to\nstabilize.\nThat's time, again, that the GPUs aren't\ndoing useful work.\nAnd as you, again, scale up, you're going\nto have more and more and more of those\nindividual little seconds.\nAnd here now, what we've observed is that\nwe turned this on as the data center was\nbeing built.\nAs Mark said, we were able to get jobs up\nin training within months of hardware\narriving.\nThere's a lot of manual labor that goes\ninto building one of these buildings.\nThere's a lot of shared points where\nfibers from one data hall are coming in\nand technicians are trying to assemble\nanother data hall or things like that.\nAnd so what we saw was that because of all\nof this kind of manual effort that was\ngoing on, links were going up and down all\nthe time, like even way more often than\nyou would kind of hope just due to natural\nfailures. We did not care.\nWe didn't even notice. MRC just took care\nof it.\nIt just kind of would detect, hey, can't\nuse that path, move on to the next one.\nDidn't care. It was incredible.\nSo the other thing this gives you beyond\njust that is because once it's handling\nthat by itself, once MRC is already\nworking around the failures, traditionally\nwe would probably still have been running\na routing protocol in the network to find\nthe paths that actually work.\nBut routing protocols themselves are\ncomplicated and switches are complicated\nand switch software is complicated.\nAnd these are all things that can fail.\nAnd we realized that actually MRC itself\nwas able to figure out which paths were\nstill working.\nAnd so actually, we just decided that we\nwould turn off the routing protocols.\nWe use completely static routing in the\nswitches at the largest possible scale.\nAnd so some paths are broken. Who cares?\nMRC will find the broken ones that still\nwork and keep going.\nAnd that just removes a whole set of\ncomplexity out of our network management\nthat we just don't need anymore.\nWe don't care about whether the switch\ncontrol plane has converged because it\ndoesn't need to. It's entirely static.\nThey have a configuration that they have\nat boot time.\nThey boot up and they never change their\nrouting tables from then onwards.\nSo this is a very big effort working with\na bunch of people.\nCare to talk about some of the partners?\nYeah, we've been working with Microsoft,\nwho build our Fairwater data centers for\nus.\nAnd then we've been working with NVIDIA,\nBroadcom, AMD, and Intel to standardize\nthis specification.\nAnd with all of these guys to actually\nbuild our hardware for our new\nsupercomputers.\nIt's interesting because as a user of this\ntechnology, and you listen to sort of the\nway we sort of think about things where\npeople talk about like, well, when's the\nnext model coming out?\nLike it's a software update that comes up\nevery year. But it's not.\nEvery model is essentially a research\nproject, and it's dependent upon what goes\non in the training.\nAnd you try to have an estimate of how\nlong it's going to take, and we try to\npredict where that's going to be.\nBut this kind of reliability sounds like\nit's going to be an incredible advantage.\nGod, yeah. I would say from talking to the\npeople who\nwere here even earlier, I mean, you hear\njust horror stories about what it was\nlike, how often they were getting woken\nup.\nI remember walking in the cafeteria and\nseeing some of the people working on\nnetworking and sort of sad faces because\nthey didn't know why something had stopped\nin a run. And it was just. Yeah.\nWe have heard nothing but universally\npositive feedback about how stable the\nclusters with MRC are, how well they're\nworking, how basically the researchers\ndon't have to think about this anymore.\nAnd then you look at the statistics and\nyou realize just how much stuff is\nbreaking all the time and they're not\nnoticing. Yeah.\nTo some degree, this is the ideal, right?\nI know I said earlier, we are pushing the\nlimits of infrastructure.\nAnd so there's never going to be a time\nthat you can just completely ignore infra.\nyou know the ideal world right is\nresearchers don't have to think about this\nbut that's never\ngoing to be the case but every time we we\nknow we've won when researchers stop\nsaying you know\nnetwork or stops you know stop needing to\nknow what network protocol this particular\ncluster is using\nand yeah it's been it's been really nice\nto basically be able to focus all of\nbecause there's\ndon't get me wrong there's plenty of other\nthings that break there's plenty of other\nwork to do\nbut this has really allowed us to remove\none of the key barriers to continuing to\nscale\nand to being able to deliver newer and\nbetter models on a much, you know,\nwe are trying to scale everywhere,\nincluding our velocity.\nAnd you have all decided to make this open\nto everybody to use.\nYeah, so the specification is due out\nthrough OCP as an open standard.\nAnd we've, as you say, decided to open\nthis up for everybody to use.\nWe're big believers in open standards and\nopen source.\nWe're building all of our networks on top\nof Ethernet, which is selfless and open\nstandard.\nAnd we benefit when the industry has\nvelocity, when the industry can keep up\nwith the things that we're trying to do on\nthe challenging side of things.\nAnd so it's in everybody's interest if\nwe're all trying to actually deploy what\nwe think are the best solutions in this\nspace.\nThere's no shortage of coverage of the\nscale of the AI build out.\nI think on a personal level, I think it\nwould be a real shame if that supply chain\nwas fractured.\nYou have people investing in totally\ndifferent technologies and underlying\nhardware just because they're trying to\nget some small advantage.\nI'm really excited that this is going to\nbe an open standard.\nI think it will really benefit other\npeople outside of OpenAI.\nIt also benefits all of us if we are kind\nof all pushing in the same direction.\nInfrastructure is kind of this shared fate\nof the whole industry.\nAnd I think it is a very good thing that\nwe are open sourcing this and kind of\nbringing everyone along.\nIt seems like it's beneficial, too,\nbecause everything is becoming very\ncollaborative.\nAnd even, you know, you take a project\nlike Stargate, which is multiple\nlocations, many partners across the world,\nand Microsoft, Fairwater, and sort of this\nidea that compute is a thing that there's\nnever going to be enough of it.\nand the more we kind of work with each\nother\nto figure out how to maximize it and\ncontinue doing it,\nprobably the better it's going to be\nfor everybody involved than treating it\nlike,\nI mean, it is a very limited resource, but\nthese protocols, like you said before,\nyou know, with Ethernet, whatnot,\nthat's really what gave us, you know, what\nwe have and things like the World Wide Web\nand a lot of the cool things\nwe realize, oh, share this because what\nwe're going to benefit\nis going to be so much better. Yeah, what\nwe're trying to do is hard enough\nwithout everybody having to reinvent the\nwheel all the time. We think this is,\nthe right way to go and we'd like\neverybody else to go in the same direction\nas us.\nWhere are the limits of this?\nMRC is a flexible standard.\nIt builds on top of Ethernet.\nSo as Ethernet scales, so will MRC.\nYou can think of Ethernet as kind of the\nprotocol that individual devices use to\ntalk to each other.\nMRC sits on top of that.\nIt incorporates kind of this static\nrouting that Mark is talking about.\nIt incorporates what we call congestion\ncontrol, which is basically if we do end\nup in situations because of failed links\nor choices that we make about how we send\ntraffic, how the endpoints should react to\nthat to make sure that we kind of use the\nnetwork fairly and efficiently.\nMy experience with networking is that\nthere will always be more work to do.\nThere's always going to be ways we can\nimprove that, make the network more fair.\nThere's fundamental limits on networks.\nSpecifically, the speed of light is a\nknown speed limit.\nAnd so the amount of time that it takes\nfor light to get from one point to another\nin a network has some lower limits on it.\nBut we're going to keep making each of\nthose links faster and faster.\nAnd so that will always kind of change the\noperating point of basically how much data\nyou have outstanding per connection at a\ngiven time.\nAnd that will always require kind of\nongoing engineering effort to make sure\nthat we're making the best use of the\nhardware that we have in a given\ngeneration.\nBut I think MRC gives us a very flexible\nand strong base to build on as we continue\nto push through the next few generations.\nAnd the key thing is because, as you say,\nit's Ethernet-based.\nI mean, Ethernet now is not what Ethernet\nwas 10 years ago or 20 years ago or 30\nyears ago or even 40 years ago.\nEthernet itself has evolved so much over\nthe years.\nAnd what we're doing is we're taking\nadvantage of all that development by the\nwhole world's networking industries.\nAnd so we want to make sure we carry on\nriding that wave of innovation that has\ntaken Ethernet\nforward. So given that all of that's\nhappening anyway, MLC, because it takes\nthe intelligence\npushed at the edge of the network, we can\nscale the cores of our networks as long as\nEthernet\nkeeps scaling. And there's no particularly\nobvious reason why that's not going to\nkeep scaling for\nat least the near future. And who knows,\nmaybe smarter people than me will figure\nout how to\nmake it work for another 40 years. One of\nthe key things we're doing, though, is\ntrying to actually\nmove the complexity out of the network. So\nas I mentioned, we turn off the routing\nand each\npacket is actually source-routed through\nthe network. We're using a technique\ncalled IPv6\nsegment routing, which allows each\nindividual packet's address to list the\nprecise set of\nswitches the packet goes through as it\ngoes through the network. And that means\nthat the\nswitches themselves can be really dumb.\nIt's really nice to be able to sort of\nsimplify a\nof the network. And as you're trying to\nscale and make things scale reliably,\nmaking the middle of the network as simple\nas possible has huge benefits to us.\nthe network as simple as possible has huge\nbenefits to us. The other piece here is,\nyou mentioned\nEthernet has this rich, wonderful history.\nWe're still building on Ethernet. That's\nbecause it's\nan open standard. That's because the\nentire industry has bought it and is\npushing in the\nsame direction. And that's exactly what we\nwant to see from MRC also, is we want to\nsee basically\nthe next layer get ready for the systems\nchallenges of AI and get widely adopted.\nWe don't think that this would do as well\nif it was an OpenAI exclusive.\nWith OpenAI investing the time and energy\nand the people into solving these\nproblems,\nit does seem like even though you're\nmaking MRC available to everybody by\nhaving teams now working on this,\nit's got to be providing a lot of benefits\nand advantage.\nI try to get a lot of benefits. So again,\nmy role is from the perspective\nof like, how do we make the most use of\nwhat we have?\nNot literally in a conservationist sense,\nbut in some sense it is, right?\nLike if we're going to be burning the\npower on these things, we want it to be\nused productively\nand we want it to be used efficiently.\nAnother one of the advantages of MRC is\nbecause it has this property of allowing\nus to spray\nover multiple paths, we can build much\nsimpler, much smaller networks that have\nmany fewer devices in it.\nSo this is not an obvious property, but\nbasically we're able to build networks\nthat are much\nflatter and basically have many fewer\nlayers of switches and use much less\npower. They also cost\na lot less. And the amount of useful work\nyou can do per watt goes up when you do\nthat because you're\nnot spending extra power on these extra\nlayers of switches. The power is more\ngenerally going\ndirectly to the GPUs and allowing them to\nactually do work.\nWhen you train models and start off with\ntext models and with certain context\nwindows, so there's a certain amount of\ndata that needed to go\nabout. And then as you go into multimodal\nmodels, is there a difference between\ntraining, let's say,\nan image model or a video model\nor just a multimodal model?\nI probably can't get into the details of\nmodel architecture, but I will say that as\nour models get more advanced,\nthe demands that they place on the systems\nget substantially harder. The amount of\ndata that\nyou have to move and the latency bound on\ndoing that.\nSo basically, the regret that you have if\nyour network goes a little bit too slowly\njust\ngets worse and worse and worse as the\ntraining cluster size gets bigger and as\nyou continue\nto optimize the rest of the stack.\nBecause obviously we have many, many very\nsmart people who are trying to push in the\nsame direction and make these models very,\nvery efficient.\nWe all want the next version of ChatGPT\nfaster. We want it to be smarter.\nOne of the big strengths of OpenAI is that\nwe have all of these incredibly smart\npeople pushing in one direction.\nAnd you have the researchers and you have\nthe infrastructure people and everyone\nkind of knows what our goal is.\nAnd so we have very smart people who are\ntrying to make the work that happens on\nthe GPU go faster.\nAnd all that means is that now we have a\ntighter bound on how long the network has\nto transfer things.\nBecause if we are the ones who are lagging\nbehind, their work doesn't matter.\nAnd so our work never stops.\nAs these things get bigger, the variance\nin whatever the slowest computation is\ngoes up.\nWe have to work on controlling that tail.\nSimilarly, on the network side of things,\nyou need to pull things in.\nOne piece that when Mark was talking about\nthis splitting over many network links,\none thing that's mildly not obvious about\nthis is that if you did this without MRC,\nthe tail statistics actually get much\nworse.\nSo if you take the same amount of\nbandwidth\nand have more paths,\nyou end up basically having worse tail\nstatistics\nbecause you're throwing the same number of\nballs into more bins.\nThe ratio of the worst bin to the average\nbin gets much worse.\nAnd so that deterministic routing that\nMark talked about\nwhere we can do this kind of very careful\nload balancing\nacross these very huge number of links is\nvery important to avoid getting into a bad\nsituation\nthat we would then have to kind of have a\nfeedback loop to recover from.\nAnd so kind of all of these pieces of the\nstack are extremely tightly coupled\ntogether, right?\nIt is extremely important that we have\nkind of low-level network hardware people\nwho have\nsome understanding of what is happening at\nthe workload layer.\nAnd we have people at the workload layer\nwho have some understanding of what the\nheck is\ngoing on inside a network switch.\nWe really wouldn't be able to push the\nboundaries of how big\nyou can make these systems without that\nkind of vertical\nintegration and everyone pushing in the\nsame direction.\nWhen people talk about data centers--\nand you mentioned this earlier--\nthere's what you use for training, there's\nwhat you use for inference.\nWhen I asked ChatGPT, I don't need to talk\nto the whole data center, I\nneed a few GPUs who are going to answer\nthis question.\nAnd there's been talk about, well, the\nnext step\nis we're going to put things into space.\nAnd my question's always been like,\nI can get-- like, I have a satellite with\nsome GPUs. It's doing inference.\nBut when you're literally spreading things\nout across thousands or hundreds of\nthousands of miles on a big network, it\nseems like you lose all of your advantage\nfor speed.\nThe speed of light is your enemy versus\nbeing in one center.\nIt's hard to envisage doing the sort of\ntraining that we do in our Stargate data\ncenters in space.\nJust the latency would be a huge problem.\nAnd just the background rate of failures\nwould be a problem.\nWe have technicians from Microsoft and\nOracle who have to go in and fix things\nall the time every day. Hard to do that in\norbit.\nYeah. Yeah. I think you can make all sorts\nof arguments. And I have\ngotten, I've gone very deep on this. As I\nsaid, I have a physics background. I'm\nvery interested in,\nyou know, I worked with people who\ndesigned satellites. You know...\nthings like that. Yeah. But I think a lot\nof smart people have had very reasonable\narguments both\nways on that dimension. The major\nbarriers, I think, are the rate of\nfailure. I mean,\nevery GPU, every generation of GPU, the\nGPU itself gets more powerful and more\nexpensive\nand more important. I think we are doing\nincredible work here on earth to try to\nkind\nof route around failures automatically but\ni think that you would find yourself with\na lot of hardware\nthat you couldn't use very quickly if\nyou're shipping these things into space\nnow is there a\nworld in which you can also put\ntechnicians in space maybe you can do all\nsorts of things the\ndreamer side of me says that would be\nreally really cool yeah the practical side\nof me says it's\nreally really hard to do this stuff on\nearth yeah like every day we are trying to\npush limits on all\nsorts of dimensions even just spinning up\nMRC was a huge effort that required very\nclose\ncollaboration with us and engineers at a\nnumber of other companies and it required\nin some cases\nyou know hands-on machines to fix things\nto test things etc these systems are hard\nenough to build\nand make work and make perform here on\nearth i think trying to push the\nboundaries of that and\nalso adding additional complications, you\nhave to make a really strong case for why\nit makes sense to do it in space.\nSo build more terrestrial\ncompute centers.\nPlease. I mean, that is what we are trying\nto do here, is build a lot of compute so\nthat we can increase the net amount of\nintelligence in the world.\nThat's awesome. Gentlemen, thank you very\nmuch.",
  "transcript_chars": 40514,
  "ingested_at": "2026-05-15T10:38:17.780670+00:00",
  "source": "channel",
  "yt_meta": {
    "view_count": 16604,
    "like_count": 510,
    "channel_id": "UCXZCJLdBC09xxGZ6gcdrc6A",
    "categories": [
      "Science & Technology"
    ],
    "tags": []
  }
}