{
  "video_id": "PJGIWDW_W2A",
  "channel_slug": "ibmtechnology",
  "channel_handle": "IBM Technology",
  "title": "Is open source safe? Featuring Mixture of Experts",
  "duration_seconds": 1523,
  "url": "https://www.youtube.com/watch?v=PJGIWDW_W2A",
  "upload_date": "20260429",
  "transcript": "Hello and welcome to a very special episode of Security Intelligence. I'm your host, Matt\nKosinski, and today we have a crossover episode with our sister show Mixture of Experts, IBM's AI\npodcast. Repping MoE. Today we\nhave Martin Keen, Master Inventor, and Gabe Goodhart, Chief Architect, AI Open Innovation. And in the\nSecurity Intelligence corner, folks, it's Jeff Crume, Distinguished Engineer, Master Inventor, Data and\nAI Security. And we're doing this little crossover today because we want to tackle questions about\nopen source. How do we enjoy the unique benefits of open source in AI and elsewhere while\nnavigating some of its unique risks? Gabe, I will start with you. Tell us how you feel about open\nsource. I've got to be the person you're referencing on Mixture of Experts, that's all in\nall the time on open source. That's that's where I live. It's what I do. So come at me. Let's see what\nwe got here. I appreciate that. Martin, how about you? Where do you land on the scale from open\nsource is great to open source is terrible. Also a big open source fan, and Gabe, you know, fair warning\nwhen it comes to cybersecurity people like Jeff here, no matter what you tell them you're doing\nand how good it is and useful it is, he will find a way to tell you that this thing is dangerous, it's\nreckless, and that they've solved this problem 30 years ago, and nobody's listening to them. And\nyou're welcome. I have no doubt I can't wait. Jeff, is that what you're gonna tell us? Where do you\nfall on open source? Well, I almost feel like I should, because we'd get better ratings if we had\na food fight here. If we all just, you know, join hands and sing Kumbaya, I don't know that\nanybody's going to care. So I'll take, cybersecurity people are contrarians by nature. We\nhave to be. I mean, we're always looking at, everybody else is looking at how a system will\nwork. Our job is to consider how it will fail and then try to prevent that from happening. So in\ngeneral. Believe it or not. Now I'll say this and then we'll just wipe this off of the recording.\nI'm actually in favor of open source. What I'm not in favor of is some of the over\ngrandiose kind of claims that are made for it when it comes to cybersecurity. So that's\nwhat I would tap the brakes on. That makes perfect sense to me. And let's circle\nback around then to Gabe, because Gabe, you're our kind of probably strongest soldier in open source's\ncorner. We all kind of dig it. But you've said, look, I'm all for it. What are the benefits? Why does\nit why are you such a champion for it? Tell us a little bit about that. In this era of AI, we have\nnever been closer between the science and things that are genuinely usable. And I think that is\nhappening in large part because of open source and because the science itself, the math, the\ntensors are actually remarkably close to something that's usable, as opposed to needing a\nwhole lot of layers between those and something that people can get utility out of. And because of\nthat, I think open source, relative to most other innovation waves, is where the\nvast majority of the actual innovation is happening because science, by its very nature, is\nopen and because the science is so close to utility, it's actually very tightly coupled with\nthe source code that's generating utility. So I think that's why open source is such a key\ndriving force in the AI revolution we're sitting in today. Now, Jeff, 100% agree open source\ndone wrong is terrible. And there are so many projects out there. In fact, a close\ncollaborator of mine is working in a project that he just moved over to, and having come from a\nvery principled open source project that managed security contributions, trustworthiness\nvery carefully into this new project, he's telling me about all the smells he's got and all\nof the vibe code spidey sense going off right and left, that this is not a secure project, so\nI believe strongly that open source can and should be a force for good in this world of AI,\nbut it is not a carte blanche solution to everything. So. And we're having this conversation\nabout open source at a time when there's a lot of debate around questions of who gets access to\nwhat models and when. Right. Martin, I'm wondering how you see open source fitting into this\nquestion about model access and who gets access to what. Do you think that's a kind of guiding\nlight for us in terms of who can see things and when, I don't know. What are your thoughts there?\nRight. We sort of got this comparison between the frontier models that are released by labs that\nare closed models, and up until now they've been quite democratized, that they're available to\neverybody. You sign up, you pay $20 a month, you get to use the latest, smartest model. And we're\njust starting to see now with models that are coming out that that's no longer the case, that\nyou have to be on the list. You have to be either in the consortium of companies or you have to\nbe, you know, at least approved by the lab to use it. So that, of course, is not the case at all with\nan open source model that you can find on Hugging Face or wherever it is that you take a model out\nand you run it on whatever hardware you want to run it on. So we're actually seeing, I think, a\nbigger and bigger divergence now with some of these frontier models are really super advanced,\nbut also very locked down as to who has access to them. But, you know, if history is anything to go by,\nthe open source models, they will catch up to where these frontier models are today pretty\nquickly. And they're not going to have those same restrictions on them. So it's going to be a really\ninteresting time for these models. Jeff, do you have any concerns on your end when it comes to\nmodel access, in the sense that the bad guys can get their hands on this stuff, maybe just as soon\nas the good guys can, and with open source models, maybe they can see even more of it. Is that\nsomething that you ever get worried about, or how do you feel there? It's a thing for sure. There's a\npart of me, by the way, even though I'm a cybersecurity guy. I also have this side\nhustle of being an adjunct professor at NC State University. Well, the academic in me, which is just\na small part, I'll have to admit, but that part is yelling out information wants to be free, so we\ndon't want to keep this stuff in the hands of just a few. The cybersecurity side of me also is\nreminding myself that security through obscurity is not an effective model either. So the idea that\nI'm going to keep secret information and that's going to make my system more secure is not going\nto work. You know, we have this idea in cryptography that's called Kerckhoffs' principle,\nwhere it basically says that the only thing about a cryptosystem that should be secret is the keys.\nIn other words, the algorithms, the generation of keys, you know, all of the where they're stored. All\nnone of that should be a secret. The only thing that should be a secret are the keys themselves.\nBecause if the algorithm or the cipher itself is something that's secret, well that\nprobably just means not enough people have looked at it and really taken it through its paces, and\ntherefore it's got vulnerabilities in it that we just haven't shaken out. So on the one hand, the\nargument that putting this stuff out there, the argument that a thousand eyes means that\nwe've got a thousand eyes out there inspecting all of this and they'll find where the problems\nare. That sounds good in theory. It's a little naive when we start talking about systems that\nare as large as the ones we're talking about, because a thousand eyes wouldn't be nearly enough\nto look at a billion parameters. So, you know, the idea that we can, just by making it\nopen source, that it becomes more secure. That's the part where I push back. I remember a debate I\nhad with a colleague 25 years ago. He was arguing that Linux was, by definition, more secure than any\nother operating system because it was open source and because of the way the kernel was designed,\nyou couldn't have malware on it. Well, how did that age? I mean, we've had countless examples of\nmalware in Linux. Linux is a good example of a system that is securable, but in\nand of itself is not necessarily secure. And the same thing, the same lessons can be applied to AI.\nThey can be securable if we have more people that know about it. But just because everybody\nknows about it is no guarantee. Because it's too big, it's scaled too large for enough competent\neyes to really analyze all these things. But the bottom line is trying to keep the information\nsecret. Eventually it leaks out. Eventually it always leaks out. So the idea that we'll keep it\njust for us, you know, we'll just keep a secret. The four of us will know this and we'll not tell\nanybody else, but somebody's going to have loose lips. Somebody is going to have it on their system,\nand somebody's going to hack into their system, and then they're going to get a copy anyway. So\nthat can't be what we're leaning on when it comes to security is what I would say. A\nthousand eyes being on a thing doesn't necessarily mean that thing is like totally free\nof vulnerabilities and we found every vulnerability. But then my question becomes, you\nknow, the thousand eyes might not make sure that every vulnerability is closed, but it does also\nlend itself to, like Gabe said, innovation. Right. And I was wondering, Gabe, if you could talk about\nthat angle of things in terms of, okay, when you have more hands on a model, we can do more\nthings with it. Could you walk us through some of that a little bit? Yeah, absolutely.\nSo before we dive into what more hands on a model mean, I want to just draw a very important\ndistinction that I think often gets missed in this conversation. It's the open source of the\ncode that implements the systems that we are all relying on today. And then it's the open weights\nof the models and the architectures of those models as defined by their configurations. Now\nthose things are coupled because those architectures and weights have to get loaded up\nby software and actually executed somewhere. But in and of itself, the weights of the models being\nopen, has a whole different set of pluses and minuses than the software itself\nbeing open. So I want to focus on the software for a minute, because fundamentally, this is no\ndifferent than exactly what you're saying about Linux, right? It is a securable ecosystem. It is not\nsecure by default. Right. And the same thing with Kubernetes, with literally any other large\nsoftware system, that's a composition of a bunch of different open source projects.\nEveryone has its attack surface. An intelligent implementation of that system needs to have those\nattack surfaces addressed meticulously, and it needs to have a good policy for staying up to\ndate when new vulnerabilities are discovered. So in that sense, the AI stack is just software.\nNow the models themselves pose an interesting different challenge. So I think one of the big\nsecurity arguments in favor of closed source models is that the software and the model are\nboth essentially inaccessible without whatever software layer is sitting in front of that, and\nthat allows the authors of that system to not only manage the vulnerabilities of the software\nitself, but manage the potential risks that the model's trained capabilities would expose. Because\nthey can put guardrails in front of it, they can put guardrails on the back half, they can detect\nusage patterns that would indicate malfeasance. They can do whatever they want at that software\nlayer to try to mitigate the harm that the model's capabilities would have. Once those model\nweights are out in the open, even if the model has trained into it some rejection capabilities, the\nfirst thing that everyone on the internet is going to do is obliterate that model and take all\nthose rejection capabilities out, right? Everyone's gotten really smart about figuring out, okay, what\nis the right vector that I have to, you know, tweak on in the embedding space to just crank out those\nrejection samples. And now I've got a free-for-all model that can do whatever I want it to. So model\nweights in the open do have that inherent danger that somebody will get at whatever is underneath,\nlike the base training level, the sort of unfiltered training. They can remove whatever,\nyou know, post-training has been done on those models pretty effectively. And so that's certainly\na risk. So I think, you know, that doesn't mitigate the fact that this will get out there\nand that the science and the innovation still wants to push forward, because the innovation\nfront of it is that once somebody puts a model out there, it's not just the weights, it's the\narchitecture, and it's the software that implements that architecture. We've seen this in a\nrecent round of innovation around the attention mechanism and using linear attention. We've got\nthese great new models coming out that scale much better with long context. And that's because this\nwas all done with open science. And people piggybacked off each other and thought, hey, how\ncan I take this interesting idea of a recurrent linear layer. And I can hybridize that with some\nattention layers. And oh, maybe I could even tweak on that linear layer to use a different, you know,\nbunch of matrix math. So all of this is building on itself with the innovation front. So it's hard.\nI don't think you can have both at the same time. Release how the model works, and also don't run\nthe risk of those models becoming obliterated and people understanding how they work. So again,\nit's sort of it's not a one size fits all with the open weights models, because we certainly need\nto then mitigate against those threats proactively that the model capabilities, you know,\nexpose. And Martin, I saw you kind of nodding along there. Did you want to jump in there. You want to\nadd what's your thoughts here? When we think of open source and AI, we're often thinking of, oh,\nopen source AI models, you know, Mistral or Llama or DeepSeek or something like that, those open\nweight models. But as Gabe says, there's, you know, these are built on the foundation of\nsoftware and existing projects that are already open source, but I think you could even go one\nstep further than that. And really, the argument here is not should you adopt open source models\nversus the closed source foundation models from the frontier labs? Because there's an extra\nlayer here, which is to say that the models themselves are working with all sorts of open\nsource capabilities as well. So if we think about MCP, for example, Model Context Protocol, that\nallows a model to go out and basically invoke another service, and that came from a\nfrontier lab. That was Anthropic, that came out with that and then donated that to the Linux\nFoundation. And that means that today I can take one of my open source models and I can use MCP,\nand I can go and invoke whatever it is I want to invoke in an MCP server. But I could also use a\nclosed source model and do that exact same thing as well. Same with skills.md. So there's an agent\nskill spec that is now an open standard that defines how these things are actually defined. How\nyou tell an agent how to do something. And again, that is an open style that works with open weight\nmodels and also the foundation models as well. So there really is no getting away from the fact\nthat open source is foundational to everything in AI now, even if we're talking about models that\nwere actually frontier closed models as well. Now, speaking of the kind of nuances of open source,\nright. We've mentioned the ways it can be incredibly beneficial, the ways it can lend itself\nto innovation. And then we've also mentioned some of the risks that come along with that. And I want\nto shift our gears a little bit then to that side of things. How do we manage those risks? And Jeff, I\nwant to start with you, and I specifically want to pose you the question because I know you've\ntalked about it before. Is this a place where AI can help us secure our AI, and if so, how? What are\nyour thoughts there? Yeah. So you're talking about nuance. And then you asked the guy with\nsledgehammer to come in and talk about nuance. So all right, let me go. This is one of those things\nactually. It is a very nuanced question. And the people that are all 100% this is the way to go or\n100% that's the way to go are probably missing a big part of the argument here, because it's not as\nsimple as that. I'll say I'm never going to be in favor of a proprietary thing, saying that it's\nmore secure because it's proprietary. What I'm going to say is the reverse is not necessarily\ntrue, that it's more secure because it was open source. So it's more secure because it was\ndesigned in a secure way, and you put the right kind of controls in place, and hopefully open\nsource will allow us to discover those kinds of things. Going back to the model discussion. Look,\none of the big things in security has always been this word, this concept of trust. You know, can I\ntrust the system? Can I trust that you're who you claim to be when you're trying to do this\ntransaction? Can I trust the stuff that I wrote to the database is still the same thing that I wrote\nto it a day ago, or somebody has changed it? Can I trust that someone hasn't made off with all of\nour confidential information? One of the things about trust when you're talking about trustworthy\nAI is transparency. How can I trust something that's not transparent? If I don't know what's\ngoing into it, then that's not trust. That's just blind faith. And that's not a technical, you\nknow, pillar of trust is blind faith. You know, I think because it's been exposed to sunlight,\ntherefore surely some smart people must have found all the problems. Again, when we look at some\nof these bugs that have been latent in open source code that have been around for decades.\nMeans all the smart people in the world still didn't find that. But guess what's going to be\nable to find it? AI because AI will be able to look at that source code and be able to\nidentify these things. But I'm going to say, even though that's kind of top of mind because of\nwhat's happening in the news right now, I'm going to say this is just a variation on a theme. We've\nalready had this issue, if you want to call it that. Previously you could give me just the\nexecutable code with no source code, the most proprietary thing that's ever happened. And I run\nit through a decompiler, and it's going to give me back machine language code. And then if I want to,\nI could find vulnerabilities in that. Or I could use an LLM to translate that machine code\ninto original higher level language and its equivalent. And then I run that through a\nvulnerability scanner. So even if you only, in other words, it's not just because the source code\nis hidden, that it's secure because I can essentially define what the source code would\nhave been. I can reverse engineer the source code, maybe not line per line, but function by function\nwhat it does, and that has existed, that kind of capability has existed before we were even\ntalking about generative AI. So this just puts it on steroids. It just amps up what that is.\nBut for a lot of people, I think they've come to the party late and think this is a brand new\nthreat that we've never seen before. No, we've seen it before. Now it's just a little bit different.\nSee, I knew you'd say that, Jeff, that this was a problem that was solved years ago. We just weren't\nlistening. Of course. Of course. If you all would just listen to me. Yeah. All the cybersecurity\nissues in the world would be fixed if we all just listened to Jeff Crume. We all know this.\nGabe, I want to swing around to you now and kind of ask you, you know, for your thoughts on\neither what Jeff just said. I saw you just raised your hand. So I don't know if you have something\nyou want to jump in with, but go ahead. What's your thoughts here, Gabe? Sorry, I'm dealing with a\nbunch of internet lags, so I had to raise my hand to jump in the conversation here. I thought he was\njust waving. I really want to tie together two things that Martin and Jeff said, because I think\nthere was actually a really interesting point that you guys brought up in combination, which is,\nJeff, you mentioned that everything is all about trust and where that trust boundary is. And Martin,\nyou mentioned the fact that there's actually a whole lot going into these model systems, even\nbeyond the open source software that's running them and the open weights or closed weights models\nthat are actually doing the math, and that is the context. And I think this is something that, you\nknow, all of the big ability to reverse engineer binaries is great. It's on steroids. Okay. We got to\nbe careful of that. But what I am even more scared of is that there is a view where, especially where\nyou have agents operating with autonomy, the agent loop is essentially a code interpreter, and the\ncode is literally any text you pass through it. And so we're all very familiar with this concept\nof, you know, having to sandbox your untrusted code. Well, now the internet is your untrusted code, like\nliterally anything on the internet can instruct your interpreter, which is your agent loop, to go\ndo whatever it can. And depending on what tools that agent loop has access to, there is a lot of\ndamage that can be done. And so I think this is a net new vulnerability that\nis very ripe for the security eye about that trust lens, right? Because I think people are used to\nsort of implicitly trusting textual, unstructured data because it's not actually ever getting\ndirectly executed. It's being filtered through some kind of mechanism that's either going to\nprogrammatically extract signal or manually, you know, it's going to go to eyeballs, and eyeballs\nare going to extract that signal. It's going to be a human brain. But when it's actually triggering\nthe reasoning that then takes action in an agent, that becomes a huge attack surface. And we've\nseen that immediately with OpenClaw and other like all bets are off nonsecure\nsystems. And again I will point out that OpenClaw is a securable system. But it's very not\nsecure by default. And there are already, everybody and their\nuncle is trying to claim that they have the secure implementation of OpenClaw. And most of\nthose things are focusing on some of the security problems like sandboxing the execution, managing\nthe tool permissions, and curating a trusted set of skills. But what I haven't seen anybody talk\nabout is managing the wild world of untrusted data that's going to get slurped in at runtime,\nand what that's going to actually do due to prompt injection. So if anybody can solve that problem, I\nam all ears, because that's the one that keeps me up at night. Let me get this straight, Gabe, are you\nsuggesting that not everything on the internet is true? Because I'm taking notes.\nI'm absolutely. All right. All right. I learned something today. In fact, some things are\ndeliberately untrue. Oh, I would have never considered that. And look, if we knew how to solve\nthat, this would be the most amazing podcast ever published. Everyone would love us because we\nsolved the prompt injection problem. No one has solved that yet. Martin, though, I want to move on\nto you and just get your thoughts as we start wrapping up here in terms of all right. So we want\nto move forward with our open source in AI, whatever that might look like. We want to use it\nresponsibly. But what needs to be top of mind as we're doing that. Like what do we need to be\nthinking about to make sure we're getting this right? What were your thoughts there? Yeah. So in\naddition to the model itself, in addition to the context problem, which Gabe is talking about there,\nwe've also got the problem of interpretability with these models that a model is made\navailable, but large language models are generally not interpretable in the same way that other open\nsource software is. So as Jeff points to, we can reverse engineer stuff. We can go look\nat the machine code that was generated from a particular open source project, figure out what\nit's doing, why it's doing it. We cannot do that with large language models because there are so\nmany connections, all of these model weights, that there is no good way yet to interpret how a model\nis going to take an input and come out with its output. So how do we know that we can actually\ntrust a given model when we can't actually go in there and look to see how it's made? So I think\nthat just adds an extra thing to be a little bit concerned about. But that is a concern\nthat applies to open weight models and to the closed ones as well, because even those foundation\nmodels, there are labs doing all sorts of interpretability studies on that stuff, but nobody\nreally knows how these things actually fundamentally work. Just a little plug for another\none of our friends on Mixture of Experts. I actually, Chris Hay has been pinging me on the\nside with his wild, crazy ideas and he has actually figured out how to unbox some of these\nopen weights models and treat them as a knowledge graph database and actually get that\nexplainability out of them, which I think is really cool. So that is a bit of a point in the\ncolumn of open weights models is that you actually can start unboxing what those weight\nmatrices are doing and what impact they have on the final output, because you can tweak and poke\nand turn and change a little weight here and there and see what the change is, and you can\nactually get quite a bit more interpretability out of that box. But of course, that comes at the\ncost that somebody could also go in and do exactly that same process and make them do much\nworse things. So it's a double-edged sword there. I expect you to fix the hallucination\nproblem then in the next couple of weeks. Now that you have this capability. Oh boy. No pressure, no\npressure at all. Folks, I wish I really wish we could keep going. But unfortunately we are out\nof time for today. Obviously, there's so much to say on this subject, and I just hope we kind of\ngot across to folks. But I want to thank our panelists, Gabe and Martin and Jeff, for being here.\nI want to thank the viewers and the listeners. Thank you to our producers. Subscribe to Security\nIntelligence and Mixture of Experts wherever podcasts are found, and stay safe out there.",
  "transcript_chars": 26454,
  "ingested_at": "2026-05-21T19:11:18.001305+00:00",
  "source": "retry-no-transcript",
  "yt_meta": {
    "view_count": 3519,
    "like_count": 138,
    "channel_id": "UCKWaEZ-_VweaEx1j62do_vQ"
  }
}