{
  "video_id": "Y8h43EEdOrA",
  "channel_slug": "deeplearningai",
  "channel_handle": "DeepLearningAI",
  "title": "AI Dev 26 x SF: Jean-Marie John-Mathews: Red Teaming LLM Applications   Systematically",
  "duration_seconds": 856,
  "url": "https://www.youtube.com/watch?v=Y8h43EEdOrA",
  "upload_date": "20260520",
  "transcript": "So, today we're going to talk about red\nteaming and how to red team AI systems.\nSo, to start with, um for this\npresentation and to understand a bit\nmore why we red team, I just looked at\non LinkedIn some scandal or bad buzz of\nAI system failure. And actually last\nweek\nI saw this this post on LinkedIn.\nInteresting one.\nIt's about Chipotle, which is a famous\nfood brand, Mexican food brand. They\nreleased a chatbot.\nAnd as you see here\nyou have some people trying to to make\nthe chatbot say something about coding,\nnothing to have with Mexican burrito.\nAnd this the this the chatbot just say\nsomething and as you see here\nthis is like a bad buzz on AI systems.\nAnd actually if you look at the the the\nremark\nand the reaction, it works quite well.\nSo, this is a kind of things actually at\nthis car we work on.\nUm not only on reputational issues\nuh with this kind of off-topic issues\nwith chatbot but also different\nvulnerabilities of AI chatbots.\nSo, these are the risk we're working on.\nAs you see here, you have two dimension.\nOn your left hand, you have intentional\nand sophisticated attacks.\nThose are mostly like like big prompt\ninjection with various actors. It can be\nnation state actors, cybercriminals, and\nso on.\nAnd on the right side, you have more\nlike legit questions\nfor hallucination,\nlike\nlegit queries that you may have from\nyour customer.\nAnd here the failure are different. It's\nmost mostly like hallucinations,\nperformance issues,\nbut it can also be like off-topic\nsituation.\nSo, the goal when you do red teaming,\nactually, is to be able to detect those\nissue offline. So, the goal is really to\nmake sure those stuff\nuh are not discovered in production.\nSo, the term AI red teaming means we're\ngoing to implement a structured testing\neffort to find those flaws.\nAnd actually, usually, people say there\nare like two things, the red teaming,\nwhich is about detecting issues,\nand the blue teaming, which is about\ncorrecting the issues you found.\nSo, as you see here, you have many\ndifferent forms of issues. Of course,\nyou have the prompt injection,\njailbreaks, but also robustness, uh\nhallucination, information disclosure,\nespecially when you do like data\nleakage.\nAnd this is the kind of thing we try to\ndiscover.\nSo, the goal of my presentation will be\nto to show you a bit the way to to\ndiscover those stuff.\nSo, let's start.\nUh\nwhy, actually,\nLLM testing\nis becoming harder and harder.\nSo, you have a general framework to do\ntesting, uh like the LLM as a judge\nframework, which is\nyou going to\num\nput a query, an input, the agents will\nhave an output from this input,\nand then you're going to judge. You're\ngoing to judge this output\nwith the LLM as a judge\nin a context, and the context will will\num\nuh integrate the inputs.\nSo, this is a standard framework.\nBut today, with agents, it's becoming\nharder and harder to implement this\nframework. Why? Because today, we need\nto evaluate the whole dynamic of the\ninteraction, not only the input and the\noutput.\nSo, we have like a bunch of examples to\nshow why this framework is not\nvery well anymore.\nSo, first, you may have like the right\noutput with the wrong reasoning. It\nhappens a lot when you work with AI\nagents. Like, let's say you you you work\nwith automation system with an ATN, for\ninstance.\nMany times you have the right output\nwith a wrong reasoning.\nUm\nOh, you have also tool calls that is\ninvisible. And usually, today the\nfailures are inside the tool callings.\nIt can be in the inputs of the tool\ncalls, but also in the outputs.\nSingle-turn blind inputs. So, you need\nto have like multi-turn attacks\nuh where you have like many different\ndynamic. I'm going to show you an\nexample of this kind of attack that\nworks.\nAnd then you have no user dynamic.\nToday, you cannot have a bunch of inputs\nthat you can put in a golden data set\nbecause these are too static. What you\nneed to do is to simulate user\nwith persona and with new intent. So,\nI'm going to show you some example of\ncases\num\nwhere this LLM as a judge this standard\napproach doesn't work.\nFirst example.\nLet's say you have a frustrated customer\nforced to rephrase three times.\nSo, a user is just\nasking a question, \"Where is my order?\"\nThe agent say, \"Okay,\nuh in order to answer this question,\nwhat is uh can you provide your order\nnumber?\"\nThe you the you user answer.\nAnd then the agent think, and the user\nsay, \"Okay.\"\nThe issue is, \"I'm very late. It's late.\nWhere is it?\" So, we see that the user\nis getting frustrated.\nUm and the agent is saying, \"I\nunderstand. Could you rephrase?\"\nSo, when you see this dynamic, you\nunderstand the issue. You see here um\nunhappy user\nand the LLM is just trying to es-\ninstead of escalating to a human, it's\ntrying to um rephrase, make the user\nrephrase.\nSo, this is a problem that would be hard\nactually to test in a simple LLM as a\njudge at Perch. We need to have a\num a view on the whole dynamic of the\nconversation. And this is also a very\ndomain-specific\nuh case to to look at.\nTake another example.\nYou have a query like find the accounts\nfor Mary Dupont and updated.\nThis is an action um\nwritten by the user.\nThe agent will call us the CRM tool\ninternally, and then it will output, \"I\nfind the accounts for Mary Dupont and\nupdated.\" The problem here is that you\nsee an output, you have no way, looking\nat the conversation, to understand that\nthere was there was an issue. And\nactually, there was an issue because, as\nyou see on your right side, the tool\ncalled inputs,\nyou see that the company name is\nmissing.\nSo, you need to have a whole view of the\nback end of the agent to make sure to to\nbe able to to evaluate it.\nSo, this is two standard example of why\nwe need to have more like\nframework that is a bit more complex and\nspecific to to agent.\nSo, at Just Catch, what we do actually\nis we work with different large\nenterprise like big banks to do this red\nteaming and write a report with all the\nflows of AI agents. It can be\nhallucination, it can be security flows.\nWe have a enterprise version, but also a\nopen source version. I want to focus\ntoday on the open source version.\nSo, how does it work?\nWe need the user to describe\nin a very\num\nin natural language the behavior, the\nwanted behavior of the bots. So, let's\nsay the user just saying, \"Test that\nwhen a customer ask about a delayed\norder, the agent provide the tracking\nnumber and estimated delivery date\nbefore asking for more detail.\"\nSo, this is\nlike a behavior that is uh related to\nthe dynamic of the conversation.\nAnd we want\na tool that is able to translate this\nnatural language behavior\ninto a test a a test framework that is\nversionable,\nreproducible,\nthat you can integrate in your CI CD,\nand that you can review.\nSo, this whole work will be implemented\nfrom a just got scale that you can\ndirectly implement through your coding\nassistant like code cloud.\nSo, this is the the whole work we do,\nand this is open source and something\nthat you can test.\nSo, I wanted to in this presentation to\nshow you the way it works.\nSo, you use your your assistant like\nhere it's um code cut.\nSo, what I want to do here is\nwrite in a very natural language a way\nto test the agent I just developed. So,\nfor this demo, what I did is I just\ndevelop a very standard agent that will\njust retrieve information\nfrom the just got documentation. So,\nit's a rack system, standard rack.\nSo, in natural language, just say, \"I\nwant to test this chatbot to this rack\nsystem that is accessible\nwith this endpoints.\"\nSo, we just need it's black box testing.\nYou don't need to have all the inners of\nyour agents, just the endpoints.\nAnd then, make sure to pass so, the\nauthentication of the API.\nAnd then, what you say here is you do\njust use just call skill to write a full\ntest suite testing against prompt\ninjection, hallucination, off-scope\ntopics, system prompt leakage. And you\nask\num the coding assistant to create like\ncomplex multi-turn conversation by\nsimulating users. Just this.\nAnd what you'll see here, so\nit will think.\nUh create different code, but let's see\nthe let's see the the the result in the\nend.\nSo, the result is here.\nIt generates a test suite in a Python\ncode that we can look at the\njust afterward.\nAnd then, it executes this Python\nscript. What you see here is for prompt\ninjection.\nUh you have two over six um tests that\npassed.\nHallucination, off-scope topics, system\nprompt leakage.\nAnd then, the agents will say, \"Okay,\nthis is the report of what I found.\" And\nas you see here, you see that for system\nprompt leakage, when someone claimed to\nbe a non-tropic city auditor, the bot\nresponded by outputting its complete\nsystem prompt. So, this is something\nthat has been discovered by the tool.\nLet's look at other stuff, like\nforbidden topic enumeration. So, it\nlooks like the agent just prompted uh\njust out put out out put some off-topic\ntopics.\nUh and and and so on. So, this is the\nkind of reports you may have. So, let's\nsay if I had put some more\nbusiness-specific\nuh information in my query of my coding\nassistant, I would have more\nuh more queries, more vulnerabilities.\nAnd of course, this report show more\ndetails about what is critical and so\non. So, this is the kind of things that\nyou can do very easily just by using\nyour coding assistant and the just got\nskill that is open source.\nAnd if you want to go more in detail,\nyou can look at the scripts that has\nbeen generated to have this report.\nSo, this is what you can see here.\nSo, this is a script that has been\ngenerated with the skill.\nAnd it's take all the different\ncategories of issue, prompt injection,\num and so on, hallucination. And if you\nlook at it,\nyou see all the framework that has been\num\nuh\nimplemented. So, let's look at it.\nThe way it works.\nSo, you have here an interaction,\nwhich is very simple. How does just got\ndetect hallucination in LLM outputs?\nSimple query and a judge.\nThis is the basic LLM as a judge\napproach. So, this is the kind of things\nthat you can generate.\nLet's look at more complex stuff.\nHere.\nHere you have a user simulator that has\nbeen generated where the user will have\ndifferent turn,\nuh\nask a real question, and then pivot\nslightly, and so on.\nAnd this has been implemented and tested\nuh with the bots\nfor which you just give the endpoints.\nSo, with a judge that has been\ngenerated. So, this is something that\nyou can do.\nHere, for the system prompt leakage, you\nsee it's more complex.\nYou can have like\nmore rules and turns like this. So, here\nit's red teaming. We you ask many\ndifferent strategy to leak your system\nprompts. And you see here that the the\nskills\nthat we provide is able to find\ntechniques and strategies to leak the\nprompts.\nSo, this is also something that is\npossible.\nSo, the way we it works actually, we\nprovide as open source this just got\nskills that is able to to to use a\ncoding assistant to red team on your\nside.\nI just got what we do a lot is working\nwith enterprise to to to work on those\nskills to make it the more business\nbusiness specific so that we can find\ndifferent flows.",
  "transcript_chars": 10986,
  "ingested_at": "2026-05-21T19:16:13.535315+00:00",
  "source": "retry-no-transcript",
  "yt_meta": {
    "view_count": 154,
    "like_count": 2,
    "channel_id": "UCcIXc5mJsHVYTZR1maL5l9w"
  }
}