{
  "video_id": "4zt7FSqZRus",
  "title": "Arize AX: Alyx Demo 2026",
  "url": "https://www.youtube.com/watch?v=4zt7FSqZRus",
  "transcript": "Your AI app is live. Actual human beings are starting to engage with your agent. But somewhere along the lines, you have the hunch that it's starting to hallucinate. What I can do to try to understand what might be occurring is actually click on a production trace. I can see the input, the output here, and all the different steps that transpired within this particular interaction. But there's quite a few traces for me to go through, and that could get a little cumbersome. So what I'm going to do instead is ask Alex, my helpful co-pilot, to help. Start with, I'm going to request it to identify some issues because I think it's hallucinating. Here we can see that after analyzing the project and traces within the project, Alex has some recommendations. First and foremost, set up a hallucination evaluator. Let's go ahead and do that. You can find that Alex starts to craft that evaluator for us. Here are all the tasks that is outlined so that it remains on task and doesn't get distracted. Now I was able to create a trace level hallucination detection eval and it's asking me if I want to run it on historical data. I'm going to go ahead and click yes. and now we can see that it starting to run Now that the task has finished processing if I scroll here to the right I can see span evaluation starting to run These are actually factual, but these are hallucinated. So now what I'm going to ask Alex to do is to take the hallucinated ones and put them into a data set. So what we've been able to do now is we've been able to identify that there is an issue, the issue being hallucinations. We've created an eval to measure and identify which traces are actually causing those hallucinations. And finally, we're taking the five out of the 10 traces that were labeled as the hallucinations and putting them into a data set so that we can look to iterate upon them and hopefully look to eradicate that behavior. You're building AI apps. You probably already have this problem. Your agent does something unexpected. Well, here at Arise, we help steer your agent in the right direction. To start with, we're going to go into our playground here. I'm going to create a new playground called Alex Demo. And we're going to go ahead and load up that data set that we had previously, where we were looking at trying to route a supervisor agent to its sub-agents. So now that I've pulled up that data set, we can see all the values within the data set. I gonna load up a prompt This is the prompt that is currently being used within our application You can see that it has delineation to its subagents It also parameterizing the input from the end user To help get to a desirable state, I'm going to enlist Alex, our AI chatbot, to help. Before engaging with Alex, I do want to pull up a couple of columns. So I'm going to pull up that eval explanation. that we saw in that prior video, as well as the label. So here we can see that this is the user question that was asked. This is the output that was originally generated. Here's the explanation of that LLM as a judge eval that ran previously, and then we have the incorrect label. To start with, I'm gonna go ahead and ask Alex to craft a improvement to this prompt by taking into account this eval explanation. Here, Alex is taking in its context, the dataset as well as the prompt, and it's remaining on schedule by creating a series of tasks so that it does not get distracted as AI agents are often to do. Here we can see a prompt that Alex has generated. I'll go ahead and hit accept and we'll place that into the bar. The next thing that I'm going to do is I'm going to ask Alex to craft an eval to determine whether or not the output is aligned with the explanation, just to see if we are continuing to focus in on the right direction You also notice here within the playground that I could elect to run this prompt with numerous different models if I wanted to. Right now we're running with GPT-5 Mini, but you can plug in any model endpoint that you like within the playground to use. Now we can see that Alex is able to generate that eval for us. We take a look at that evaluator. It is looking at the output and then the corresponding explanation from our LLM as a judge eval that ran in production. So now I'm going to close out of Alex and run an experiment. Now we can see that that experiment has run. And we also see the evaluator tagged here, but along with a score and an explanation. And then at the top, we can also see that 64% of the time now, as opposed to 0% of the time, because previously all of these were labeled as incorrect, our output is correctly aligned with the output from the eval that we had run previously. So this is definitely improvement from what we had before. So I might want to now go ahead and save this version of the prompt as a new version and deploy it to production. If I go into VHub, I can now see that there are a few different versions of this.",
  "transcript_chars": 4951,
  "transcript_filled_at": "2026-06-06T14:44:01.772556+00:00",
  "transcript_filled_by": "tk-bulk-groq-retry-20260606"
}