{
  "video_id": "H4o7h6ZbA4o",
  "title": "DSPy GEPA Example: Listwise Reranker",
  "url": "https://www.youtube.com/watch?v=H4o7h6ZbA4o",
  "transcript": "Hey everyone! Thank you so much for watching this tutorial showing you how to use the new GEPA optimizer in DSPi. This is so much fun to put together. It's so exciting to see the performance metric go up using DSPi's optimizers and there are a few things that I learned along the way in order to get this to work that I hope to share in this video. So a quick takeaway just to TLDR it in case you just want to have a quick overview how to use it basically. The core thing to understand is that GEPA is using a Pareto frontier along your validation data set. So say you pass in three samples, A, B, and C, and you have this population of prompt candidates. Each of your prompt candidates that survive the selection operator in the genetic optimization loop are performing the best on at least one of your samples. Say you have a prompt that's able to achieve the task A or sample A, but not B and C. And so this is the idea of your Pareto frontier. And that's key to understanding the metrics that you get from weights and biases that are built into GEPA and just sort of how you're going to be observing your run, and then sort of how you design the size of your validation set those are probably the two most important things to note is just how to monitor it how many samples should i pass in what does it mean max what the max metric calls and things like this so i think another it's just amazing to see that we'll describe this as a six-step video that will show how to use geppa for this particular task of optimizing a list-wise re-ranker something particular to say we've ate and information retrieval but But this particular thing of looking at prompts, there's one prompt that's able to achieve a successful re-ranking of those documents compared to the other prompts as it's running for a while. I think it's just really incredibly interesting. So I hope this video is helpful and interesting. Thanks so much for watching. Firstly, before even diving into it, if you want to follow along with the code, you can access this notebook on WeV8Recipes. That'll be the first link in the description of this video. And you can see the link on top with where to find it in WeV8Recipes on GitHub. So to kick it off, we're going to be optimizing a list-wise re-ranker. So as a quick background on the task that we're trying to solve with our large language model or our large language model program, in this case, we're just going to be optimizing a single prompt. But with DSPy, you can imagine stitching together several inferences or, say, optimizing an agent with the function calling loop, whatever task you're trying to solve. In this particular case, we're going to be trying to solve the re-ranking task. So re-ranking describes the LLM is given as input the query and a list of candidate documents that have been identified by a first stage ranking algorithm. So this could be something like vector search, BM25, hybrid search. It produces an initial set of candidate documents, and the re-ranker then predicts a new ranking of the documents, sort of tip mostly by their, most commonly by their relevance to the query, but you might also have, say, like diversity rankers. But anyway, so there are two common forms of re-rankers, cross-encoders, and list-wise re-rankers. Cross-encoders take as input each query and candidate document in isolation, whereas a list-wise re-ranker takes the query and all candidate documents as input to then output the new ranked list. This notebook is going to be simplifying the task a little bit, focusing on identifying just the most relevant document rather than re-ranking the entire set. So we're given five candidate documents and we're just predicting which one is the most relevant document. So just optimizing that recall at one metric. And this is kind of motivated by some work we're doing at WeV8. We're building these retrieval pipelines. I'm just really interested in this kind of gap between recall at five, recall at one, and just generally seeing if this kind of thing would work to close that gap. And it's exciting to see that it is. So as a quick primer on this, we particularly in this eval that we're going to find, we optimize from 32% to 45% with our GEPA optimizer. But I think it's worth noting that this is a small scale optimization run, 500 metric calls or rollouts if you like to think about it like that running in about an hour and a half it's super likely that we could get a higher gain by running this for longer as in as is indicated by the Pareto Frontier that we'll be explaining in a second as we go through the notebook and how this works but generally I'm just so excited to get this working that I just really wanted to stop working on it and make this video but definitely we're super excited to see how much farther we can keep pushing this okay so this notebook there'll be six steps to this video that you can see in the chapters of this video. Firstly, setting up our DSPy program, then loading our training dataset, wrapping them in dspy.example objects, and then more particular to GEPA, we'll be defining our metric with natural language feedback. This is one of the big new things about GEPA is now rewriting your metrics to interface natural language feedback for the prompt optimizer. Then we'll be evaluating our unoptimized best match re-ranker. Then we'll run the GEPA optimizer. I'll show you some things, some hyperparameter configurations, some monitoring tricks that help me use this, and then we'll evaluate our optimized best match re-ranker. Step one, we'll define our DSPy program, the best match re-ranker. I should preface this by saying this video isn't going to go over all the details of the DSPy programming model and how to define your language programs, compound AI systems, or agents with DSPy. The link in the description below is Erica's video on how to use the DSPy programming model to write blog posts and decompose the blog post writing task into four inferences. That's a great way to get started. and I also highly recommend checking out the official DSPy documentation to understand some of the newer things with DSPy 3.0 and how generally the DSPy programming model is evolving. This will be this best match re-ranker is a pretty simple DSPy program setup. So we have some different models that we're using. This may not be super clear but this is kind of a part of the system that I kind of took this out of a system that we're using to just put it in a notebook and show this part so it's a little messy in that sense but that's not really what we're going to be primarily focused on in this video. So to kick things off with the DSPy things, we have our signature, the best match re-ranker. So this is an initial prompt that I came up with that I thought would be a solid start for how to re-rank documents. Identify the single most relevant passage to the query. Your task is to analyze all passages simultaneously and identify the one passage that is most relevant for answering the query. So this is an interesting thing about DSPy and prompt optimization is the prior, what you want to start off with, with your optimizer run. Do you want to start off with something completely vague like find the most relevant document and just that's all that there is to the prompt or do you want to spend a long time crafting your prompt and you start off by writing the perfect prompt and that's what you give to dspi so i think this is somewhere in the middle of that this isn like you know like i spent a little bit of time thinking this might be a decent prompt to start out with but i didn spend a lot of time writing this prompt And that maybe one of the most interesting things out there to be exploring It kind of like priors and deep learning I think kind of, I think all the same concepts in transfer learning of having a really strong initial network to then fine tune, having a really strong initial prompt would, seems like a great way to seed these optimizers. And it's generally a really interesting thing to be thinking about with these prompt optimizers. So these are our inputs. We have the query, we have our search results, and then we output the best match ID, the single most relevant passage to the query. So then we have our best match re-ranker program, the DSPy module, where we're going to be orchestrating our internal inferences and the logic of how they process their inputs to produce outputs. So we're going to be using DSPy.chainofthought. This is where our chain of thought is going to first produce a rationale and then the outputs to your task. And DSPy has all these different inference strategies, as well as these optimizers. It's a really interesting aspect of DSPy. is these abstractions around adapting to these different inference strategies. But in this case, we're just going to be using the chain of thought. And I left a note here, this isn't something I got around to in this tutorial, but I think it could be really interesting to interface this rationale from chain of thought with the metric, with the natural language feedback in the metric. That's not something that we got into in this video. So now just to kind of wrap this up and hurry up and get to the GEPA part, we have our forward pass. It just says we take our question, our candidates, we pass them to the re-rankers, we use this to identify the best match result, and then we return our sources. And this is a little messy because I kind of just am taking this out of a different system. In step two, we'll be loading the data set. So this is the first interesting insight I found with trying to make the GEPA optimizer work and generally learning about this is, in my opinion, and I know that there's kind of a big topic around sort of training data distributions and machine learning, but I found that this is way more effective if your training data set consists of hard examples. So the way that we found hard examples is by using this re-ranking filtering algorithm. So firstly, if you want to use this data set, it's available on Weevae's Hugging Face repo. And so now to explain what this data set is, is that we've been really diving into these retrieval pipelines at Weevae. We've been benchmarking on all these different recall metrics, FreshStack, Enron QA, Wix QA, BEER, all these things. We'll be publishing some things about this soon. But we found that there's this really interesting gap between recall at five and recall at one. Recall at five, making sure the most relevant document is in the top five, that's pretty good for most of these benchmarks. It's getting better and better as we have all these state-of-the-art advances in embedding models, hybrid search, multi-vector, cross-encoders. And so there's still a bit of a gap between recall at five and recall at one. So what we found is by using the Enron QA data set, this is a data set of emails and you're asking questions about the emails. And so we went through a thousand questions from Enron QA and we identified 138 questions where the pre-trained re-rankers are able to achieve recall at five, but not recall at one. So this is a particularly challenging data set. The state-of-the-art cross encoders aren't able to find this most relevant document. And that's kind of our first insight with using the GEPA optimizer, especially as we then will get into this Pareto Frontier concept, is we found it useful to have a very difficult training data set and evaluation data set. So more broadly in machine learning, it's not necessarily best practice to have the training data set only be hard examples. And I think that's a broader conversation topic around all this. But so, and I'm not even kind of presenting this as a paper. I'm just sort of for our notebook. That's something that we found with the GEPA Optimizer to Work is having a difficult training set. so there is a question of maybe it's now optimized to expect the question to be difficult and therefore it won't generalize well to easy samples but then you could kind of argue that you don't even really need like as you build these retrieval pipelines you could calibrate the score there's clever tricks you can do in that kind of thing but anyway so this is the first thing is understanding our data set we have the five relevant candidate passages and then we have our okay so there's one other thing that we did to pre-processing this data set. So these emails in Enron QA, they're like really long emails. They're like 2000 tokens. So we also summarize the relevance of the query in each candidate document. And so as seeing as input, the summarized relevance, this is part of building these retrieval pipelines as you step into these compound AI systems, having more than one inference in your DSPY program. But so right out of the gate, we found a difference in with our training set, we found a pretty big performance jump by passing in the summarized relevances of the candid documents to the query rather than the entire email. So one last thing, just setting up the context for the tasks that we're going to be optimizing, we add one more thing to the dataset preprocessing. We're making sure the ground truth is available in our examples. This is just like a mistake I made when I was preparing the dataset that I fixed in this notebook. Okay, and then we just run our re-ranker with the samples just to kind of make sure that it's all connected. And then Something you can always use with DSPy is you can access the LM. This is something I particularly like to do is put the LMs in the DSPy modules. It helps when you have multiple DSPy modules, but you can use Inspect History. This is one way to get a look at it to just see the prompt that it's using, and you can see how particularly it's structuring the prompt with the input fields, describing the input and output fields, and then where it interfaces the instructions. Then you can see the particular input that it's applying the inference on, and then the response from the language model. This is the chain of thought rationale, and then this is what it's predicting the most relevant document, passage two. Okay, so now to finally start diving into some of the GEPA things, step three, defining a metric with natural language feedback. So this is one of the key innovations in GEPA that is so interesting, is rather than having our metrics just return scalar rewards, like recall being 0.7, the metric now sends natural language feedback about what went wrong or right with the inference for the optimizer to use. So typically the optimizer would take that 0.7 and it would like calculate the partial derivatives of each of the weights of your neural network to tweak the system to hill climb towards a higher metric. But now it can have this natural language feedback and it can now navigate this optimization space in a totally novel way because of this. So this is a pretty simple example of a way to interface natural language feedback. So now when it's right, similar 1.0, it's just awesome. The system correctly predicted the top document But then the more interesting thing is when it incorrect it saying incorrect document selected for the query question The correct answer was ground truth content The system incorrectly predicted predicted content So now instead of just seeing 0 the optimizer is able to reason about what it had predicted and what it did, what the ground truth was in natural language. So next step four will be to run the unoptimized evaluation. We'll be evaluating our best match re-ranker before we optimize it with GEPA. So we split our data set. We have 138 examples and we're going to be using 100 for training and 38 for testing. This is a bit arbitrarily chosen. It's not a super big test set. And so you complain there, but again, it's just a notebook. So hopefully take it easy on me with that one. But so we run our evaluation. We set up DSPY as evaluator. We pass in our recall metric with the feedback. You can pass in however many threads you want to have to have it run faster with multi-threaded evaluation. And then we see that we get 12 out of 38 or 32% with this unoptimized best match ranker. Step five, the exciting part, running the GAPA optimizer. So sorry that this took us 12 minutes to get started. We've gone through setting up the DSPY program, defining your compound AI system, your agent, or just a single LLM inference, creating some kind of data set, and then defining our metric with natural language feedback. And now that we have all those building blocks, now we can optimize our system using DSPY's optimizers like GEPA. So as a quick primer, it's critical for GEPA to understand this idea of this Pareto frontier. GEPA is an evolutionary algorithm where it has a population of potential prompts that at each step it's going to be evaluating and sampling some of them, mutating them by proposing some kind of new instruction inspired by the one that it just sampled and maybe some history about the lineage, and then using that to propose a new prompt that is then evaluated with this Pareto frontier idea. So each candidate on the frontier is better than all the other candidates on at least one of your validation samples. Let's say you just have three samples, A, B, and C. And let's say instead of, in this case, we have a zero, one, or let's keep the zero, one. So let's say one of your prompt candidates is able to identify the correct document on sample A, but not B and C, whereas you have the most dominant one is succeeding on B and C, but not A, this kind of idea. So instead of just sampling the program that is succeeding in B and C and just continuing to mutate that one, we organize this diversity in our prompt candidates by how they're performing on our validation data set. So this is maybe the, and one other thing about setting this up is, so really quickly before we configure the hyperparameters, because then it is, you know, this wall of text. So I wanted to add a few images before we get into that part. So I highly recommend setting up the weights and biases logging. So the weights and biases logging, this is already built into GEPA. You just need to pass in your weights and biases API key and the name of your project. And so there are two key things that kind of help you preserve your sanity while you're watching this run, which is the best validation set aggregate score. This is as it's running on your entire validation set. So in this run particularly, it starts at 0.32. And as it's going, it hits 0.36. and then there was this terrible period where it just wasn't improving from step 5 to step 21, and then it upticks, and this is a ton of fun, this is kind of the fun of optimization, and then you get to have this iteration where this is just kind of like the pulse as you see that it's still logging things. And then you have, this is another really interesting one to look at, is the Valset Pareto Frontier Aggregate Score. So what this says is, when it's saying that this is at 0.52 compared to this being at 0.4, That means that on our Pareto Frontier where we have the validation samples, if you just took the best performing prompt, sort of like a theoretical limit, because it's helpful for sanity testing, like can I even tweak the prompt to achieve these tasks? Is it even a prompt optimization problem? So seeing that this Pareto Frontier is going up is telling you that there is the opportunity there to improve it because some of your prompts are achieving these other samples. You just haven't found the prompt that's generalizing across all of your samples. So to conclude on the weights and biases metrics, if you're worried that your aggregate score across the entire validation set isn't improving and you want to stop your run, maybe take a look on this validation set Pareto Frontier aggregate score. If this is continuing to improve, that suggests that your optimizer is just in that expiration phase and there's still a chance that it's going to find that generalizable prompt. So the next advice that I'd recommend is something that I found helpful is, as we'll see in a second, when we run the GAPA optimizer, it's logging all this text of how it's going. I highly recommend just copying and pasting all that. If you're, say, running a Jupyter notebook and you just hit copy, sell output, and dropping that into Gemini and just saying, can you analyze this prompt optimization run? How's it going? And it's really interesting to chat with an LLM about how the LLM optimization run of an LLM system is going. Awesome. So now let's look at the code to run our GEPA optimization run. So the first thing we have, we pass in our metric, and then we're going to set the max number of metric calls or the maximum number of rollouts of the system that we're optimizing. So this is key to controlling the cost of the optimization run. So it's probably worth calculating how much does each of my inferences cost me. Say you're using GPT 4.1, you can calculate the sort of average input tokens, output tokens, and do it this way. And you can use DSPY's usage to get a sense of that. But this is how you're passing in how many times it's going to call the program that it's optimizing. So then you have the reflection LM. So in this case, we're using GPT 5 with 32,000 max tokens, temperature 1. This is the model that's going to be looking at the inputs outputs natural language feedback and proposing the new prompt based on this So this is your your prompt mutation operator sort of so it's pretty important to have this sort of most capable model for doing that step So then we have the reflection mini batch size. So this is how many samples it's going to be Sampling in order to run that input output natural language feedback that it's going to be used for using to propose the mutations So then you have the use merge So the system where we're merging in GEPA, this is something that it's using this lineage tree to try to merge. Every time it proposes a new prompt, it's also going to output a rationale of what makes this new prompt different. And so when you're merging, you're going to try to use these rationales. Say you had two mutations that somehow took them into different sets of the Pareto frontier You going to try to merge that feed the rationales of changing the prompts to try to produce a new prompt It definitely an interesting thing at the frontier of this Then you have the maximum number of times you want the system to do that merge operation. Then you have the number of threads for running GEPA. You have the weights and biases logging, and then you have your sample. So this is probably the most interesting thing to focus on, I think, with this. Obviously, you're going to be tuning the metric calls with how long you want to run it for. So here's the interesting thing. The size of your validation set is you want it to be larger because you want that aggregate validation score. You want this to be telling you something. But at the same time, the validation score, that's what's being used for your Pareto frontier. And so if you have a ton of samples, each new candidate prompt is more likely to be on the frontier because it's just, if you have 300 samples or so, it's going to be easy to find some sample that this prompt performs better than the others on. So having a smaller set results in a more informative Pareto Frontier really helps with that exploration step. So I think that's one of the most, this is the most interesting hyperparameter of GEPA, is understanding the size of this validation set, because you're using it for the Pareto Frontier. And I think it'll probably be interesting to see if these two things get separated as GEPA continues to develop, the aggregate, the large-scale validation score compared to the Frontier. but this is definitely an interesting detail of learning how to get this working sort of right now with the state of it right now okay so then once you run it you'll get this output it'll tell you hey we're saving it on weights and biases and you'll see the output it tells you the scores it's getting as it's sampling these subsets it'll output the prompts that it's proposing and it goes on with this for a while and then you'll see this logging of the Pareto frontier so again probably i think the most interesting thing is that kind of um uh is it where is it um or the this one is the Pareto Frontier score so this is a little hard to find because in this case it just started so it doesn't have a significant difference or it does it it has the it also proposed something it did terribly then it's going back to the frontier with the candidate the seed that I had given it and then it's continuing to go so so it goes on it goes on it goes on and so I recommend just copying and pasting this throwing it into Gemini as it's running monitoring the weights and biases to see how it's going and see it's starting to improve it gets you know up to 0.36 then it goes on for a while longer and then eventually i just kind of want to show where this gets to at the end before we move into evaluating the optimized program okay let's see what this one looks like uh so we're on 22 so when i was running this it definitely hit a wall where in the middle it just really wasn't succeeding you see it has this totally chaotic parade of frontier where it's because i have i think another issue maybe to note is I have this binary score of zero or one. So as a result, all the, it's kind of hard to sort of identify the document, but you can see that as this case, like in this example, only the only prompt zero and prompt six are, are successfully finding the optimal doc, the most relevant document. And then in this case, you only have two and five. So this is what's interesting is it's amazing to me that this, that there's these two prompts that successfully that then result in this re-ranker identifying the most relevant document on this particular sample, but not for this other one. And so that's kind of the, that's the idea kind of is understanding that you see here you have 0, 3, 4. And so this is like the interesting thing about the Pareto frontier. Okay. So just, sorry, to scroll down to the bottom, the very end after this has run for a while. Okay. So is this even the one that I want to show? Oh, okay. Yeah. So this is kind of interesting thing so at the end it kind of stalled the best one it can find on all the samples is point forwarding this optimization but then it gets but then the optimal on the frontier is 0.52 so just taking if you just had this oracle where you're taking the best prompt on each of the samples that gets you to 0.52 and you can see again the samples here where 0.6 is doing it but the others aren't successfully finding the right document again with this one this one you have only one prompt is able to find the right document on this one so this is sort of i think the key concept with GEPA. So lastly, we have step six, our moment of truth step, evaluating the best match re-ranker, the optimized best match re-ranker on our held out test set. So during this run, it's not really using the validation set in the sense that it doesn't look at the validation samples in order to propose mutations to the prompt, but it still kind of is using that set to navigate the space. So it would kind of be unfair to report that as the final score. So you still have this held out test set idea. So we're using that same held out test set in the beginning. firstly we save our optimized program so we have the save prompt or in if you're using optimizers like bootstrap few shot or mipro you would have the few shot examples as a part of the serialized form of your optimized program so let's say you had multiple modules or lm inferences as a part of the dspy program now we rerun our dspy evaluator and we find that that score improves from from 31.6 up to 44.7 so pretty exciting jump and i think this is just scratching the surface i think we could easily run this for longer and get even further performance gains. So finally, to see the optimized prompt, again, we just ran the inference of our optimized vRanker as it ran through our evaluation set. We can now take a look at the optimized prompt. So I think the first thing to note about these GABA optimized prompts is that they're pretty long. The prompt that it finds in this run particularly is 2,000 tokens, so it's quite a long prompt, but then it's still a shorter prompt compared to having the few shot examples as well. So that's kind of an interesting thing to note about instruction optimization versus few shot example optimization and that kind of story. But you can see all the rules it learns. It's trying to, again, the task is it has this search query, it has these candidate documents, and then it's predicting the most relevant search result. So it's learning all these different rules to try to figure out how to assess the best document. I think it's just such an interesting thing. So I really hope this video was useful. Again, if you want to find the code, it's available on WeV8Recipes. And I'd love to know about your experiences with using the GEPA Optimizer. And I really hope you'll subscribe to the WeV8 YouTube channel for all sorts of things about WeV8, vector databases, agents, compound AI systems, and all this awesome stuff. Thanks so much for watching.",
  "transcript_chars": 29438,
  "transcript_filled_at": "2026-06-06T14:49:15.578377+00:00",
  "transcript_filled_by": "tk-bulk-groq-retry-20260606"
}