How To Build And Evaluate Search Agents

Channel: Hamel Husain
Published: September 3, 2026
URL: https://www.youtube.com/watch?v=pi-IW4HYJwU

Join the AI Evals October 2026 cohort: https://maven.com/parlance-labs/evals?promoCode=yt-2026

Most people judge a search agent the same way: run it on a benchmark, look at the accuracy, take the model at the top. But that number hides almost everything that matters. It does not tell you whether the model or the retriever earned the score, how many searches it burned to get there, or whether it spent half its run asking the same question in slightly different words.

Transcript

Welcome everybody. Today we have Nandan Thakur and he is a researcher. He's been focusing on search for a really long time—search and retrieval especially for LLMs and AI. He gave a talk last year in a similar series where we went pretty deep on retrieval and he did a talk on the retrieval evals. He's done a lot of work on this topic. He's created several very popular benchmarks and so today he's going to be talking about search agents and how do you go about evaluating those? He's done a lot of work on that as well. Search and search in general in any AI product tends to be the weakest point and so this is a very important topic if you're building AI products. So I'll go ahead and hand it off to Nandan.

Thanks for the introduction and without any further ado, let's dive deep into what makes a good search agent. I'm Nandan Thakur and I'm currently a postdoctoral researcher working at Microsoft Research India. I completed my undergraduate from Vitzellani Gua campus and then for a few years I spent as an NLP research assistant at the UKP lab in Germany. After that from 2021 till a few months ago I was doing my PhD from University of Waterloo and currently, actually last week, I started as a postdoctoral researcher back here in Microsoft Research India.

You would probably recognize me from one of these benchmarks or works where I've been affiliated with. If you would have either heard about the BEIR benchmark, MIRACL, FreshStack or you would have heard about the TreeRAG efforts, I've been a part of these works.

So today we're going to talk about search agents and this is really a new topic which has emerged since the past year. So this is a very rough talk. Not many things are formalized yet. Many of these things are me trying to understand and my point of view of how I view search agents.

We will first start with the motivation and introduction. After that, I've broken my talk into three parts. The first part will focus on search agent evaluation. The second part will focus on the data and the training part. And the third part will focus on trajectory analysis. Finally, I'll conclude this presentation and give out some thoughts for the future.

Motivation and Introduction of Search Agents

So as you know, search is evolving and agents is the new thing which is pretty hot these days. It's been there since the start of the year and maybe early or late last year. Last year we did this series and the theme was RAG. Just to revisit what RAG was: you would have a particular user query which would first go through a retriever model, which in turn searches over a corpus of documents. This could be any private purpose which you have and then that would retrieve you top K documents for your particular user query, which in turn becomes a retrieved context for your LLM to then generate you an answer on this retrieved context. The main idea here was you would retrieve once in the beginning and then provide this context to your LLM for it to generate an answer. This was a naive RAG architecture which was pretty common until last year.

Ever since agents have come into the picture, the architecture has sort of changed a bit. What we now call agentic search involves the same user query but then instead of first going to the retriever model, now it actually goes to the LLM model first. In this case, the LLM model is the main driver of this where it can either reason first and see whether it needs to retrieve. There is a retrieval method which is present as a tool. So the LLM, if it wants to retrieve some information from the corpus, it would provide a query to this retrieval tool and in turn the retrieval model would then return back some top context which is used by the LLM to check some facts and refine its queries. This is an iterative process and finally once the LLM feels like it has almost reached the answer, it would then give you out the final answer.

The major difference between RAG versus agentic search is that there was usually retrieval once and then generate in the RAG paradigm. Here you have the LLM which is interacting with a retrieval tool—could be interacting with multiple more tools. But for search purposes we are just considering the retrieval tool here today. And then once it has iterated through these tools and it has done some reasoning, it then tries to give you a final answer.

Ever since the beginning of this year and even late last year, there has been an outpouring of search agent papers. If you actually see in the literature there are many many search agent papers out there. It might feel very new as a field but the field these days is developing so quickly. There are many papers which are coming out which are either developing search agents or developing benchmarks and so on and so forth.

Challenges in Search Agents

Let's try to broaden the perspective and try to see what are actually the challenges in search agents. Today's presentation I'll be focusing on three aspects very broadly. First, I'll be trying to focus on how you would evaluate search agents—such that how can we reliably test these search agents on complex reasoning tasks, how can we create benchmarks which can be used for evaluation. The second part I will be focusing more on the data part. How you can generate training data, synthetic training data in an efficient way so that you can use this training data to train search agents. And the third part I'll focus on analysis where we'll focus on why search trajectories or these search agent trajectories are important and crucial and can be used to identify particular trends which are there within search agents.

Part One: Search Agent Evaluation

Evaluation of search agents is hard because since search agents are intermixing retrieval and reasoning, their evaluation needs to be thought through and you need to actually benchmark which can evaluate the progress or the capacity of these search agents. What we find is that existing retrieval based benchmarks or QA based benchmarks such as HotpotQA, Natural Questions or BEIR contain only simple factoid queries and these are not hard enough actually to do search agent evaluation.

So the motivation becomes: how can we test the complex reasoning capabilities of search agents and what could be like a fair and reliable setup where you can actually reproduce the scores of an agentic search model and you can faithfully reproduce it plug-and-play the retrieval model. These are the two motivations for coming up with a benchmark.

Riddle-Style Questions

The first style of benchmarks which are out there—I call them riddle-style questions. These are easy to verify but actually hard to search and solve. Here is an example of this question. This is the BrowseQA benchmark which was introduced by OpenAI sometime last year. In this case, the question asks for the title of a research publication which was published before June 2023 that mentions so and so forth and was co-authored by three individuals. One of them was an assistant professor in West Bengal and another one holds a PhD. The answer is this research publication.

As you can see from this particular example, the question is pretty complex and it has multiple aspects involved within it. How I imagine you would answer or you would tackle this particular problem is you would do something which I have termed to be a funnel search. Most of BrowseQA queries require or somewhat give you a funnel search in which the first part—if you take for example if you want to figure out research publications which are prior to June 2023 which mentions cultural traditions so on and so forth—you may have more than 100 plus papers. After that you actually need to check the number of co-authors in these particular publications and let's say as just an example this reduces the funnel which originally had 100 papers which were satisfying the first criteria down to 20 papers which were satisfying both the first and the second criteria down to only one which will satisfy the last criteria—which is the assistant professor in West Bengal and the other author holding a PhD.

BrowseQA was actually pretty challenging. How they initially generated these queries: they would use a human to actually construct the question and they would give this query to another human which had access to search and other tools and another human would not be able to solve this question within 10 minutes. So these particular queries were then filtered and chosen for the benchmark. Also they tried GPT-4 which were not able to solve these queries. So BrowseQA was primarily used a lot for search agent evaluation or deep research model evaluation.

You would often see this kind of figure which was taken from one of these research papers where they would report the accuracy—basically finding whether the answer is present within the search agent or not. However, with this evaluation there are certain issues:

Since BrowseQA only provides you queries and answers, a lot of times papers when they are reporting numbers they're not reporting what search tool they're using for their experiments. The search tool which you use can actually impact the EM accuracy a lot.
Are these numbers really fair and reproducible? Since you don't know what search tool was used or what retrieval model was used for searching these documents, you could have actually used a very good retrieval model for one of these LLMs but not a good one for the other one. Who is there to actually confirm this?
Since we are just measuring the EM accuracy, we don't know how much actually the retrieval tool in itself contributes to the end EM accuracy.

BrowseQA Plus

Due to these issues, I was a part of this project known as BrowseQA Plus which was led by Zoey Chen from University of Waterloo where we actually constructed a more fair and reproducible BrowseQA version. What this did was it converted the QA task into actually an open retrieval problem setting. The idea was very simple. You would take the BrowseQA question and answer pair. We would then use OpenAI O3 model back then to actually gather the gold web pages which were somewhat relevant or which can contain clues or information which is required to answer the question. These are coming out as URLs or evidence web pages and then we would actually provide these documents to a human annotator who would actually annotate like "this document is useful to answer this part of the question." A human verification step would be there where a human annotator would actually check these documents are really relevant in answering the question and whether it contains the final answer or not.

So this is how we constructed BrowseQA Plus. Originally we had the queries and the answers. Now in addition we have two things:

A web corpus, which was constructed using these positive documents which I just described. In addition, we also generated what's known as hard negatives which is usually put in the corpus to improve the retrieval difficulty.
Relevance judgments, since we have humans which have judged what relevant documents are there for a particular question. We have relevance judgments which are available for each query.

The web corpus has almost 100k documents which is a very comprehensive coverage. It is also very difficult since you have positive and hard negative. So it makes the retrieval task to be difficult and it's of a practical size for evaluation. It's not too big so that researchers can adopt this corpus. On the other hand, relevance judgments are pretty great because you have human supervision on this and this typically follows how a retrieval evaluation setup would look like.

To measure the answer accuracy we came up with the exact match accuracy metric and for measuring the retrieval accuracy we look at two metrics: either the number of search calls which were used during an agent's turn or the recall metric which denotes how many positive passages were retrieved in your subset out of the total positive passages available.

Key Findings from BrowseQA Plus

Some results and key takeaways which we found from the BrowseQA benchmark is this interesting diagram which is there in the paper. On the x-axis you have the number of search calls and on the y-axis you have the EM accuracy. An interesting finding which you would find is that actually both the search model and the LLM both of them contribute to the EM accuracy. As you can see, GPT-5 is the best LLM out of all because, if you're actually considering vertically, it's on the top. On the other hand, you can actually see that whenever we move from BM25 to Qwen 3, which is a dense embedding model, we are actually taking fewer search terms. So a number of search calls going towards zero is actually better for us and it's also improving the accuracy. With the same model, using a better retrieval model actually gives you a better accuracy and it also takes less number of turns to reach the final answer.

Key takeaways from here: both retrieval and reasoning are important for agentic search and a good retriever should (a) increase the accuracy and (b) be able to reach your final answer with hopefully fewer search terms.

Recent Work on Search Agent Evaluation

More recently there was also subsequent work which was done. This was another very interesting plot which was taken from the Reason Modern ColBERT paper from their blog post. They actually used ColBERT-style model to evaluate a search agent on BrowseQA Plus. Here also an interesting finding is that agentic search is not just the LLM problem. It is a very big retrieval problem as well. From this plot actually we found that if you just keep your LLM to be GPT-5, depending upon whether you use BM25 or a Reason Modern ColBERT, which is a very good late interaction model which was also fine-tuned on reasoning-based training data, you can actually go from 60% on BrowseQA Plus to actually 88%, which is a huge jump with the same LLM. This shows that the retrieval model you choose is very important here.

Finally, we also find that sometimes you can use BM25 and actually modify BM25 to get and tune it pretty well to actually get a good accuracy. This is another table which has been actually taken from Pyerini, which is another paper and the most interesting part which I found was the DS Flash, DS Pro and the one which is highlighted in the line. Actually with just BM25 they have achieved 60 to almost 80% accuracy. So it shows that when you tune BM25

(capture appears truncated)