RAG Retrieval Lost 40 Points When Distracting Passages Grew to 2.68M

本文由「RAG Retrieval Lost 40 Points When Distracting Passages Grew to 2.68M」的公开内容整理，完整内容与后续更新请点击文末链接阅读原文。

On a collection of 100 research papers, BM25, a standard text search method, put a paper that answered the question in its top five results for 99.7 percent of questions. On 1,801 papers, that number was 98 percent. I built the test to find the threshold at which that number drops, and it barely moved.That result looked like good news for anyone running retrieval augmented generation, or RAG, where an AI model answers a question using passages that a search step fetched first. If search stays good as the collection grows, the rest of the system has solid ground to stand on. Then I ran the same kind of test on 2.68 million Wikipedia passages, and the picture changed.This article reports both tests as one story. The first test used questions from my earlier test of agentic retrieval. The second used public questions that nobody wrote for my setup.The code, questions, labels, and results for every test are in a public repository.What did I want to find out?

I wanted to know how often the search step of a RAG system, the part that finds documents, misses the right one as the collection grows. I also wanted to know whether it matters if the extra documents look like the right answer.A distractor is a document that does not answer the question. Some distractors are random, and some are close in topic to the right document.Each test asked the same plain question. If I add more distractors and change nothing else, does the search still find a document that answers the question?

Before running anything, I fixed a rule for what would count as a meaningful drop, so that I could not choose the bar after seeing the results. A size counted as a degradation point only if the score fell at least five points below the smallest collection. Five points means one more question in every 20 loses its answer, and a point here means one percentage point.The rule also needed the drop to be clearly real and not luck. My tests use only some questions, so a different set of questions would give a slightly different drop. I checked this with the 95 percent range, which is the span where the true drop probably sits.Think of a bathroom scale that can be wrong by about 2 kilograms either way. If it says you lost 1 kilogram, you cannot tell whether you lost anything. If it says you lost 8 kilograms, you really did.My measured drop is the scale reading, and the 95 percent range is how jumpy the scale is. If that span includes zero, which means no change, the drop could be luck.For example, a 6 point drop with a span from a 9 point drop to a 1 point gain does not count, because that span includes no change. A 6 point drop with a span from a 9 point drop to a 3 point drop counts, because every value in that span is a drop.Three scores appear in this article: recall at 5, success at 10 and exact match. The first two ask whether search put a good passage near the top, and exact match asks whether the model's answer matched an accepted answer. I define each one where it first matters.Test one, 2,000 arXiv papersThe first collection held 2,000 recent arXiv abstracts in machine learning, language, and search. I used 298 questions, each written from one paper and checked by hand, so every question had a known right paper. No paper was used twice, so the 298 questions came from 298 different papers.For each question I built collections of 100, 250, 500, 1,000 and 1,801 papers, and each larger collection contained the smaller one. The right paper was always inside. The rest were distractors of two kinds.Every other paper in the 2,000 could be used as a distractor. That included the 1,702 papers with no question and the 297 papers that belonged to other questions. For any one question, those papers were documents that should not answer it, and I checked that next.Ordinary distractors were picked at random from papers that were not close to the right one. I made three separate random picks, which I call draws.Look-alike distractors were the papers nearest to the right one,