Hard Negatives, Hard Lessons (Findings of EMNLP 2025) prunes the BGE retrieval training collection from 15 datasets to 7, reducing it from approximately 1.6 million training pairs to 680,000. Despite this reduction by a factor of about 2.35, E5-base gains approximately 1 point in average nDCG@10 across 14 BEIR datasets.

The remaining data still contains labeling errors. Some passages answer the query but are treated as hard negatives during training. The paper proposes RLHN (ReLabeling Hard Negatives), which uses two LLM stages to identify these passages and relabel them as positives. Compared with training on the pruned data with its original labels, relabeling improves E5-base and Qwen2.5-7B by approximately 0.7 and 1.4 points on BEIR, respectively.

Incorrect supervision from hard negatives

A retrieval training instance typically contains a query, positive passages, and a set of hard negatives. Hard negatives are usually selected from highly ranked retrieval results. They resemble the query but are treated as irrelevant during training, helping the model distinguish relevant passages from irrelevant ones.

Many retrieval datasets have sparse relevance judgments. Annotators assess only a small portion of the candidate documents, so an unjudged document is not necessarily irrelevant. Mining across the full corpus can retrieve relevant passages that were never labeled.

The paper illustrates this with a HotpotQA query: “What park contains the Wild Beast and a 20 acre water park?”

The two passages on the left are the original labeled positives, describing the Wild Beast roller coaster and Canada’s Wonderland. Splash Kingdom Waterpark, at the upper right, is in California and is irrelevant to the query. Splash Works, at the lower right, explicitly states that it is a 20 acre water park within Canada’s Wonderland. The blue text highlights evidence that supports the answer. Treating Splash Works as a negative makes contrastive training push down the score of a relevant passage.

Not every case is this clear. The paper’s manual analysis also finds incorrect original positives, queries that match several passages, and candidates that answer only part of a query. Assessing these cases requires judgments about degrees of relevance and query ambiguity.

Cascaded relabeling with RLHN

Dataset pruning narrows the scope of cleaning, but it cannot identify which individual passages are mislabeled. RLHN relabels instances in 6 of the 7 retained datasets. ArguAna requires retrieving counterarguments, so the paper keeps its original labels.

Each API call provides the LLM with a query, its existing positives, and up to 25 hard negatives. The model assesses the candidates’ relevance to the query and compares them with the existing positives. GPT-4o-mini scans the instances to flag those that may contain false negatives. Only flagged instances proceed to GPT-4o for a second assessment.

The instance on the left contains a query, existing positives, and hard negatives. GPT-4o-mini passes only potentially problematic instances to the second stage. If it finds no false negatives, the instance receives no further review. GPT-4o’s judgments determine which candidates move into the positive set and which remain hard negatives. The output is a change to the training labels.

Using the May 2025 prices cited in the paper, the estimated cost is approximately $300 for the first stage and $3,000 for the second. The paper reports these as maximum cost estimates, using actual input token counts and an estimated average of 2,048 output tokens per call. Cascading reduces the amount of data the more expensive model must process. The paper compares three ways to use the detected false negatives:

1. Remove the entire training instance, including its original positives and remaining hard negatives.
2. Remove only the detected false negatives from the negative set.
3. Add the detected false negatives to the positive set while retaining the remaining hard negatives.

The third approach is RLHN. It retains the newly identified relevant passages as positives, allowing a query to have multiple positive passages. If an instance contains more than 7 detected false negatives, the paper discards it entirely. The authors consider such queries potentially ambiguous and unsuitable for training.

Dataset pruning results

The 2.35-fold reduction comes from dataset pruning before relabeling. The paper removes one of the 15 BGE retrieval training datasets at a time, fine-tunes E5-base on the rest, and measures the effect on BEIR. It then selects the retained datasets using these results together with performance on individual tasks.

Each bar shows the average score after removing the dataset named below it. Training on the full collection gives a baseline of 0.519. A bar above that baseline means removal improves performance, while a bar below it means performance declines. The selected combination of 7 datasets reaches 0.529, shown by the “7 Pruned” reference line. This comparison between approximately 680,000 and 1.6 million training pairs is the source of the roughly 1 point gain in the opening paragraph.

ELI5 is one example: removing it alone raises average nDCG@10 from 0.519 to 0.525. The final collection retains FEVER, HotpotQA, ArguAna, FiQA-2018, MS MARCO Passage, NQ, and SCIDOCS RR. Selection accounts for both average performance and individual tasks. It is not simply a matter of removing every dataset with a negative average contribution.

The footnote on the paper’s first page states that pruning retains approximately 42.5% of the original training pairs. Dividing approximately 1.6 million by 680,000 gives about 2.35, equivalent to removing approximately 57.5% of the data. These figures describe dataset pruning, not the fraction of false negatives detected during relabeling.

Retrieval and reranking results

The relabeling experiments hold the model and training settings fixed, using the pruned data with its original labels as the baseline. E5-base and Qwen2.5-7B gain approximately 0.7 and 1.4 points in average BEIR nDCG@10, respectively. This comparison measures the effect of changing labels, separate from dataset pruning. The two BEIR evaluations cover 16 and 14 datasets, respectively, so their gains cannot simply be added.

The three treatments differ in how much supervision they preserve. Removing an entire instance also discards its original positives. Removing only the false negatives stops using those relevant passages. Relabeling adds them to the positive set, allowing the model to learn additional associations between a query and relevant passages. For E5-base, the three treatments score 0.506, 0.511, and 0.515 on BEIR, respectively, supporting the choice to retain and correct these labels. The hard negative count ablation also shows that increasing the number of hard negatives after relabeling can still improve performance.

The gains extend to domains not covered during training. On zero-shot AIR-BENCH, E5-base and Qwen2.5-7B improve by approximately 1.8 and 1.7 points, respectively, supporting the interpretation that correcting the data helps generalization across domains. Training a Qwen2.5-3B reranker on the same relabeled data improves its average BEIR score by approximately 0.8 points. The data benefits both retrieval and reranking models.

Direct label correction is not the strongest option in every setting. Cross-encoder distillation for E5-base performs similarly to RLHN on BEIR and reaches 0.401 on AIR-BENCH, compared with RLHN’s 0.386. Distillation remains a strong baseline when the goal is retrieval performance in this setting. RLHN provides relabeled data that can be reused to train different models.

Label reliability and scope

Better training results do not mean that every label is correct. In a human evaluation of 670 query and hard negative pairs, GPT-4o-mini and GPT-4o achieve Cohen’s kappa scores of 0.320 and 0.390 against human judgments, respectively. The more expensive model agrees with humans more often, but the qualitative analysis still finds mistakes. The evaluation samples also come from instances already flagged by RLHN, so they cannot establish how many false negatives were missed across the full training set.

Relevance judgments also depend on the task. RLHN asks whether a candidate can answer the query, using existing positives as a reference. This criterion does not directly apply to ArguAna’s counterargument retrieval task, which is why its labels remain unchanged. Incorrect original positives and ambiguous queries can also lead to relabeling errors. Results on English passage retrieval still need validation on larger datasets, multilingual tasks, and long documents.

Summary

Dataset pruning shows that more training pairs do not necessarily produce a better retriever. RLHN further shows that some mined hard negatives are worth retaining as positives. One selects data sources, while the other corrects instance labels. Together, the experiments support improving training data quality, without establishing that false negatives caused all the harm from the pruned datasets or eliminating errors in LLM judgments.