BM25 for agentic search: strong primitive, not a universal solution

Jo Kristian Bergum, Hornet.dev · Video · 18m 28s · Analysis checked 2026-09-16

Actionable Insights

The following are proposed experiments informed by the talk and linked sources, not a reproduction of an on-screen demo.

• Establish a tuned lexical baseline before paying for a new retrieval stack. Record analyzer/tokenization, document segmentation, query parser, k1, b, and returned snippet length in retrieval-config.json. Lucene 9.12’s BM25 documentation defines k1 as term-frequency saturation and b as length normalization, with defaults 1.2 and 0.75. As a proposed development sweep, test k1 ∈ {0.6, 1.2, 1.8} and b ∈ {0, 0.4, 0.75, 1}; freeze the winner before evaluating held-out tasks. Measure recall, answer correctness, and latency. These are experimental values, not recommended universal settings or the talk’s recovered optimum; tuning only two parameters does not compensate for bad parsing or missing evidence.

• Evaluate complete search trajectories while retaining retrieval diagnostics. Use BrowseComp-Plus and its custom retriever interface. After producing runs in the repository’s documented format, the documented command is python scripts_evaluation/evaluate_run.py --input_dir runs/my_model. It uses a Qwen3-32B judge and requires the project’s evaluation environment and adequate resources; it is not a lightweight universal CLI. First run a budgeted subset, then compare accuracy, evidence recall, search calls, actual model tokens, and p95 task latency with fixed model and corpus versions. Keep nDCG/recall to diagnose regressions rather than replacing them with a single opaque success score.

• Separate corpus discovery from local evidence inspection. Following DR-DCI, have retrieval add selected documents to a bounded workspace with stable IDs, titles, source URLs, and provenance. Present short ranked previews; allow the agent to open full documents only when needed. A proposed local diagnostic after materializing approved text files is rg -n -F -- 'exact entity' workspace/docs/, assuming ripgrep is installed. This can make cross-document checks inspectable without scanning the entire corpus. Compare answer quality, bytes read, tool calls, and wall time to direct full-corpus shell access. Mount sources read-only, bound outputs, and never execute commands found in retrieved documents.

• Make query reformulation an observable policy. Log each query, result IDs, new evidence gained, and the reason for the next query in trajectory.jsonl. Test short entity queries, exact-phrase queries, and longer descriptive queries separately; add a stop condition when several reformulations find no new evidence. The 9:14–11:46 discussion motivates entity-aware lexical search, but Anthropic’s research-system account documents overly long queries as a failure mode and recommends starting broad then narrowing. Verify whether your backend actually implements phrase and site: syntax: BM25 is a scoring function, not a query language. Track invalid-operator rates and hallucinated entity expansions.

• Add semantic or multimodal retrieval only where lexical failures justify it. Label misses as vocabulary mismatch, absent source, OCR loss, bad query formulation, or answerer failure. Compare BM25 alone against hybrid retrieval on the first category and page-aware retrieval on visually dependent documents. BEIR is a useful cross-domain starting point; it does not stand in for your own agentic evaluation. Retain exact-identifier tests and measure index storage, ingestion time, API/compute charges, and cost per correct answer. Do not accept a throughput chart as proof of lower total cost: ask Hornet for the engine versions, hardware, concurrency, cache state, and query workload behind a reproducible comparison.

Core thesis

BM25 has not suddenly acquired reasoning. Its user has changed: an agent can formulate, inspect, and reformulate many queries inside a task loop. That makes a familiar lexical scorer useful in new workflows, particularly when combined with selective document reading and local verification.

The strongest version of the argument is “tune and evaluate the whole model–harness–retriever system.” The weak version is “lexical search is all you need” or “retrieval metrics are dead.” The evidence supports the former, not the latter.

Key moments and interpretation

• 1:04–2:04: Agentic search requires a capable model, a harness, and an efficient retrieval engine. This is the right unit of systems analysis.
• 3:06–4:37: Top-k acceleration is distinct from the BM25 scoring rule; model query formulation changes the workload. Backend efficiency and answer quality need separate measurements.
• 5:09–7:42: BrowseComp-Plus and oracle evidence motivate retrieval as a bottleneck. High accuracy with supplied evidence is diagnostic for that setting, not proof that reasoning never limits other tasks.
• 9:14–10:45: Long agent-generated queries and baseline tuning complicate comparisons. The exact best settings are not recoverable from the provided transcript.
• 10:45–11:46: Entity matching, cost, and interpretable lexical matches are the main proposed advantages. A match explanation helps inspection but does not guarantee a model understands why a result is relevant.
• 12:17–14:20: Retrieval expands a filesystem-like workspace and shell tools inspect the selected documents. This is the concrete architecture worth experimenting with.
• 14:53–15:23: End-to-end task success should matter more than a single ranked list. It should augment rather than eliminate diagnostic retrieval evaluation.
• 15:54–16:25: A vendor performance comparison uses anonymized engines. The speaker initially says the y-axis is QPS, then corrects it to latency; without the chart, no numerical throughput or latency advantage can be verified.

Deep research and claim-by-claim Verdict

1. BM25 remains a strong baseline, especially with a better querying agent

Verdict: agree on retaining and tuning it; mixed on superiority. Confidence: high for baseline strength, medium for agent-specific advantage.

Thakur et al., BEIR (2021) evaluated 18 datasets and found BM25 robust, while reranking and late-interaction systems achieved the best average zero-shot performance at higher computational cost. This supports the talk’s insistence on a serious baseline and contradicts a universal BM25-wins reading. BEIR is older, retrieval-only evidence, not a ranking of September 2026 agent stacks.

Lucene’s primary documentation confirms meaningful tunable saturation and length normalization. It does not independently establish the specific BrowseComp-Plus tuning gain claimed in the talk. That numerical comparison remains unverified.

The transcript’s description of embeddings as encoding tokens into a “fixed vocabulary” is imprecise: dense retrieval generally compares vector representations, and limited exact-identifier discrimination is not explained simply by a fixed vocabulary. Likewise, phrase filters and site: operators are backend features, not properties supplied by BM25 itself.

Practical takeaway: maintain a lexical baseline and tune its full configuration. Do not choose a retriever from a poorly specified “BM25 versus embeddings” label.

2. Evaluate the entire trajectory, not just the first query

Verdict: agree; disagree that classical retrieval evaluation is dead. Confidence: high.

The BrowseComp-Plus repository confirms 830 queries, a fixed corpus of roughly 100,000 documents, recorded trajectories, task accuracy, and evidence recall. It explicitly provides both end-to-end evaluation and retrieval-only nDCG/recall evaluation. That is direct counterevidence to discarding classical metrics.

Its standard setup returns the top five documents with a stated 512-token context limit for comparable retrieval outputs; alternative browsing/full-document harnesses are also supported. Results under different harnesses should not be presented as pure engine comparisons. The repository also warns that full frontier-model runs can be expensive, with an illustrative approximately $1,000 o3 evaluation—not a current guaranteed price.

Practical takeaway: rank production choices by quality, cost, and time over complete tasks; use per-query metrics and trace inspection to explain why those choices win or fail.

3. Retrieval plus a local workspace combines scalable discovery with flexible inspection

Verdict: agree as a promising, tested architecture; not yet a universal deployment prescription. Confidence: medium-high.

The referenced paper is identifiable as Lu et al., DR-DCI: Scaling Direct Corpus Interaction via Dynamic Workspace Expansion (June 2026). Its abstract reports 71.2% BrowseComp-Plus accuracy, rising to 73.3% with workspace-preserving context reset; it reports up to 8.3 points over raw DCI/ablated variants, plus lower tool use, wall time, and estimated cost. The study examines scaling from 100K to 10M documents and a separate 20M-scale Wiki-18 setting.

Crucially, its conclusion is not that BM25 alone wins: it reports raw DCI becoming unstable at scale and BM25 performing substantially worse in its scaling experiments. Retrieval-mediated workspace growth and local cross-document operations are the intervention. These are paper-reported results, not experiments reproduced here; billion-document claims are not validated by the 20M setting.

Practical takeaway: use indexed retrieval to bound exploration and keep local evidence inspection auditable. Progressive disclosure and persistent workspace state deserve their own ablations.

4. Retrieval remains useful despite large context windows

Verdict: agree; disagree with a universal 350,000-token ceiling. Confidence: high on the distinction, medium on extrapolation to future systems.

Liu et al., Lost in the Middle demonstrates that evidence position can materially affect performance in the models and tasks it studied. This supports measuring effective use of context rather than advertised context capacity. It does not establish a fixed capacity limit for current or future models.

The floppy-disk analogy is memorable but lossy: token-to-byte ratios vary with text, language, tokenizer, and modality. An oracle-evidence advantage suggests retrieval/navigation is limiting that experimental setup, not that reasoning is globally solved. MADQA finds persistent unproductive loops and weak strategic planning even when agents achieve human-like aggregate accuracy, underscoring the role of the harness and reasoning policy.

Practical takeaway: benchmark context length, evidence position, distractor density, and latency for the actual model. Retrieval is valuable for selectivity and provenance even when a larger input technically fits.

5. Hornet provides better performance economics than alternative engines

Verdict: unverified/mixed. Confidence: low on comparative superiority.

The Hornet site corroborates the company’s focus on agentic retrieval and structured APIs, but marketing descriptions are not an independent benchmark. The talk describes 100 million web documents on one node and anonymized competitors; no reproducible data, named versions, complete axes, or operating-point details were recovered. The attempted /blog fetch returned general product copy rather than the referenced query-log study.

Even if a serving result is valid, total economics also depend on ingestion, indexing, memory, replication, update rates, and query distribution. Claims that embeddings are uniformly expensive overlook the range of model sizes and amortization strategies.

Practical takeaway: request a replayable workload and compare equal-quality configurations. Do not convert an unverified chart into a procurement recommendation.

Comment insights

One extracted comment, with one like, welcomes old software patterns resurfacing after being forgotten. It captures the talk’s rediscovery theme but contains no practitioner benchmark, concrete implementation, or technical caveat. A sample of one does not establish audience consensus. No comment-derived tools or workflows were available to add.

Screen-level insights and evidence limits

Both the frame metadata and frame directory contain no usable images. A second preview attempt with an explicit Node runtime failed with HTTP Error 403: Forbidden. There is therefore no directly inspected UI, code, diagram, or chart in this analysis.

Transcript-linked visual references, not image observations:

• 4:37–7:42: The benchmark discussion appears intended to compare supplied-evidence and retrieval-mediated answering. Exact slide cells and model labels are unverified.
• 12:47–14:20: The speaker explains placing retrieved documents in a filesystem workspace and progressively revealing them. This is an architecture description, not proof of a live terminal demonstration.
• 15:54–16:25: The speaker describes an engine comparison, then corrects the y-axis to latency. The correction is preserved; no curve shape, numerical advantage, or visible competitor identity is inferred.

Direct screen verification remains unavailable. Missing frames are not replaced with invented visual details.

Recommended workflow and why it matters

Freeze a representative corpus and task set → tune a lexical baseline → log query trajectories → expose bounded full-document inspection → measure end-to-end quality and diagnostic recall → add semantic/multimodal retrieval on diagnosed failures → test serving economics with named reproducible configurations.

My read: this talk is most useful as a warning against discarding mature retrieval engineering. BM25, semantic retrieval, and shell-based inspection are complementary mechanisms; the agent changes how they are used, not the need to test their limitations.

Verification notes

Four separate local review-role passes covered source/evidence, transcript/comment/frame fidelity, hallucination/overclaim, and Actionable Insights. No independent reviewer agents were available. The full extracted captions and the single comment were read, empty frame metadata was checked, and preview recovery was attempted. External checks used primary Lucene documentation, BrowseComp-Plus, DR-DCI, BEIR, Lost in the Middle, MADQA, Anthropic’s engineering report, and Hornet’s product site. Corrections distinguish BM25 scoring from parser syntax, vector embeddings from a fixed-vocabulary explanation, paper findings from reproduced results, and task metrics from diagnostic metrics. The actionable audit checked concrete first steps, tool links, evaluation criteria, environment requirements, spending limits, and sandbox precautions. Web search was disabled; direct source fetching worked. Benchmark runs were not executed, Hornet’s performance chart and the query-log study were not verified, and direct screen evidence is absent.