Design knowledge agents around evidence, not just coding tools

Benjamin Clavié, Mixedbread · Video · 17m 54s · Analysis checked 2026-09-16

Actionable Insights

These are proposed implementation experiments, not commands demonstrated in the talk.

• Create an evidence-based task contract before choosing tools. For an ambiguous research request, write research-plan.md with the decision to support, jurisdiction/date constraints, unknowns, required sources, and a stopping condition. Require every researcher to return a memo containing claim, source_id, page, supporting_excerpt, counterevidence, and uncertainty. This operationalizes the partner–researcher analogy at 11:16–14:20 and the delegation lessons in Anthropic’s engineering report. Evaluate unsupported-claim rate and missed constraints against a single-agent baseline. Do not let a polished memo substitute for opening the underlying evidence, particularly for legal or medical decisions.

• Run a retriever × orchestration ablation, not a vendor bake-off with changing models. Start with a small held-out task set and compare tuned BM25, semantic/hybrid retrieval, and the same retrieval with an orchestrator plus two non-overlapping research roles. Hold the answering model, corpus snapshot, and total token allowance constant; also report an unconstrained run separately if useful. Use the BrowseComp-Plus custom-retriever guide for a reproducible text benchmark. Record correctness, evidence recall, tokens, calls, and wall time. Promote a configuration only if the improvement survives repeated runs and its cost per correct answer is acceptable; otherwise extra agents may merely be buying extra compute.

• Diagnose the PDF failure before replacing the whole stack. Assemble a test set covering tables, scans, multi-column text, and ordinary prose. For each missed answer, ask whether the source page was absent, OCR lost the relevant relation, or the answerer misread correct evidence. Compare OCR+BM25 with page-aware retrieval, preserving document/page provenance. Mixedbread’s Python SDK and quickstart are concrete options referenced by the creator’s ecosystem; the official site documents pip install mixedbread, but authentication and an existing store are prerequisites. Use approved, non-sensitive sample documents first. Evaluate page recall and numeric answer accuracy separately: better retrieval does not establish better table interpretation.

• Expose distinct search primitives with explicit routing rules. Describe lexical search as useful for identifiers and exact wording, semantic search for paraphrases, and page inspection for visual or structural evidence. Add tool-contract test cases such as the speaker’s “30 days” ambiguity: the agent must preserve whether it means a deadline, retention rule, or grace period. Mixedbread offers MCP and skills documentation, but benchmark your tool descriptions rather than assuming installation solves routing. Log chosen tool, intended evidence, filters, and whether the result resolved the question. Treat retrieved documents as untrusted content, not instructions.

• Measure the oracle gap without confusing it with all errors. Use BrowseComp-Plus’s evidence judgments or your own reviewed labels. On a labeled subset, give the same answerer the known evidence, then compare with normal retrieval. Save both runs and compute oracle_accuracy - retrieved_accuracy; measure actual billed input/output tokens and search charges rather than inferring savings from call counts. Use a human oracle as a separate reference, not a clean retrieval-only ceiling. This distinction matters for the 14:20–14:51 claim: a 3.5-point improvement from 88.9 toward 99.4 closes about one third of that gap, not automatically 40% of all mistakes.

Core thesis

A coding assistant is one kind of knowledge worker, not a universal template for all knowledge work. Open-ended document research needs intent clarification, appropriate evidence retrieval, decomposition, and synthesis. Better tools and better organization should be designed together.

The most useful idea is not “never use grep.” It is to identify which stage is limiting task quality before changing the architecture. A legal research task may fail because the question was framed incorrectly, because a page was not found, because a table was parsed incorrectly, or because correct evidence was interpreted badly. These failures demand different interventions.

Key moments and interpretation

• 3:40–5:42: Code has identifiers, paths, and explicit task boundaries; a phrase like “30 days” in documents needs context. Good motivation for metadata and intent-aware evaluation, but not proof that all software tasks are narrow or all non-code work is harder.
• 7:13–8:44: Tools and specialization form a feedback loop. The operational lesson is that cheaper evidence acquisition can make previously uneconomic workflows worthwhile; the historical examples are illustrations, not measured evidence.
• 9:14–10:46: The speaker contrasts optimized and unoptimized BM25 and emphasizes calls/cost as well as accuracy. This is a fair-baseline argument, not evidence that lexical retrieval is inherently inadequate.
• 11:16–12:18: The law-firm analogy separates problem framing from delegated research. Role boundaries and source-bearing memos are the transferable engineering mechanism.
• 12:48–14:51: The transcript renders the PDF benchmark name poorly; external evidence identifies the relevant benchmark as MADQA. The reported improvements motivate modality-aware retrieval and orchestration, but exact slide configurations were unavailable.
• 15:52–16:55: Teach the agent which primitives exist and organize work around finite context. This is more defensible than treating any one retrieval method as universally sufficient.

Deep research and claim-by-claim Verdict

1. Knowledge work needs a different design from a generic coding-agent workflow

Verdict: agree with the design principle; mixed on the categorical contrast. Confidence: high on the need for task-specific design, medium on broad domain comparisons.

Anthropic’s multi-agent research report independently supports orchestrator–worker research and explicit delegation. It also says dependency-heavy tasks and shared-context requirements are poor fits for parallel agents. This supports selective specialization, not copying a human organization chart wholesale.

A useful counterpoint is DR-DCI, Lu et al. (2026): retrieval expands a local workspace, then shell-based operations support cross-document filtering and verification. The paper reports 71.2% BrowseComp-Plus accuracy, or 73.3% with workspace-preserving context resets, in its own configurations. Coding-style tools can therefore remain useful inside knowledge agents; these results are not directly comparable to the talk’s differently configured leaderboard.

Practical takeaway: keep shell/file inspection where it helps, but add evidence routing and task framing. The overclaim would be to reject coding-agent primitives simply because the task is not programming.

2. Better retrieval can improve accuracy and make workflows economical

Verdict: agree directionally; disagree with equating fewer calls to proportional savings. Confidence: high.

The BrowseComp-Plus repository describes a fixed corpus of roughly 100,000 documents and 830 queries, with both end-to-end and retrieval-only evaluation. This corrects the talk’s inconsistent corpus-size descriptions and supports controlled comparisons.

Mixedbread’s March 24 report reports 90.48% with a get_document scaffold versus 80.00% in a standard scaffold for its search system. That is vendor-reported evidence that the interface matters, not an independent replication or the same numbers as the talk. Its OfficeQA comparison reports fewer calls and lower latency under a fixed Codex setup, while acknowledging imperfect isolation of retrieval effects.

The talk’s “20% fewer calls” does not imply 20% fewer tokens or dollars: result lengths, model reasoning, caching, search fees, and coordination overhead can differ. Likewise, calling a small score change run variance requires repeated-run uncertainty estimates, not visual proximity alone.

Practical takeaway: measure cost per correct, cited answer and retain uncertainty intervals. Stronger retrieval is a hypothesis to test, not a guaranteed budget reduction.

3. Multimodal evidence can overcome OCR/text-only bottlenecks

Verdict: agree, with task-dependent magnitude. Confidence: high on the failure mechanism; medium on vendor-specific gains.

The MADQA paper, Borchmann et al. describes 2,250 human-authored questions over 800 PDFs and an accuracy–effort protocol. It finds that matching human aggregate accuracy can hide different solved questions and inefficient search loops. This supports the talk’s concern about navigation but cautions against treating equal scores as equal capabilities.

Mixedbread’s OCR study reports NDCG@5 of 0.738 for Azure OCR+BM25, 0.773 for ground-truth text+BM25, and 0.865 for its multimodal store on its OHR evaluation. These are vendor measurements of retrieval quality, not final-answer accuracy. The same article cautions that direct vision-only answer generation was unreliable in its tested setting—important counterevidence to “just use vision.”

The March Mixedbread post describes a 500-question MADQA evaluation, whereas the paper abstract describes 2,250 questions. These counts must not be silently merged; evaluation subset/version details need reconciliation before reproducing a leaderboard comparison.

Practical takeaway: preserve original pages alongside OCR, and evaluate retrieval and reading separately. Text search over extracted PDF content remains valid; a PDF is not intrinsically unsearchable with BM25.

4. Delegated research closes a large fraction of the remaining gap

Verdict: mixed. Confidence: medium on orchestration helping; low on the exact talk percentage.

The transcript reports 88.9% agent accuracy, a 99.4% human-oracle reference, and a 3.5-point gain. Taken literally, the gap is 10.5 points and the gain closes 33.3%; total error falls from 11.1 to 7.6 points, a 31.5% relative reduction. The speaker’s approximate “about 40%” is not supported by those exact stated inputs. Captions or slide rounding may explain part of the discrepancy; no frame was available to resolve it.

Anthropic supports the mechanism but reports roughly 15× chat token consumption for its multi-agent systems and notes that token budget explains much performance variation. Its comparison is to chats, not a universal 15× penalty relative to single-agent research. Mixedbread’s March report shows a different 88.2% one-shot and 91.7% hybrid system comparison with different models/harnesses; it does not independently verify the talk’s exact ablation.

Practical takeaway: compare matched budgets and the same answering model. A human-oracle gap conflates retrieval with differences in human and model reasoning.

Comment insights

No comments were extracted. A dedicated top-comment retry also returned zero. There is no audience agreement, disagreement, practitioner addition, or social proof to infer; this means none were available to this run, not proof that nobody has commented.

Screen-level insights and evidence limits

Both frame metadata and the frame directory yielded no usable images. A retry with an explicit Node JavaScript runtime still failed with HTTP Error 403: Forbidden. No visible UI, code, slide labels, or diagram detail has been verified.

Transcript-linked visual references, not frame observations:

• At 9:14–10:46, the speaker describes a leaderboard comparing retrieval configurations. Its purpose is to connect baseline tuning to accuracy and call count; exact labels and cells cannot be certified.
• At 11:16–12:18, he describes a doodled client/partner/researcher workflow. The relevant step is decomposition followed by research memos, not an observed software demo.
• At 12:48–14:51, he discusses a PDF benchmark and oracle-gap comparison. Unavailable slides prevent reconciliation of the captioned figures with the external report.

This analysis does not satisfy direct visual verification; the limitation is retained rather than manufacturing screen evidence.

Recommended workflow and why it matters

Frame the decision → identify required evidence → run an economical lexical baseline → inspect retrieval versus interpretation failures → add multimodal tools or decomposition where justified → synthesize source-bearing memos → audit claims → release only when quality and cost gates pass.

My read: the valuable change is moving from “give the model more tools” to “design an accountable evidence workflow.” Both retrieval and coordination can help, but neither replaces source checking or domain judgment.

Verification notes

Four separate review-role passes were performed locally: source/evidence, transcript/comment/frame fidelity, hallucination/overclaim, and Actionable Insights. No independent reviewer agents were available. Sources include the full extracted caption sequence, an empty comment retry, frame-failure metadata, primary benchmark repositories/papers, Anthropic’s engineering account, and explicitly labeled Mixedbread reports. Corrections include MADQA naming, corpus-size inconsistency, the oracle-gap arithmetic, call-count versus monetary savings, and non-comparable benchmark configurations. The actionable audit checked first steps, links, metrics, prerequisites, and privacy/integration cautions; recommendations are explicitly distinguished from demonstrated commands. Web search was unavailable, but direct primary-source fetching worked. No experiments were executed; exact slide values, visual workflow, and the reported orchestration ablation remain unverified.