How We Solved Agent Building — Andrew Qu, Vercel

AI Engineer17:34Transcript availableAdded Sep 15, 11:51 pm GMT+8

Watch the talk · 17m 34s · Analysis and external documentation checked 15 September 2026.

Bottom line: The useful lesson is not that agent building is solved. It is that a well-documented semantic layer, discoverable files, and a small tool surface can outperform a brittle chain of specialist agents. The strongest quantitative evidence is still a vendor-reported comparison on only five questions. Treat this as an architecture experiment worth reproducing, not a general reliability guarantee.

Actionable Insights

These are recommended experiments, not a claim that the following implementation was executed during this review.

Run a controlled tool-surface ablation before rewriting your agent. Compare your current pipeline with a filesystem explorer plus a narrowly scoped SQL execution tool. Start from the bash-tool repository and AI SDK; the repository documents npm install bash-tool just-bash for its lightweight setup. A real VM uses a different sandbox backend; do not assume the emulated shell and a full operating system have equivalent capabilities. Hold model, semantic-catalog snapshot, permissions, questions, and budgets constant. Use a held-out question set covering ambiguous metrics, join fan-out, empty results, expensive queries, and unauthorized requests. Score answer correctness against analyst-reviewed results, not merely SQL execution; also record tokens, p95 latency, warehouse cost, and refusal correctness. Vercel’s five-query result 1 is a reason to test this, not a release threshold.
Make company semantics inspectable before adding autonomy. Create a small reviewed catalog such as catalog/metrics.md, catalog/joins.yaml, and catalog/examples/; these names are suggestions, not required Vercel conventions. Document metric grain, time zone, valid join keys, exclusions, and an owner for each definition. Try grep -R -n 'net_revenue' catalog/ and verify a human can find the authoritative definition and its dependencies without tribal knowledge. Mount only the permitted catalog into the agent workspace and load relevant files on demand. Evaluate against independently calculated answers on a fixed data snapshot. Vercel explicitly credits its existing semantic documentation, while Snowflake semantic views provide an alternative place to centralize these definitions. Do not maintain divergent definitions in both systems without synchronization checks.
Preserve authorization while simplifying reasoning. Use a disposable sandbox for exploration and a separate, service-enforced SQL boundary with a least-privilege database role, query timeout, result limits, and audit logging. Keep warehouse credentials out of files the agent can inspect. Test attempted writes, cross-tenant reads, and malicious instructions embedded in catalog text before connecting real users. Passing syntax validation is not authorization, and a prompt asking for read-only behavior is not a permissions mechanism. The example in 1 still contains ExecuteSQL alongside command execution despite its “single tool” headline; removing redundant schema wrappers need not mean removing security controls. This boundary design is my integration recommendation, supported by Anthropic’s sandbox/guardrail guidance, not a verified description of d0’s complete security implementation.
Turn successful recurring work into reviewed skills, not automatic policy. The talk describes a recurring job distilling recent queries into roughly 100 skills at 10:23–10:53. Start with one frequent analysis: capture triggering questions, authoritative metric definitions, parameterized query examples, expected outputs, and known counterexamples in a versioned skill. bash-tool’s skills examples show packaged SKILL.md usage; skills.sh is the directory mentioned in the talk, not an endorsement of every listed package. Redact customer data from traces, require analyst review, and re-run the held-out suite with the skill enabled and disabled. Promote only if correctness is preserved and cost or latency improves; expire skills when schemas change. The bash-tool skill API shown in the current README is explicitly experimental.
Try eve in an isolated pilot, with current prerequisites. The official getting-started guide specifies Node.js 24+, npm, and a model credential. Its documented scaffold command is npx eve@latest init my-agent, followed by npm run dev inside the project. Review the generated dependencies and lockfile before production use. Current authored paths include agent/instructions.md, agent/tools/, agent/skills/, and agent/channels/; sandbox seed files belong under agent/sandbox/workspace/, and evals live alongside agent/. Use eve info to diagnose discovery. Start with a sanitized, read-only question and verify the loaded files, tool trace, and final answer. eve is currently beta; test recovery after interruption, authorization on channels, and sensitive-data handling rather than equating a successful scaffold with production readiness. These commands and paths come from current docs, not a transcription of the slide.
Benchmark a boring alternative alongside the agent. Following the strongest audience objection, answer your ten most frequent questions using reviewed analytical views and parameterized reporting. Compare that baseline with a filesystem agent and, where applicable, Snowflake’s semantic-view/Cortex workflow using the same business definitions. Track analyst interruption time as well as answer quality and total operational cost. Use deterministic reporting for stable requests and reserve exploration for questions it demonstrably handles better. “Build your own” should mean owning the semantics and evaluation, not automatically owning every layer of infrastructure.

Core thesis and architecture journey

Qu describes d0, Vercel’s internal data-analysis agent, progressing through five designs:

Schema pasted into a prompt: a quick feasibility test, with SQL copied and executed manually.
Specialist chain: planning, querying, execution, and reporting separated into stages with dedicated tools. This reduces manual work but restricts recovery and loses detail at handoffs.
One agent with multiple modes: a shared history lets it revisit schema exploration after errors. Trusted-user feedback exposes a large gap between internal expectations and real questions.
Filesystem-oriented agent: a sandbox contains the semantic layer; familiar command/file tools let the model retrieve context and work iteratively. The talk credits Claude Code and Opus 4.5 as the inspiration, then describes a purpose-built implementation.
Reusable skills and eve: recurring query patterns become reusable procedures, while a filesystem-convention framework packages tools, skills, channels, and runtime infrastructure.

My interpretation: the durable asset is a governed semantic corpus plus feedback from real questions. The execution harness should remain replaceable. The talk’s shift from “mega context” to discovering files is important: putting files in a sandbox is not the same as loading every file into the model’s context.

Best timestamped moments

3:46 — Manual schema-to-SQL prototype. Good feasibility testing; valid SQL alone does not establish business correctness.
4:48 — Specialist chain. The useful question is whether stage boundaries prevent an execution error from triggering better retrieval.
5:48 — Shared-state agent. Qu identifies information loss in summary-only handoffs, not a proof that all multi-agent systems are inferior.
6:48 — Trusted users dislike a system its builders thought was strong. This is the clearest argument for representative evals.
7:18 — Approximately 30% on their evals is mentioned before the filesystem transition. The denominator and rubric are not supplied.
7:48 — Filesystem access described as the unlock. “Basically AGI” is comparative enthusiasm, not evidence of general intelligence.
8:52 — Semantic layer placed inside a sandbox. This changes context access, not the need for permissions.
9:22 — Eval score reportedly doubles. Do not combine this with the separate blog’s 80%→100% comparison.
10:23 — Repeated questions distilled into skills. Useful organizational memory, but stale or erroneous recipes need review.
12:24 — eve introduced as filesystem conventions for agents. Packaging is the contribution; it is not an accuracy proof.
14:30 — Customer-success and observability claims. Without a disclosed evaluation, the success comparison remains a testimonial.
15:31 — Company-specific knowledge emphasized over generic products. A strong argument for context ownership, but not necessarily for building the whole stack.

Deep research and claim-by-claim verdicts

1. Fewer, more general tools can outperform a specialist architecture

Supporting evidence: Qu’s December 22, 2025 Vercel article reports five representative questions: 4/5 versus 5/5 successes, 274.8 versus 77.4 seconds average execution, roughly 102k versus 61k tokens, and about 12 versus 7 steps. Its description identifies a Cube semantic layer in YAML, Markdown, and JSON. This is a concrete implementation account, but it is from the same author and organization, not independent replication.

Counterweight: Anthropic’s agent guidance recommends simple designs but also identifies cases where fixed workflows, routing, parallelism, and orchestrator-workers are appropriate. Their context-engineering article 2 warns that tool ambiguity and unnecessary context hurt performance, while describing retrieval and multi-agent approaches as complementary options.

Verdict: agree conditionally; high confidence in the design principle, low confidence in the magnitude or generality of the reported gain. Five questions cannot establish universal reliability. The talk also changes model/harness across its development story, so it does not isolate filesystem access as the sole cause. The practical takeaway is an ablation with a fixed model, not wholesale removal of specialized tools.

Numerical correction: 80%→100% is a 20-percentage-point increase, or 25% relative improvement. The article’s rounded 102k→61k figures imply approximately 40% fewer tokens, although it labels the change 37%; without raw measurements that discrepancy cannot be resolved. Neither figure should be presented as independently reproduced. Its displayed code is illustrative and contains syntax issues; use maintained package documentation rather than pasting it as runnable code.

2. Filesystem discovery is a better context strategy than hand-built retrieval

Supporting evidence: Anthropic’s context-engineering article describes just-in-time loading through paths, targeted queries, and shell commands, explicitly using Claude Code as an example. It also favors small, non-overlapping toolsets and structured external notes.

Counterweight: The same source says runtime exploration can be slower than precomputed retrieval and describes hybrid designs. It also warns of context degradation as history grows. A directory does not itself solve search relevance, permissions, missing documentation, or memory quality.

Verdict: mixed; high confidence. Discoverable files are an effective option for bounded, well-structured corpora, not a universal replacement for indexing or retrieval. Keep a small authoritative context upfront, retrieve details on demand, and measure search misses and wasted tool calls. “One agent with all the mega context” overstates what should remain in working memory.

3. Automatically distilled skills compound organizational knowledge

Supporting evidence: The talk reports recurring distillation and approximately 100 skills. bash-tool’s current repository supports modular skills with instructions and optional scripts; eve’s docs support on-demand skill slots 7. These establish available mechanisms, not the effectiveness of d0’s learned recipes.

Counterweight: No disclosed controlled measurement separates the skills’ impact from other changes. Trace-derived instructions may preserve incorrect joins, private data, or obsolete business definitions. Those are integration risks, not reported incidents at Vercel.

Verdict: agree with governance; medium confidence in benefit, low confidence in this deployment’s incremental effect. Review and version skills, attach examples and counterexamples, and measure them against an unchanged baseline. The underemphasized requirement is maintenance, not merely accumulation.

4. Company-specific context makes custom agents better than off-the-shelf products

Supporting evidence: Qu describes why customer/web-property relationships matter to Vercel. Snowflake semantic-view documentation independently explains the mismatch between business vocabulary and physical schemas, and how centralized dimensions, metrics, and relationships address it.

Counterweight: That same documentation describes using semantic views with Cortex Agents and the Cortex Analyst REST API. Therefore, supplying company semantics is not unique to a custom filesystem harness. Vercel does not identify the compared vendors, test questions, or operating costs in this talk.

Verdict: agree on context ownership; mixed on build-versus-buy; high confidence. Test hosted and custom options with equal context quality. The stronger insight is that semantic modeling is essential, not that packaged products cannot use it.

5. eve is a portable “Next.js for agents” and the best way to build them

Supporting evidence: eve’s official site describes a filesystem-first, Apache-2.0 framework for durable backend agents, runnable on Vercel or self-hosted infrastructure. Its current guide documents concrete file-discovery conventions, tools, channels, skills, sandbox files, and subagents. This verifies the framework and its documented shape.

Counterweight: The site labels eve beta. Documentation establishes intended capability, not successful migration of a production workload or superiority over other frameworks. Anthropic 3 cautions that frameworks can obscure behavior and encourage unnecessary complexity. The talk’s customer testimonial supplies no reproducible benchmark.

Verdict: agree that the abstraction is concrete; withhold agreement on “best”; high confidence. Pilot it where conventions and runtime integration remove real maintenance work. Verify recovery, data boundaries, observability, and self-hosting requirements for your application before committing.

6. Agents free the data team for higher-value work

Qu reports thousands of daily queries, roughly 20 useful internal agents, and a more productive data team. The blog also describes the aim of reducing analyst interruptions, but offers no independent time study or financial result.

Verdict: plausible but unverified; low-to-medium confidence. Do not translate adoption counts into labor savings. Measure analyst interruption hours, corrected answers, user trust, and total platform/warehouse cost before and after a controlled rollout. The claim that a blog drove 70% of vercel.com traffic is also speaker-reported and unverified here.

Comment insights

The extraction contains seven comments, a small engagement-selected sample, not a representative audience survey.

Novelty skepticism dominates the highest-liked criticism. The 20-like and 7-like comments argue that the techniques are already familiar. That criticism is consistent with the earlier context-engineering literature, but their broad dismissals are not empirical evidence against the implementation. My read: judge the talk as a case study and product introduction rather than a foundational discovery.
The actionable counterproposal is analytical views. A 3-like comment asks why the data team did not build them. This motivates the deterministic-reporting baseline above. It does not prove views cover every exploratory question; ordinary analytical views and Snowflake semantic views are related alternatives, not identical objects.
“JIT Context injection” captures an actual mechanism. Another 3-like comment aligns with Anthropic’s just-in-time retrieval description. Turn that slogan into a measurable retrieval design, not an instruction to load arbitrary content without trust boundaries.
Hypergraph theory is unsupported in this thread. The highest-liked comment proposes it, and a reply asks for resources; no usable implementation or reference appears in the extracted sample. There is insufficient evidence to recommend adding a graph layer.
A “three weeks” delivery estimate and a bare name reference provide no substantiated engineering guidance. Neither is treated as evidence of feasibility, nor expanded into an invented tool recommendation.

Screen-level insights: recovered storyboard evidence

The preview-video download failed and the original frame manifest contained no frames. I recovered 12 actual 320×180 YouTube storyboard tiles from the video’s extracted metadata and visually inspected a contact sheet. Times below are approximate metadata-derived positions, not frame-accurate synchronization; nearby transcript ranges provide the interpretation. These are stage-presentation slides, not a verified live coding session. Small code and UI labels cannot be read reliably.

Approximately 3:46 and 5:25: code slides accompany the prototype and specialist-tool discussion at 3:46–5:48. The visible code layout confirms implementation slides, but exact identifiers come from transcript or external docs, not pixel-level OCR.
Approximately 4:56: the readable heading “More agents, more problems” sits above a horizontal chain of boxes. This matches the staged-agent explanation at 4:48–5:48; visually, the handoff boundaries are the focus.
Approximately 6:24: “One agent, many modes” shows a shared enclosing structure rather than a long chain. Nearby narration explains revisiting earlier work after an execution error. The diagram supports the state-sharing interpretation, not a measured accuracy gain.
Approximately 7:42 and 9:01: “The agent got a workspace” and “File system agent” shift the diagram toward a workspace and connected resources. This corresponds to the 7:48–9:22 explanation of shell/file exploration. The significance is access to retrievable working material, not visible proof of sandbox isolation.
Approximately 9:41: a compact code slide appears near the bash-tool explanation. The API cannot be confidently transcribed from this tile; the maintained repository 4 is the implementation reference.
Approximately 10:50: “Patterns show up” shows connected workflow boxes, matching the recurring-query-to-skills discussion. It illustrates reuse but does not verify the reported count or quality of skills.
Approximately 12:28: a filesystem-style list accompanies the framework proposal. Folder conventions are independently confirmed by 7; the low-resolution slide is not used to certify exact paths.
Approximately 13:27 and 14:16: dark product/example interface screenshots are visible. Their detailed labels and numerical contents are unreadable at this resolution; they cannot substantiate the later customer-success or observability claims.
Approximately 14:46: the slide clearly displays https://eve.dev. That readable destination was fetched directly and verified against current documentation.

Verification notes

Four distinct post-draft review passes were performed in dedicated auditor roles within this run; these were self-review passes, not independently spawned agents.

Source/evidence audit: checked the same-author five-question benchmark against Vercel’s article, and compared the architectural claims with Anthropic, Snowflake, eve, and bash-tool documentation. Sources establish mechanisms and documented capabilities; they do not independently replicate d0’s gains.
Transcript/comment/frame fidelity audit: used the full extracted transcript (final substantive chunk starts at 17:04), all seven entries in comments_top, the failed preview manifest, and 12 recovered storyboard images. Retained approximate frame timing and avoided claims of legible code or a live demo.
Hallucination/overclaim audit: separated the talk’s 30%/doubling statements from the blog’s 4/5→5/5 result; flagged the token-reduction arithmetic discrepancy, same-author evidence, unverified adoption/productivity figures, and beta status. Normalized obvious caption errors such as Vercel, AI SDK, Claude Code, and skills.sh using the external references.
Actionable Insights audit: checked that each recommendation has a concrete first step, relevant tool/source links, an evaluation criterion, and an integration caution. Current setup commands were verified against docs; suggested catalog paths and security tests are explicitly recommendations, not video quotations. No installation, framework benchmark, warehouse experiment, or production security test was run for this analysis.

Residual uncertainty: no raw d0 evaluation set, independent replication, controlled productivity study, or full-resolution video inspection was available. Documentation was read directly from known primary sources; this was not an exhaustive web-wide literature search. Some retrieved documents were length-limited, and only inspected passages support the claims above.

Sources

Andrew Qu / Vercel — We removed 80% of our agent’s tools, December 22, 2025. Same-author implementation and five-query benchmark.
Anthropic — Effective context engineering for AI agents. Just-in-time retrieval, tool design, hybrid context strategies, and context limits.
Anthropic — Building effective agents. Simplicity, workflow trade-offs, evaluation, and guardrails; the page notes its original December 2024 tooling discussion has evolved.
vercel-labs/bash-tool repository. Current install command, sandbox options, AI SDK compatibility notes, and experimental skill support.
Snowflake — Overview of semantic views. Business definitions, modeled relationships, and integrations with Cortex workflows.
eve official site and source repository. Declared framework positioning, license, and beta status.
eve — Getting Started. Current prerequisites, scaffold commands, filesystem conventions, and discovery diagnostics.