Agents That Write Their Own Tools at Runtime — Sandhya Subramani, AWS

AI Engineer20m 40sTranscript availableAdded Oct 16, 9:52 pm GMT+8

Actionable Insights

Reproduce runtime tool loading with a harmless local function before giving an agent a shell. Read the Strands load_tool implementation, write a pure character-counting function with @tool, and load it in a disposable environment. Test empty input, ASCII text, and Unicode against explicitly chosen semantics. Python string length and user-perceived characters need not be the same. Success means a trace shows the new function being called in the existing session and producing the expected results, not merely a plausible chat response.
Separate generated files from approved tools. My suggested layout is generated-tools/, approved-tools/, and tests/, with a manifest recording the source hash, dependencies, tests, and approval decision. Keep the generation directory outside any automatically watched tool directory. Inspect and test the file before promotion. Loading Python executes top-level code, so reviewing only the decorated function is insufficient. Confirm the behavior of your pinned SDK version rather than assuming the talk’s directory-watching example matches your installation.
Limit the execution environment before testing self-repair. Use a disposable workspace without production credentials, allow only necessary network destinations, and make external data read-only initially. The guardrail slide at 17:27 explicitly includes network controls, scoped identity, and human approval for high-impact actions. Test a denied write and a denied network request. The agent should report the restriction rather than generate another tool to evade it.
Benchmark against a fixed tool set. Run the same tasks with a reviewed calculator or parser and with runtime generation enabled. Record completion rate, verified correctness, latency, tokens, and the number of repairs. The demo’s arithmetic and character-counting tasks do not by themselves justify generated code. Adopt generation only where the task variation produces a measurable benefit over a small library of existing tools.
Add an approval boundary around external effects. For a pipeline-repair prototype, let the agent propose a replay or deletion against fixture data first. Require a preview of affected records, an idempotency rule, and an explicit authorization check before a real write. Log tool source hashes, arguments, outputs, and failures without secrets. These are proposed implementation controls responding to the talk and audience questions, not a production system demonstrated on stage.

Core thesis

Subramani demonstrates an agent that writes and loads new tools during a running session. She then applies the same pattern to agents exposed as tools. An editor creates code. A shell inspects or operates in the environment, and a loader registers the code for invocation.

Runtime extension is distinct from safe autonomous repair. The demonstrations show that an agent can add executable behavior. They do not establish that the behavior is correct, that it repairs a production fault, or that it has authority to expand the product’s scope.

Key ideas and timestamped moments

1:32–3:09: The initially small agent creates a calculator and calls it without restarting the conversation. The important event is registration and invocation of a new tool.
3:39–4:39: A character counter provides a simple second example. The speaker herself notes the need for input whose answer can be checked. Output alone is not verification.
7:40–9:13: The prompt specifies a tool template and storage location. The editor, shell, and loader supply the actual executable capabilities. Calling this an agent with no capability understates those permissions.
9:44–10:14: International flight booking is a hypothetical reason to extend a domestic-only agent. New code cannot supply a missing commercial agreement, API entitlement, or user authorization.
11:46–13:51: The itinerary demo creates specialized agents. At 13:20 the speaker acknowledges the absence of real-time information access. This is planning text, not verified current flight availability.
15:25–16:56: Evaluation covers the end goal, response grounding, tool choice, parameters, and inter-agent communication. These are different failure modes and need separate checks.
17:27–18:31: Sandboxing, constrained tools, permissions, and observation qualify the earlier enthusiasm for autonomy.
19:31: The claim that Python Strands wrote the TypeScript version is an attribution to the speaker. The sources checked here confirm Python and TypeScript support, not that development history.

Deep research and claim-by-claim Verdict

Runtime loading is a real mechanism

The Strands loader source describes loading Python files without restarting and implements registration through tool_registry.load_tool_from_filepath. It supports decorated tools and the older TOOL_SPEC pattern. This is stronger evidence for the mechanism than a slide alone.

The same source states that loading executes the file’s top-level code. It prompts for consent in its ordinary interactive path and has a configuration option to disable loading. Its documentation also describes automatic loading from the working directory’s tools/ folder. Those paths need separate testing in the SDK version actually deployed. A file-existence check and a tool schema do not certify code safety.

Verdict: agree. Confidence: high for runtime extension, not for production correctness. Keep the loader behind a trust decision. Do not use an automatically watched directory as an unreviewed staging area.

A few tools provide a small implementation, not a small permission set

The demo’s editor, shell, and loader can create and run arbitrary behavior within their effective permissions. This follows directly from the loader’s execution semantics. The current Strands repository documentation supports model providers, tools, lifecycle controls, tracing, and multi-agent patterns. It does not turn unrestricted code execution into a narrowly scoped application function.

Verdict: mixed. Confidence: high. The construction is compact and useful for experimentation. The claim that the initial agent has effectively no capability is misleading because general execution tools are powerful capabilities. Evaluate the filesystem, network, process, and credential permissions, not the number of entries in the tool list.

Sandboxing enables safer experimentation but does not guarantee safe external effects

The inspected guardrail slide is more specific than a generic instruction to be careful. It calls for network egress controls, read-only defaults, typed schemas, timeouts, retries, rate limits, secret isolation, scoped identity, per-action authorization, and human approval for high-impact actions.

That supports a layered design. It also limits the spoken claim around 18:00 that sandboxing guarantees other environments will remain untouched. A sandbox that can call an authorized deletion API can still delete data through that API. Isolation of local execution and authorization of external effects solve different problems. This is a security inference from the described architecture, not a reported exploit.

Verdict: agree with layered controls, disagree with an unconditional safety guarantee. Confidence: high. Test both local isolation and external authorization. Observability can reveal an action after it happens; it does not substitute for denying an unauthorized action.

Self-repair and model portability need task-level evaluation

The Strands documentation confirms support for multiple model providers and describes lifecycle controls and evaluation facilities. That supports changing a backend without necessarily rebuilding the application architecture. It does not establish equivalent tool selection, prompt following, latency, or repair quality across models.

The talk’s repair examples are descriptions and small demonstrations, not a comparative production study. No independently audited reliability result was retrieved. The current repository also differs from the historical demo, so copying an old constructor option without checking the installed API is risky.

Verdict: mixed. Confidence: medium for useful bounded repair, low for general production self-healing. Use a fixed set of injected failures and an unchanged success criterion when comparing models. Limit attempts and preserve the original failing artifact so that a repair cannot silently redefine success.

Generated agents do not create fresh data access

At 13:20, Subramani explicitly says that the demo lacks a tool for real-time information. The repository documents mechanisms for adding tools and model providers, but those mechanisms do not supply data entitlements or live travel feeds by themselves.

Verdict: agree with the speaker’s stated limitation. Confidence: high. For a real itinerary workflow, add an authorized data source, check timestamps, and distinguish suggestions from bookable inventory. Do not treat delegation to an agent named for flights as evidence that it queried flights.

Comment insights

@nikhiltale943 asks how a sandboxed agent can repair an external pipeline when the repair requires deleting data or retrying work. This identifies the unresolved boundary between isolated computation and privileged external action. The useful next experiment is an authorized, idempotent replay on fixture data, followed by a denied destructive operation.

@showmethemoney824 asks who audits generated tools. The talk’s trace-level evaluation is relevant, but source retention, approval ownership, and immutable version identifiers are needed to answer that operational question.

@Ikka-l3v and @alejandromarrero1905 challenge whether arithmetic and character counting justify runtime generation instead of prebuilt tools. Their objection motivates the fixed-tool baseline above. These comments had zero or one like at extraction and are a small sample, not evidence of broad consensus. They are valuable because they propose concrete tests of necessity and accountability.

Screen-level insights

At 8:12, the inspected slide is titled “Self-extending agents” and says Strands can watch a tools directory and use new or updated tools immediately. A code example visibly highlights load_tools_from_directory=True in an agent construction line. Nearby narration explains that directory loading later became a separate tool. The frame supports the runtime-loading mechanism, but it does not establish that the historical option remains appropriate for every current version.

At 17:27, the inspected “Guardrails” slide has four panels for the environment, tools, permissions, and observability. It explicitly lists human approval for high-impact actions and per-action authorization checks. This matters because a summary that reduces the talk to unrestricted self-modification would omit controls that are visible on screen and discussed aloud.

Neither inspected frame proves that those controls were active in the live demo. No production incident repair, authorization test, or sandbox-escape test was run for this analysis.

Verification notes

Checked the extracted transcript and comments, the 8:12 and 17:27 frames, the Strands loader source, and the current SDK documentation. The synthesis separates runtime loading from correctness, local isolation from external authorization, and generated itinerary text from real-time travel data. Historical constructor behavior, production repair reliability, and the claimed origin of the TypeScript implementation remain unverified.

After the unslop editorial pass, the synthesizing agent performed four separate role-based passes, each with a fresh re-read: source/evidence, transcript/comment/frame fidelity, hallucination/overclaim, and Actionable Insights. These were not independent agent reviews. The fidelity and overclaim passes checked that quotations, technical content, facts, numbers, confidence, and uncertainty survived the edit. The Actionable Insights pass checked first steps, linked tools, evaluation criteria, and integration cautions. Suggested experiments were not executed. Web search was unavailable; research used directly retrieved primary sources. Frame coverage is limited to the images identified above. Historical analyses and raw extraction artifacts were not rewritten.