Intent-driven UX: how to make generative interfaces controllable

Video: The End of the Static Screen: Architecting Intent-Driven UX — Gus Iwanaga, commercetools / AI Engineer, 23:18

Actionable Insights

1. Start with a declarative component catalog, not model-generated HTML. For a production pilot, expose only 8–15 low-risk components such as Page, Section, KpiCard, Table, Chart, Form, and ApprovalBar. Define each component’s props, allowed children, data requirements, empty/error states, and permitted actions with a runtime schema such as Zod. Then make the model emit a validated UI description that your application maps to native components. A2UI, Vercel’s json-render, and OpenUI all implement variations of this constrained-catalog pattern. Evaluate schema-valid generation rate, render failures, accessibility violations, and task completion against the current static UI. Do not let a valid schema be mistaken for a good experience: validation prevents malformed or unauthorized structures, not confusing layouts or misleading copy.

2. Codify information architecture as template → slot → component-category rules. The speaker’s most reusable technique is to stop asking the model to invent a page from a flat bag of widgets. Create a small ui-layouts.yaml or equivalent containing templates such as analysis, planning, review, and approval; give each named slots (summary, primary, evidence, actions) and restrict each slot to eligible component categories. At runtime, retrieve components for the user intent, then map them upward into sub-slots, slots, and an approved template. Begin with one frequent workflow and replay 30–50 representative prompts. Score hierarchy stability, required-information coverage, component count, and whether the primary action remains predictable. The caution is overfitting: templates should constrain composition while still supporting legitimate domain variants.

3. Build a replayable generative-UI evaluation set before polishing the orchestration. Save representative requests in evals/generative-ui.jsonl, including paraphrases of the same intent—exactly where the video’s early Q1 report failed by alternating between “Q1” and “January–March” and changing visual density. For each case, encode invariants: date range, required entities, mandatory actions, maximum component count, allowed templates, and critical copy. Generate each case several times and compare semantic consistency, layout stability, action safety, and latency. Fail the build when a response changes the meaning of the request or omits required information; use human UX review for visual hierarchy and taste. This turns “guided by us” into an auditable quality loop rather than an informal prompt-tuning claim.

4. Separate intent/tool orchestration from UI composition. Use a pipeline such as intent classification → authorized tool selection → normalized domain result → eligible-component retrieval → layout policy → schema validation → native render. Keep tool outputs independent of presentation so a campaign-planning agent cannot smuggle arbitrary markup into the renderer. If interactive content must come from an MCP server, compare a constrained native renderer with MCP Apps, whose official model uses sandboxed iframes, declared UI resources, CSP, and host-mediated communication. Test authorization failures, malicious strings, oversized results, missing data, retries, and approval boundaries. The model should recommend an action; consequential writes should still pass through explicit application controls.

5. Treat the catalog as a versioned API contract. Give every component schema an owner, examples, compatibility policy, accessibility checks, and telemetry. Add catalog tests that generate valid/invalid specimens, render every state, and verify that unknown components, props, and actions fail closed. Record which component versions each prompt/evaluation set targets, because an agent trained or prompted against an old catalog can produce syntactically valid but operationally obsolete specs. Success is not catalog size; it is a high percentage of user intents covered by a small, coherent set without one-off escape hatches. The speaker is right that “every property matters,” but catalog governance is the missing operational consequence.

6. Move designers and PMs from drawing every screen to authoring rules and examples—but preserve direct user testing. Create a cross-functional review for schemas, layout policies, interaction states, copy constraints, synthetic prompts, and counterexamples. A useful weekly artifact is a contact sheet of repeated generations for the same intents, annotated with the violated or satisfied UX rule. Measure time-to-complete, error recovery, trust, consistency across turns, and accessibility with real users; do not optimize only model validity or stakeholder taste. Microsoft’s HAX Toolkit is a useful source of human-AI failure scenarios and interaction guidance. Generative UI changes the design material from fixed pixels to a bounded possibility space—it does not remove research, content design, or accountability.

Core thesis

Generative UI becomes credible when the model does not receive unlimited freedom. The useful middle ground is a declarative protocol: an orchestrator resolves intent and data, an agent selects from a curated component catalog, layout rules constrain information architecture, a schema validates the description, and the client renders native design-system components. The model can adapt a screen to the task while the product team retains control over safety, consistency, copy, and interaction quality.

This is stronger than the talk’s opening “end of the static screen” framing. The practical destination is probably not the disappearance of stable screens, but a hybrid: deterministic shells and high-risk flows combined with adaptive, task-specific composition where variability creates measurable value.

Big ideas and best timestamped moments

• 2:33–5:38 — Software still makes users learn each application’s mental model. Iwanaga uses dense enterprise screens to argue that feature accretion transfers cognitive load back to users. The diagnosis is directionally persuasive, although screenshots are illustrations rather than “hardcore evidence” about onboarding cost.
• 6:09–8:40 — Naive generation produces semantic and visual drift. Four responses to effectively the same Q1 sales-report request vary in date wording, component count, charts, and hierarchy. This is the talk’s best evidence: personalization without stability can manufacture confusion.
• 8:42–10:13 — Orchestration supplies context; a UX agent composes the result. The improved campaign-planning demo separates intent extraction, first/third-party tools or MCP servers, and UI rendering. The demo looks more coherent, but no task metric or repeated-run result is presented.
• 10:14–14:50 — UI generation is a control spectrum. One end selects a fully prescribed component; the other emits open-ended HTML in a sandbox. The speaker favors the declarative middle, where the model can choose and compose but only within a catalog.
• 14:51–16:54 — The UI description is the contract. Intent leads to tools and data; eligible catalog components are mapped to entities; a schema-checked specification becomes native React UI. This architecture is more important than the specific protocol name.
• 16:55–19:59 — Information architecture must be encoded. The team borrows from atomic design and represents page composition as layouts, slots, sub-slots, categories, and components. The runtime mapping is effectively reversed from retrieved components back toward an eligible template.
• 20:00–21:02 — The component catalog becomes the agent/UI boundary. Props and layout attributes are not implementation trivia; they determine what the agent can express and how reliably the renderer can interpret it.
• 21:03–22:34 — Product work shifts from pixels to schemas, rules, and synthetic cases. Designers and non-technical PMs now discuss catalogs, mapping logic, generated queries, and interaction patterns. That is a real role change, but the presentation underplays user research, accessibility, governance, and operational monitoring.

Practical takeaways / recommended workflow

1. Select one read-heavy, reversible workflow where users currently assemble information across multiple screens.
2. Define 20–50 real intents and paraphrases, plus semantic invariants and unsafe outcomes.
3. Normalize tool output into domain objects before any presentation decision.
4. Build a minimal typed component/action catalog and render it with native design-system components.
5. Define two or three templates with named slots and eligibility rules; avoid free placement initially.
6. Generate multiple outputs per evaluation prompt, validate schemas, then score meaning, stability, usability, accessibility, safety, and latency.
7. Put all consequential actions behind explicit confirmation and server-side authorization.
8. Pilot against the existing static workflow. Keep the adaptive UI only if it improves completion/error metrics without unacceptable variance.
9. Version the catalog and replay the evaluation set on every component, model, prompt, or orchestration change.

Comment insights

Only four comments were retrievable, so this is weak audience evidence rather than a representative sentiment sample.

• The most concrete positive suggestion is that conversational generative UI may be especially interesting for individual usage. That points toward personalization as a better fit for personal workspaces than shared operational screens, where consistency and training matter more.
• The most-liked comment jokes that the conference buzzword list omitted “taste and verification.” It aligns closely with the talk’s strongest lesson: orchestration is insufficient without UX judgment and repeated evaluation.
• One viewer calls the broader AI Engineer material excellent but this talk poor; a reply asks why, but no substantive critique follows. The negative reaction therefore cannot support a specific technical counterclaim.
• There is no practitioner report in the comments confirming that commercetools’ architecture works at scale, lowers onboarding time, or improves task outcomes.

Deep research on the main claims and assistant verdicts

Claim 1: Declarative UI protocols offer a safer, more controllable middle ground

Supporting evidence. The official A2UI specification site describes declarative messages rendered through pre-approved native components rather than arbitrary generated code; it explicitly frames the design as crossing trust boundaries safely. json-render similarly constrains generated JSON to developer-defined components and actions, while OpenUI generates a structured stream from a registered component library. These are independent implementations of the same broad architecture.

Contradicting/limiting evidence. A constrained protocol guarantees neither good composition nor semantic accuracy. A2UI was still listing an evolving specification landscape when checked, and MCP Apps notes that client support varies. OpenUI’s token-efficiency numbers are project-published benchmarks, not independent usability evidence. None of these sources demonstrates that generated interfaces outperform well-designed static flows for enterprise users.

Verdict: agree, high confidence on the architecture; medium confidence on product outcomes. The talk correctly identifies constrained declarative generation as safer and more governable than free-form HTML. It overclaims if “possible” is read as “proven usable at scale.” The practical takeaway is to adopt the contract pattern but require task-level evaluation before replacing existing screens.

Claim 2: Stable design systems and explicit information architecture are essential

Supporting evidence. Brad Frost’s Atomic Design defines a hierarchy of atoms, molecules, organisms, templates, and pages, emphasizing reusable components, cohesion, testing, and consistency. The generative-UI projects above all rely on catalogs or registered component libraries. The video adds a useful production-specific extension: eligible components need placement policies, not merely schemas.

Contradicting/limiting evidence. Atomic design is a design-system mental model, not empirical proof that the speaker’s exact layout → slot → sub-slot → component hierarchy is optimal. Excessive templates can recreate the rigidity and accumulated complexity that generative UI was meant to reduce. A schema can also encode poor information architecture consistently.

Verdict: agree, high confidence on the need; mixed on the exact hierarchy. The catalog and layout policy are credible control surfaces. The speaker underclaims the need for catalog lifecycle management, accessibility, localization, responsive behavior, and backward compatibility. Use his hierarchy as a testable starting model, not a universal ontology.

Claim 3: Adaptive intent-driven screens can reduce cognitive load relative to static SaaS

Supporting evidence. The early demo clearly illustrates a plausible mechanism: showing task-relevant information can avoid forcing users to navigate an application’s full feature inventory. Microsoft’s Guidelines for Human-AI Interaction and HAX Toolkit synthesize extensive human-AI guidance and emphasize designing around human needs and anticipated AI failures, supporting the need for task-aware behavior and recovery.

Contradicting/limiting evidence. The same demo supplies the countermechanism: repeated prompts produce different date labels, density, and placement, increasing learning cost and reducing predictability. The talk provides no controlled onboarding, completion-time, error-rate, or longitudinal-learning comparison. Stable interfaces let users develop spatial and procedural memory; unnecessary adaptation can remove those anchors.

Verdict: mixed, medium-high confidence. Intent-driven composition can reduce irrelevant options, but adaptation is not synonymous with lower cognitive load. The claim is overbroad for frequent, collaborative, regulated, or safety-critical workflows. Preserve stable landmarks and actions; adapt supporting content only where experiments show a net benefit.

Claim 4: An orchestrator plus tools/MCP gives the UX agent enough context to render a meaningful interface

Supporting evidence. MCP Apps documents a real architecture for tool-linked interactive interfaces, including UI resources, sandboxed rendering, bidirectional JSON-RPC communication, CSP, and host-mediated capabilities. Separating tools from rendering makes authorization and testing more tractable.

Contradicting/limiting evidence. More context does not establish relevance, correctness, or good UX. Tool errors, stale data, intent misclassification, permission differences, and conflicting sources can all flow into a perfectly valid but wrong screen. MCP Apps’ sandboxed HTML approach is also a different control point from a native declarative catalog; teams should not conflate transport, tool invocation, and UI-description protocols.

Verdict: agree with conditions, high confidence. Separation of orchestration and rendering is sound. The phrase “enough ammunition and context” underclaims the normalization, provenance, authorization, uncertainty display, and failure handling required between tool calls and UI generation. Preserve source metadata and make missing/conflicting data visible.

Claim 5: Designers and PMs will shift from designing pixels to curating schemas, rules, and data

Supporting evidence. If runtime composition is real, someone must define component semantics, allowed combinations, interaction states, examples, and evaluation cases. The typed-catalog APIs in json-render and OpenUI make this work concrete rather than hypothetical.

Contradicting/limiting evidence. Generated interfaces still render pixels and still require visual craft, content design, accessibility, usability research, and end-to-end service design. Moving rules into schemas can exclude non-technical contributors unless tooling and organizational practices evolve. The talk offers team experience but no evidence that this role model generalizes or improves outcomes.

Verdict: mostly agree, medium confidence. The design object expands from a fixed screen to a system of constraints and examples; it does not eliminate screen design. The talk overstates “we do not design the pixel anymore” and underclaims the organizational investment needed to make schemas legible and governable.

Screen-level insights

The normal preview download failed with YouTube HTTP 403, so screen evidence was recovered from YouTube’s 320×180 storyboard. These frames are low-resolution and their timestamps are approximate (roughly within one storyboard interval); unreadable code or small labels are not claimed.

• ~4:00: a slide visibly says “Different apps. Different logic. Every single time.” over layered enterprise screenshots. It establishes the visual argument that cross-product complexity is cumulative, not merely a single bad-screen problem.
• ~6:20–7:20: the projected browser shows multiple generated white-panel dashboards/reports. One has prominent green bars; another has a sidebar and denser fields. This visual variability matches the nearby critique that equivalent Q1 requests produced inconsistent scope and hierarchy.
• ~9:10: the improved demo visibly uses a conversation/work area on the left and a structured planning/form surface on the right, ending in a prominent action button. The importance is not aesthetics alone: the output is now an actionable workflow rather than an uncontrolled pile of KPI cards.
• ~13:20: a dark generative UI example shows a centered prompt/result with several colored pill-like nodes. The nearby transcript identifies this as an open-ended Claude org-chart example; visually it demonstrates expressive generation, while the speaker questions whether a company should delegate the outcome fully.
• ~15:00–16:00: the control-spectrum slide visibly positions “Declarative” between controlled and open-ended approaches, followed by a declarative architecture diagram. This is the talk’s central design decision: adaptability is bounded by a structured description and catalog.
• ~17:40–19:00: an atomic-design slide displays stages from small elements through larger compositions; the next slide shows a nested layout diagram. These frames connect the abstract design-system reference to the concrete placement problem the UX agent must solve.
• ~20:30: the slide states that what the agent can build “literally flows from your schema,” alongside a dark catalog/schema interface. The visual reinforces that component definitions are the expressive and safety boundary.
• ~22:10: the slide says “We don’t design the pixel anymore” and shows a process flowing through structured artifacts. This matters because it makes the organizational claim visible: design effort moves upstream into reusable constraints, even though the phrase itself is too absolute.

My read / why it matters

This talk is valuable because it moves generative UI past the seductive demo question—“can a model draw a screen?”—toward the harder product-engineering question: “what constraints let a variable screen remain trustworthy?” Its answer is a catalog, schemas, placement rules, orchestration boundaries, and human taste.

The missing half is evaluation. A generated interface can be native, schema-valid, on-brand, and still semantically wrong or harder to learn. The strongest implementation pattern is therefore bounded generation plus invariant testing plus user evidence. Keep stable shells and consequential actions deterministic; let the model adapt information selection and low-risk composition; widen autonomy only after repeated generations and real users demonstrate improvement.

Verification notes

Four distinct verification passes were completed before publication:

1. Source/evidence audit: checked claims against official A2UI, json-render, OpenUI, MCP Apps, Microsoft HAX, and Brad Frost’s Atomic Design sources. Added limitations where project documentation establishes capability but not usability or scale.
2. Transcript/comment/frame fidelity audit: checked timestamps against direct YouTube captions; retried comment extraction and recovered four comments; retried video download with a JavaScript runtime; after HTTP 403, extracted and inspected 12 low-resolution storyboard frames. Marked frame times approximate and avoided claims about unreadable text/code.
3. Hallucination/overclaim audit: downgraded “hardcore evidence,” “possible at scale,” protocol guarantees, cognitive-load reduction, and “we don’t design pixels” to appropriately qualified interpretations. Separated protocol transport, sandboxed apps, schemas, and native catalog rendering.
4. Actionable Insights audit: verified that every item names a first implementation step, concrete artifact/tool, evaluation criteria, and caution; direct links are included for protocols, schemas, MCP Apps, and HAX. Generic “use generative UI” advice was replaced by catalog, layout-policy, replay-eval, pipeline, governance, and cross-functional workflow actions.

Residual uncertainty: the captions contain minor transcription errors (notably protocol names); no slide deck or readable code snippets were available; storyboard frames are coarse and approximate; only four comments were exposed; and the commercetools demo supplies no independent production metrics, architecture repository, or repeated-run results.