Charter

A tool waits to be used. A coworker takes work off your desk, makes judgement calls inside it, and hands something back. Software has started doing the second thing, and the boundary between the people doing the work and the systems doing the work is dissolving inside engineering itself. That is not a UX change. It changes who is accountable for an outcome, what review means, and what it takes to trust a result you didn't watch being produced.

Foundation models will keep improving, and that trajectory is not really in question. But most of the next decade's useful progress will come from better systems, infrastructure, evaluation, tooling, and workflows — not from larger models. The gap between what a model can do in principle and what a system built on it reliably does in production is where nearly all of the remaining engineering lives. That gap is closed by engineers, not by scale.

Interfaces churn every product cycle: chat, voice, embedded agents, ambient UI, whatever is next. What survives underneath is reasoning, memory, orchestration, and the boundaries between them. Architecture coupled tightly to "chat" is built on the layer most likely to be replaced first. Build so the interface is the cheapest thing to throw away.

A good model is a moment — something better replaces it within the year. Good infrastructure is a compounding advantage: it outlives whichever model sits on top of it, and every system built on it gets stronger for free when the layer underneath improves. The boring, invisible layers nobody demos are worth more over time than the flashy one on top.

The interfaces that let systems talk to each other tend to outlast the organisations that introduced them. HTTP outlived the companies that fought over the browser. That is the bet worth making: build against open, inspectable protocols rather than a vendor's surface area, because the protocol will still be here after the vendor's roadmap has moved on.

Benchmarks measure capability. Production measures usefulness. A score tells you what a system can do in a controlled setting; it tells you almost nothing about whether it is worth trusting with a real task, run by a real person, on a Tuesday when the upstream API is degraded and the input is malformed. We would rather ship something narrow that holds up than something broad that scores well and falls over the first time reality gets messy.

Retries, fallbacks, degradation paths, and human-in-the-loop gates have to shape the architecture from the start. They cannot be bolted on after, because the failure modes of a reasoning system don't surface as exceptions at the end of a test run — they surface as confident, plausible, wrong output that a QA pass has no way to flag.

Benchmarks measure capability; evals measure whether your specific system is trustworthy enough to use for your specific task. If you cannot measure whether the system did the right thing, you do not have a product — you have a demo that has not failed publicly yet. The eval suite is not test infrastructure sitting beside the product. It is the thing that tells you the product exists.

If you are not testing against adversarial inputs and edge cases before production, your users are your test suite and your incident channel is your reporting tool. Simulating the failure is always cheaper than explaining it afterwards, and it is the only way to find the failure modes you weren't imaginative enough to predict.

When these systems break in production, the cause is rarely the model. It is the seam between services, between a retry and a timeout, between what one component assumed about state and what another actually wrote. That changes where the engineering time should go: into the plumbing, the contracts, and the handoffs, not into the prompt.

A prompt change, a policy change, or a tool-definition change alters what the system does in production as surely as a code deploy does. It deserves the same review, staging, and rollback discipline. "We just edited the prompt" is how outages happen and how nobody can say afterwards what the system was actually doing at the time.

If killing a running agent cleanly — mid-task, with no orphaned side effects, no half-finished writes, no dangling external calls — is hard, the system was not designed for production. The ability to stop something is what makes it safe to start.

A system acting on someone's behalf needs a verifiable, revocable identity, the same as any employee or API key. Without one you cannot answer the two questions that matter after something goes wrong: who authorised this, and how do we take the access away. An unattributable actor with production credentials is not an architecture decision you postponed. It is an unlocked door.

A system that could do more than its task requires is a larger attack surface than one scoped tightly, no matter how carefully its instructions are written. Scope the permissions to the job, not to the roadmap. Capability you granted "just in case" is capability an attacker inherits.

A safety constraint expressed in a model's instructions is a suggestion. The same constraint expressed in code, permissions, and infrastructure is real. The difference matters most precisely when the system is behaving in a way you didn't anticipate — which is the only moment the constraint was ever for.

If your system's constraints live in a wiki page rather than in enforced guardrails, they do not exist — they are a description of what someone hoped would happen. Governance that cannot fail a deploy, block a call, or refuse a write is documentation, and documentation has never stopped an incident.

If deleting an account doesn't delete what the system learned about that person, the privacy policy is fiction. Memory is not a cache — it carries a version of someone's history. Until deletion, correction, and ownership are actually solved in the system, you haven't built memory, you've built a liability with a friendly name.

Software should do the work leading up to a decision. It should not make the decision itself until you have shown — with evidence, not vibes — that it is right often enough to be trusted unsupervised. Most companies blur this line deliberately, because the blurred version demos better. The line is worth holding anyway.

If a person only sees an agent's work once it has already shipped, you don't have a human in the loop. You have a human doing damage control. Review has to be a state the system can be in — something it waits for — not a meeting that happens afterwards.

How much a system is allowed to do on its own should be a function of how well you can measure what it does. Autonomy granted ahead of evidence is not a product decision, it is a wager. Move the boundary out as the evals justify it, and be willing to move it back.

A slow agent gets double-checked by a human, which defeats the point of delegating to it. Speed here is not UX polish — below some threshold, delegation stops being worth it at all, and the system quietly becomes a thing people supervise rather than a thing that helps.

Cost per task has to be designed for from day one. Discovered late, it is not a margin problem you can fix with a pricing change — it is a structural constraint that makes the system too expensive to actually delegate work to, which was the entire point of building it.

People adopt infrastructure they can audit. Code and research that others can read, run, and criticise gets held to a higher standard than anything built behind a wall. Closed systems ask for trust. Open ones earn it. We would rather earn it.

Model-agnosticism is usually pitched as a value. It is better understood as risk management. A system whose behaviour, economics, and availability are all determined by one vendor's roadmap has taken on a dependency it cannot price and cannot exit quickly. Name the risk rather than dressing it up as a principle.

Standard engineering advice says don't. We think that advice is over-applied in a field this young, where the existing wheel was often built for a different vehicle and nobody has looked closely at it since. Building the thing yourself, once, is frequently the fastest way to understand why it is shaped the way it is — and occasionally the fastest way to find out it shouldn't be.