The Demo Was The Easy Part

August 25, 2026

The good news was that the outage started early Thursday morning. A deploy had gone out overnight for the new feature, a downstream service came back slower than expected, and every policy request that arrived during the outage stalled the same way: the agent picked up the file, made a call that sat waiting on a stalled service, and stopped mid-decision. The bad news arrived once someone actually looked. Nothing had failed loudly, because nothing had happened – the queries were still waiting, the process stayed up, and every health check came back green. Forty minutes of requests sat frozen in a state nobody had written code to recognize, and none of them could say exactly where it had stopped.

Retrying a stalled request for a policy quote meant re-running steps that had already run, and in insurance a decision posted twice means a duplicate customer entry, so retries had to be idempotent. Knowing which requests were mid-flight when a service went down meant writing state down at every step, because – as the team learned that fateful Thursday – it was nearly impossible to reconstruct the processes from application logs after the fact. Each of these items was scoped to take weeks. Headcount had not changed and the ship date had not moved, but the people closest to the code were starting to worry about hitting the launch deadline.

Forty Queries, One Question

Forty stalled queries against an unmoved ship date forced the question nobody had asked directly yet: Does the agent execute correctly, every time?

The answer: Not most of the time – or not correctly, at least according to the last time someone checked, but recently the technology world has seen agents executing correctly almost all the time, across every query and every timezone. That is the thing about operating inside a large installed base of customers, while the direction of pressure changes, while obligation to your customers does not.

The customers whose policy queries sat in that queue hold SLAs that make no distinction between the mature products and the new capability attached to them.

Mistaken for a Staffing Problem

This is where a Day Two problem gets mistaken for a staffing problem. It is tempting to look at all those stalled policy asks and conclude the team needs to move faster, needs another engineer, or needs to be more careful about deployment timing. The gap between a demo and a production system is a defined set of engineering problems, all solvable and none of them specific to this company. Some of those problems include:

• Durable execution across a restart.
• Idempotent retries that do not create duplicate decisions.
• State tracking granular enough to say, of any given claim, exactly where it stopped.

The Layer Beneath the Loop

None of this is an indictment of the framework because an agent framework promises the loop: prompts, tools, model calls, and the orchestration of a single run while the process hosting it stays alive. Durable execution sits underneath that promise with queues, retry policies, idempotency keys, a state store, resume logic. Every serious framework, including the one this team chose, leaves that layer to the builder. These parts exist on every serious stack and somebody still has to assemble them, then test the paths that only run when something is already wrong. Assembly does not compress because a date is fixed.

This is what deadline math usually looks like. A team can estimate the agent logic fairly accurately, because logic is the part they can see. The layer underneath reveals its size one dependency at a time. Add a retry, and a decision can be posted twice, so the retry needs an idempotency key. The key needs a durable record of what was already decided, and the record needs to survive the deployment that triggered the retry. Every step in that chain is reasonable and the chain is a platform, assembled piece by piece.

Durable by Design

There is a different starting position. When execution is durable by design — when no run lives and dies inside a single process, and every step writes its state the moment it happens — a restart becomes an event the system was built for. The stalled call turns into a recorded fact: this request, this step, waiting on this service, since this timestamp. When the service comes back online, the run picks up from where it stopped, and each step runs exactly once. Retry logic, state tracking, and resume-after-restart behavior are present before the first requests for policy quotes, instead of stitched on after an outage explains why they were needed.

Which is the second thing that defending the moat requires. In this example, the pressure came from a different direction, but the constant is the same: an installed base does not judge a new capability on what it can do in a demo. It judges whether the product its contracts, processes, and daily operations sit on is still the product it can depend on — at 6:40 a.m., after a deploy, with no one watching.

The Series

This is Post 2 of six, and the second of two about defending the moat. Read the thinking behind this series in the manifesto.