The Demo Was the Easy Part
Thread AI Editorial · August 25, 2026
The good news was that the outage started early Thursday morning. A deploy had gone out overnight for the new feature, a downstream service came back slower than expected, and every policy request that arrived during the outage stalled the same way: the agent picked up the file, made a call that sat waiting on a stalled service, and stopped mid-decision. The bad news arrived once someone actually looked. Nothing had failed loudly, because nothing had happened – the queries were still waiting, the process stayed up, and every health check came back green. Forty minutes of requests sat frozen in a state nobody had written code to recognize, and no one could say exactly where each had stopped.
Retrying a stalled request for a policy quote meant re-running steps that had already run, and in insurance a decision posted twice means a duplicate customer entry, so retries had to be idempotent. Knowing which requests were mid-flight when a service went down meant writing state down at every step, because – as the team learned that fateful Thursday – it was nearly impossible to reconstruct the processes from application logs after the fact. Each of these items was scoped to take weeks. Headcount had not changed and the ship date had not moved, but the people closest to the code were starting to worry about hitting the launch deadline.
Stalled Queries, One Question
Forty minutes of stalled queries against an unmoved ship date forced the question nobody had asked directly yet: Does the agent execute correctly, every time?
The answer: not every time. Agents now execute correctly almost all the time, but “almost” is not a standard an installed base accepts. That is the thing about operating inside a large installed base of customers: the direction of pressure changes, but the obligation to your customers does not.
The customers whose policy queries sat in that queue hold SLAs that make no distinction between the mature products and the new capability attached to them.
Mistaken for a Staffing Problem
This is where a Day Two problem gets mistaken for a staffing problem. It is tempting to look at all those stalled policy asks and conclude the team needs to move faster, needs another engineer, or needs to be more careful about deployment timing. The gap between a demo and a production system is a defined set of engineering problems, all solvable and none of them specific to this company. Some of those problems include:
- Durable execution across a restart.
- Idempotent retries that do not create duplicate decisions.
- State tracking granular enough to say, of any given request, exactly where it stopped.
The Layer Beneath the Loop
None of this is an indictment of the framework because an agent framework promises the loop: prompts, tools, model calls, and the orchestration of a single run while the process hosting it stays alive. Durable execution sits underneath that promise with queues, retry policies, idempotency keys, a state store, resume logic. Every serious framework, including the one this team chose, leaves that layer to the builder. These parts exist on every serious stack and somebody still has to assemble them, then test the paths that only run when something is already wrong. Assembly does not compress because a date is fixed.
What the framework promises
The loop: prompts, tools, model calls, and the orchestration of a single run — while the process hosting it stays alive.
What sits underneath — left to the builder
This is what deadline math usually looks like. A team can estimate the agent logic fairly accurately, because logic is the part they can see. The layer underneath reveals its size one dependency at a time. Add a retry, and a decision can be posted twice, so the retry needs an idempotency key. The key needs a durable record of what was already decided, and the record needs to survive the deployment that triggered the retry. Every step in that chain is reasonable and the chain is a platform, assembled piece by piece.
01A retry can post a decision twice
02So the retry needs an idempotency key
03The key needs a durable record of what was already decided
04The record needs to survive the deployment that triggered the retry
Durable by Design
There is a different starting position. When execution is durable by design — when no run lives and dies inside a single process, and every step writes its state the moment it happens — a restart becomes an event the system was built for. The stalled call turns into a recorded fact: this request, this step, waiting on this service, since this timestamp. When the service comes back online, the run picks up from where it stopped, and each step runs exactly once. Retry logic, state tracking, and resume-after-restart behavior are present before the first requests for policy quotes, instead of stitched on after an outage explains why they were needed.
run state
written as it happened
request
policy quote #4127
step
3 of 5 — waiting on policy-service
since
Thursday 06:02:14
on restart
resume at step 3 · run each step exactly once
Which is the second thing that defending the moat requires. In this example, the pressure came from a different direction, but the constant is the same: an installed base does not judge a new capability on what it can do in a demo. It judges whether the product its contracts, processes, and daily operations sit on is still the product it can depend on — at 6:40 a.m., after a deploy, with no one watching.
The Series
This is Post 2 of six, and the second of two about defending the moat. Read the thinking behind this series in the manifesto.
Your Customers' Regulators Will Ask What the Agent Did – and Why
For companies whose customers already depend on their product, adding AI means answering to those customers’ regulators too. This post follows a claims software company that ships an agent to review insurance claims, then can’t explain one of its denials to a state examiner because the reasoning was never recorded when the decision was made.
The Demo Was the Easy Part
Businesses that run their operations on a product hold every new feature to the same service commitments. This post follows an insurance software company whose overnight deploy leaves quote requests stalled mid-decision while every health check stays green, because nothing recorded where each request stopped or how to resume it.
What Can This Agent Access for Each of Your 200 Clients?
Turning an internal tool into a product for a new market pays off if it ships while the opportunity is still open. This post follows a company that plans to sell its internal dispute-reconciliation agent, then learns that keeping each client’s data, billing, and limits separate could take quarters to build, long enough to miss the opening.
A 97% Success Rate Is a Liability
Teams moving fast into unfamiliar territory often launch first and plan to iterate later. This post follows a company building its first logistics product, whose exception-handling agent slips from 99% to 97% success as volume grows, leaving 1,500 failed runs a day with nothing built to handle them.
Assembled by Your Team, Owned by Your Team, Forever
Codifying expertise lets a firm reach clients its experts will never have time to serve. This post follows a pay-consulting firm that turns its partners’ methodology into an agent product for 40 clients, then finds that keeping each client’s limits in place has it staffing like an infrastructure company.
The Day the Ship Date Stopped Moving
Turning a playbook into a product means answering to every client’s reviewers. This post follows a life sciences consultancy that builds a supplier-review agent in weeks, then watches its launch date move three times as the client’s quality, security, and audit teams each ask what the agent did, and why.