A 97% Success Rate Is a Liability

Thread AI Editorial · October 21, 2026

These early successes gave the team confidence they were on the right path. Even though they knew they had launched the product without robust failure mechanisms in place, even though they knew the agent wasn’t perfect, they figured they could continue iterating and tweak the product post-launch.

This perspective seemed to work, until it didn’t. As the team began to scale the product with more customers and real-world scenarios, they experienced failures that couldn’t be accounted for, like runs that stalled halfway through or tasks that were lost mid-run when the team deployed an update. What started at 99% slowly decreased to 97% over 10 months, and as volume grew, daily failures went from less than one at rollout to 1,500, each one a real problem someone was waiting on.

Those failures landed on the support team, the same team the agent was supposed to relieve. Instead of handling fewer inbound problems, they now spent their time untangling failures that aren’t tracked or easily resolved.

Where the Failures Went

The company had never run a logistics product before it built this one. About a year earlier, it spotted an opening: enterprise shippers were handing every delivery problem to their call center support teams, which was costly and often got overwhelmed with call volume. Shippers wanted an agent that could triage delivery exceptions on its own, without routing every issue to a call queue, and the first company to offer one stood to capture them as early customers. The team picked an open-source agent framework, wired it to the client service rules already in place, and had a working version live in about eight weeks.

That early agent wasn’t pretty, but it did the job. The agent read each exception, such as a package stuck at a hub, a damaged shipment, or a delivery window a driver couldn’t hit, checked it against the client’s own rules, and either resolved it automatically or routed it to a human. Occasionally a run failed, meaning the agent stalled or errored partway through. In the beginning, this was rare, and someone would notice the stuck ticket, fix it by hand, and move on. Although fixing failures by hand wasn’t a long-term solution, with so few runs failing and first-mover advantage at stake, the team decided a more robust system could wait until after launch.

Behind each of these failures was a real package and a customer waiting on it. Each one sat wherever the agent had stopped working on it, with no record of how far it had gotten or whether anything else downstream, like a reshipment or a refund, depended on it.

Getting there first had answered whether the team could ship fast. It hadn’t answered the question sitting underneath the dashboard: Does the agent execute correctly, every time?

The agent executed correctly the overwhelming majority of the time, and the technology kept improving month over month. None of that helped the customers behind those 1,500 daily failures. Each of them was waiting on a delivery that never resolved, and each of them had a name.

Success Rate Is Not a Single Number

Look past the percentage, and the dashboard’s real problem comes into view: a success rate measures a share, but customers experience a count. At low volume, the count is small enough to look past. At high volume, the count is thousands of customers every single day, whether the aggregate metric looks reassuring or not.

What the team had put off until after launch was the layer that catches the failures and does something deliberate with them without needing someone to step in and triage. That layer means durable execution that survives a restart mid-run, idempotent retries that don’t resolve the same exception twice, and a completion guarantee so every run ends in a known state: finished, failed cleanly, or still in progress. It also means building in failsafe logic that is able to handle runs that deviate from defined paths. Those pieces exist on any serious stack, and somebody still has to build them, then prove they hold up at the speed the business is actually running.

That proof work does not shrink when the ship date has passed. It’s the same job before and after, but doing it with a growing number of live failures a day is considerably harder than doing it before the first client site goes live.

When durable execution and completion guarantees are part of the platform from the first run, a failure becomes a recorded, resumable event, handled the same way every time. The team doesn’t have to choose between launching first and handling failures well, because the machinery underneath was built for the business the company was going to become, not just the pilot it started as.

That is the version worth building toward: an agent that behaves reliably whether a run succeeds or fails, so no one is left reconstructing what happened or guessing how to fix it. Both of these realities are critically important because the failures, however infrequent, are never invisible to the team responsible for them or the customers who experience them.

The Series

This is Post 4 of six, and the second of two about moving up the stack. Read the thinking behind this series in the manifesto.