Your Customers' Regulators Will Ask What the Agent Did – and Why
Thread AI Editorial · August 18, 2026
The email came in on a Tuesday, addressed to a support alias. It sat in the queue for two days before anyone flagged it. In the email, a compliance analyst at a mid-sized regional carrier had a simple request: A denial letter had gone out the week before and she wanted to know how her team should describe to a state examiner the reasoning behind the claims decision underneath it. She asked politely and needed the answer by the end of the month.
The company on the other end of that email had been building claims administration software for two decades, and everything about the business reflected its installed base. Several hundred carriers ran on the product, with multi-year contracts, SLAs, and internal processes built around it. Renewals were steady because switching away from a system your operations depend on is expensive. The roadmap was boring in the way mature products earn the right to be boring. Then the asks from that installed base changed. Customers wanted the product to become more adaptable, and agentic claims review was the version of an ask with a date attached.
The previous spring, the leadership team had stood on a stage at their user conference and committed to shipping it in the fall. That build had gone well. The team picked a capable open-source agent framework, wired it into services they already maintained, and got to a working version in less than six weeks. That version moved a claim through intake, pulled the policy terms, checked coverage against the loss description, flagged exceptions, and recommended an action. The demos were fast and performed correctly.
What, and Why
The email from that compliance analyst changed everything. Once it came in, an engineer went looking for what the system produced around one specific decision. The logs were all there and showed the sequence of calls, prompts, tool invocations, token counts, and latency.
What
Reconstructing what the agent had done was quick and took an afternoon. Calls, prompts, tool invocations, token counts, latency. Every serious framework on the market handles that well now.
Why
Reconstructing why took considerably longer, and what came back was not something anyone wanted to put in front of a state examiner.
The record showed a model output. It did not show:
- Which policy provision had governed the decision.
- Which version of that carrier's coverage rules had been in effect that week.
- Which reviewer had cleared the exception.
- What had changed between the recommendation and the letter that actually went out.
While those facts existed somewhere in the system, none of them existed together, attached to that claim, and in language a person outside the engineering team could read. That's when everything came screeching to a halt.
Swap the Nouns
Swap the nouns and this story is applicable anywhere a product has an installed base in a regulated market. Trade the state examiner for a bank regulator, the claim for a loan decision, a prior authorization, or a KYC flag, and the email reads the same. The moment the product starts making or recommending decisions, those customers' regulatory obligations become the product's requirements. The regulator never contacts you; they contact your customer, and your customer answers with whatever your product can produce.
The whole feature release turned out to hinge on a question that no one had written on a ticket or in a spec: What did this agent do, and why, in terms the customer's regulator will accept?
The stall wasn't a failure of the framework or engineering efforts. The agent reasoned, called tools, and left a trace of both; every serious framework on the market handles that well now. The problem was that the evidence the analyst needed sat in a different layer.
The Audit Trail Is Part of the Interface
In a regulated product, the audit trail carries the same status as the user interface. Customers see it, evaluate it, and eventually depend on it. It turns up in procurement questionnaires a few lines below uptime. Teams tend to schedule it like documentation, as work that happens near the end once the behavior has settled, and explainability at the business level resists this treatment.
Built in, the same capability looks unremarkable from the outside. A decision opens, and the record that opens with it carries the inputs, the version of the rules that applied, and the reviewer who cleared the exception and when. Everything is captured as the work happens, readable by someone whose title has nothing to do with engineering. This is why the answer is already in the record by the time that compliance email arrives.
That is what defending the moat requires. This is what it means to serve those several hundred customers who trust the product enough to build their contracts and processes around it and drive revenue with it. Making the product more adaptable is the ask; doing it without asking the installed base to absorb the risk is the goal.
The Series
This is Post 1 of six, two for each archetype. Start with the series introduction for the framing behind all three archetypes.
Your Customers' Regulators Will Ask What the Agent Did – and Why
For companies whose customers already depend on their product, adding AI means answering to those customers’ regulators too. This post follows a claims software company that ships an agent to review insurance claims, then can’t explain one of its denials to a state examiner because the reasoning was never recorded when the decision was made.
The Demo Was the Easy Part
Businesses that run their operations on a product hold every new feature to the same service commitments. This post follows an insurance software company whose overnight deploy leaves quote requests stalled mid-decision while every health check stays green, because nothing recorded where each request stopped or how to resume it.
What Can This Agent Access for Each of Your 200 Clients?
Turning an internal tool into a product for a new market pays off if it ships while the opportunity is still open. This post follows a company that plans to sell its internal dispute-reconciliation agent, then learns that keeping each client’s data, billing, and limits separate could take quarters to build, long enough to miss the opening.
A 97% Success Rate Is a Liability
Teams moving fast into unfamiliar territory often launch first and plan to iterate later. This post follows a company building its first logistics product, whose exception-handling agent slips from 99% to 97% success as volume grows, leaving 1,500 failed runs a day with nothing built to handle them.
Assembled by Your Team, Owned by Your Team, Forever
Codifying expertise lets a firm reach clients its experts will never have time to serve. This post follows a pay-consulting firm that turns its partners’ methodology into an agent product for 40 clients, then finds that keeping each client’s limits in place has it staffing like an infrastructure company.
The Day the Ship Date Stopped Moving
Turning a playbook into a product means answering to every client’s reviewers. This post follows a life sciences consultancy that builds a supplier-review agent in weeks, then watches its launch date move three times as the client’s quality, security, and audit teams each ask what the agent did, and why.