An Inside Look

Martin McRoy - VP of Engineering at Thread AI

This is the third post in Facts, Rules, Runtime, our series on the system we use to manage agentic software development. The previous post, The Verification Layer, From Policy to Exit Code, explained how a contract determines whether a task can move to the next stage.

Verification asks whether the task met its contract. Measurement asks whether the agent’s decisions held up through engineering review, CI, and later changes to the code.

This is how the harness learns: AI can process captured engineering judgment to find recurring corrections and propose new rules. Engineers review those proposals and turn them into guides, verification policies, routing rules, and task structures that future runs use.

This post follows the loop from engineering signals to an accuracy ledger, then to measurement and changes to the harness.

Build the Smallest Useful Loop

Start with one decision the system makes often: a review verdict, a triage decision, a task classification, or an eligibility decision. Record what the agent decided and connect it to the judgments and code changes that follow.

A PR is the package humans review. Within it, the harness can track individual classifications, review findings, accepted or rejected suggestions, and failed checks. Commits, changed files, and line ranges connect those events to code that can be examined again later. The unit of learning is smaller than the PR.

Keep the evidence behind each proposal. The team needs to see which corrections led to a rule, where it should apply, and whether it addresses the original problem.

Capture the Judgment Already in the Workflow

Engineers already label agent work all day. They approve it, request changes, leave comments, resolve threads, accept suggestions, force-push fixes, revert commits, and watch CI fail or pass. These actions supply evaluation signals as part of ordinary development. The harness needs to record them as structured events without asking engineers to enter their judgment again.

Start with pull-request review

The code review system is usually the first integration worth doing deeply. In our stack, that is GitHub. A review verdict tells us whether a change was acceptable; comments explain why; follow-up commits show what had to change; and resolved threads show whether the fix satisfied the reviewer. Capture these alongside check results, commit updates, and whether the PR was eventually merged.

Reviews from multiple models and the process that reconciles their findings can feed the same ledger. They contribute evidence; the engineering workflow still determines the outcome.

In the migration example, the requested-change comment identifies the missing test, the follow-up commit shows the correction, and the context snapshot records which guides the agent received. Connecting them makes the reviewer’s judgment useful beyond that one PR.

Choose signals by what they help you decide

Prioritize sources that give you usable labels, explain corrections, improve diagnosis, or reveal what happened to accepted code. Each source serves a different purpose, so evaluate it with the measurement that fits that purpose.

Interpret code survival with care from the outset. Code can change because the product changed or because the original implementation was fragile. Review comments, task context, and later intent help distinguish the two.

Preserve Decisions in an Accuracy Ledger

The ledger preserves the original agent decision, later human judgments, and the links between them and the task’s code and context. Put a durable, replayable message queue between the source tools and the ledger so the system can handle retries, backfills, and changes to downstream readers.

The ledger can use a database table, an object store, or a dedicated event store. What matters is that events have a defined structure, new entries are appended, and identifiers connect each decision to its task, review, commits, and context. An AI consumer can process events from the queue and consult this linked history to find recurring corrections across tasks.

When an engineer confirms or overrides a decision, the harness appends a new event linked to the original. This is append-only adjudication. It does not rewrite the earlier event, so the agent’s decision, the human correction, and the time between them remain visible.

Keep workflow labels readable. Review, execution, triage, reconciliation, and release tell the team where a decision was made. If you record both a lifecycle stage and a more specific workflow step, keep those fields distinct and use their names consistently. Agent and human labels belong to their respective linked events.

Read Metrics That Point to Work

Once decisions are linked to later judgment, the next question is where disagreement concentrates. Decision quality and code lifetime answer different questions. Read both by workflow step and task context to identify which part of the harness needs attention.

Code lifetime starts with the commit SHAs and changed files the verification layer records. The harness follows Git history to see whether accepted code remains, is rewritten soon after merge, or is reverted. This gives the team a way to examine work that passed checks and review but later proved fragile.

Context helps identify what to change. If overrides spike when a particular guide is attached, inspect that guide. If code from broad tasks is consistently rewritten soon after merge, inspect how those tasks are decomposed. In the migration example, group overrides by task type and attached guide to see where missing compatibility tests recur.

Use AI to Extract Candidate Rules

Metrics show where to look. The next step is to extract a requirement from the evidence. Give an AI consumer the queued review, CI, and Git events, together with the related decisions and context in the ledger. It can group similar corrections, compare the fixes engineers accepted, and propose a rule that addresses the recurring problem.

Ask for a concrete proposal: the instruction or check to add, the tasks or files it should apply to, and the guide, verification policy, or routing rule it would change. Each proposal should cite the reviews, commits, and context that support it. Conflicting feedback and exceptions should remain visible so an engineer can judge the scope.

In the migration example, the recurring requirement is compatibility coverage. The AI can propose requiring compatibility tests for API migration tasks and attaching the migration guide when the relevant files are touched. The proposal should make that scope explicit, rather than extending a migration requirement to every code change.

Use the recorded mistakes and accepted corrections as evaluation cases for the proposed rule. Check whether it would have caught the missing tests and whether its scope fits the evidence. These cases help engineers assess a change to the harness before promoting it, and provide regression checks for later changes.

Turn Repeated Overrides into Rules

A single override is not a rule. A cluster of repeated overrides is a starting point for a proposal. AI can draft the rule; an engineer decides whether to accept, revise, or reject it. Promotion happens only after that review, so a one-off disagreement does not become a permanent instruction.

The change should address the reason for the repeated correction. A naming convention belongs in a guide; a missing compatibility check may need an executable verification policy. Routing and task decomposition address different causes.

Once the proposal is reviewed, update the relevant harness files and preserve the link to its supporting evidence. The next run must receive those files for the rule to take effect. Its review results then return to the ledger, where the team can check whether the same correction is becoming less common.

Deliver the Change to the Next Run

Post 4 takes up deployment. A new guide, verification policy, or routing rule has to reach each supported agent provider’s configuration consistently. Otherwise, developers’ agent setups drift and the reviewed change has no reliable effect on future work.

Capturing evidence, using AI to propose rules, reviewing those proposals, and delivering the resulting changes requires the same orchestration discipline behind Lemma, our workflow platform. Here, that discipline connects engineering feedback to the next agent run.

Captured judgment informs proposed rules. Engineering review turns them into harness changes. Deployment puts those changes into use.