Harnessing Autonomous
Intelligence

Build an operator around expert judgment

Our goal is an autonomous operator that owns a defined business outcome and does the work the way the company’s best people do it. That is harder than it sounds. A model that learns from every past case equally will reproduce the average of them, including old habits, workarounds and mistakes. The average of a support desk’s history includes the week a new hire guessed at the refund policy.

So every stage of the pipeline is built to separate good examples from the rest. We learn from the people the client names as the standard, give more weight to current practice than to old practice, and ask the expert about cases where the record disagrees with itself. The result is a written policy that captures the company’s best practice and gives the operator a basis for deciding what to do next. That policy is one part of the operator, alongside its working context, access to business systems and review controls.

We start with a bounded responsibility and a clear definition of success. Over time, the aim is for operators to take on the connected work of an entire department. That scope has to be earned through evidence, with explicit limits on what each operator can decide and when a person must step in.

Build an accurate record of how the work is done

Multi-modal datasets

Companies usually have far more material than they realize: screen recordings, ticket and email histories, chat threads where hard cases get discussed, handbooks, reply templates and policy documents. Each covers a different part of the job. We join them into one record per case and rebuild the state of every system as it was when the decision was made, so the operator learns from what the expert could see at the time, without hindsight.

Developing an action grammar

Before we can learn what an expert does, we need words for it. We define an action as a change in application state. Mouse movement is not an action, and copy and paste counts as data flow, showing where a value came from. From the recordings we build a short vocabulary of verbs, such as look up account, compose reply, escalate and mark handled, and the client approves it. Every past case then becomes a sequence in that grammar.

Understanding why

Records show what happened. To learn why, we use narration from the expert while they work and short interviews about the moments that the footage alone cannot explain. Silent footage is valuable too, because it shows the steps nobody thinks to mention.

Consistency flagging

Histories disagree with themselves. When the same kind of case was handled two ways, we check when. If one practice replaced the other after a date, that is drift, and we follow the newer practice. If both happened in the same period, that is an inconsistency, and we flag it for the expert. Questions are batched a few at a time and ranked by how many cases they affect, because expert time is the scarcest input in the whole build.

Turn thousands of cases into a few pages of rules

Looking at historical examples

We group similar cases, then regroup them by how they were handled. From each group we draft candidate rules. Each rule states when it applies, which actions follow, what the reply should say, and which past cases support it. Rules cite their evidence and never quote it, because a rule that quotes a case has memorized it.

Iterating on pruning and accuracy

We prune and we measure accuracy for different reasons. Pruning keeps the rulebook auditable. We believe a good ruleset is one a person can read end to end, check against their own judgment and sign off on. Our first consolidated draft ran to tens of thousands of words, which no one could audit, so the main body is now capped at two to three pages, enforced in code, with templates and evidence in appendices. A rule earns a place in that space only if enough cases support it.

Accuracy is what makes the operator perform. Every rule has to keep a minimum precision on the cases it claims to cover, and we measure each one on those cases before it is kept.

We iterate between the two because we want to give the operator room to experiment. In each round, the operator works practice cases with the current rules, and alternative versions of weak rules compete on the same cases. Versions that help are kept and refined. The rest are pruned. Over several rounds the rulebook gets both shorter and more accurate.

Rolling validation and stability testing

A rule that only worked last spring is a liability. We validate on a rolling basis: learn from an older window of cases, test on the window that follows, then move forward and repeat. A rule is kept only if its accuracy holds across windows. A rule whose accuracy falls in recent windows is a sign of drift and goes back for review.

Test against the record, then against the edge cases

Data freezing for hypothetical and counterfactual tests

For testing, we freeze a snapshot of every case: the message, the state of each system at that moment, and the policy in force. Frozen data makes results reproducible, and it lets us ask questions the history never answered. A counterfactual test changes one fact and checks that the operator’s action changes the way the rules say it should. A hypothetical test builds a rare case from scratch, such as an exception that came up twice in three years.

Evals and baselines

We need a good way to measure how the operator performs. Did it really accomplish the task, or did it produce something that only looks right? And how do we know our own process is right, that the rules we mined do better than a simpler approach would? Evals measure the operator against the record and against the client’s standards. Baselines give us a simpler way to get an answer that a learned rulebook has to beat, and the choice of baseline decides what “better” means.

For an operator, evaluation also needs to cover the work beyond a single action: whether the case reached the intended state, whether required follow-up happened, and whether exceptions reached a person with enough context to act. The tests below examine decision quality, one necessary part of that broader standard.

We tested three common baselines. Each is useful, and each can mislead.

Token overlap shows the risk clearly. On an earlier version of the support system, strict exact match on held-out cases rose from 16.9% for a generic prompt to 47.5% for the learned rulebook, while the share of correct actions fell by 3.8 points. We now report the parts of a score as well as the total.

Judging pitfalls and calibration

A model acting as judge can grade thousands of replies, but it is only as reliable as its reference. If it is shown only the client’s templates, it rewards replies that copy templates and marks down correct expert decisions it was never told about. It can share blind spots with the operator it is grading, and its scores vary between runs. We calibrate every judge against grades from the client’s own people, give it the client’s stated policy, and pair it with deterministic checks that need no model at all.

Earn autonomy in ongoing operations

Deployment starts in draft mode. The operator proposes each action, and a reviewer accepts, edits or rejects it. Every correction is traced to the rule, the evidence and the policy version that produced it, which turns review into a steady supply of labelled examples.

• Improving existing rules. When corrections cluster on one rule, that rule is revised and re-tested on its own cases and the held-out set before release.
• Detecting new rules. Corrections that no rule explains are grouped. A group that keeps growing usually means a new kind of case or a policy change, and it becomes a candidate rule that goes through the same mining and testing as the originals.
• Widening autonomy. Each action type earns autonomy on its own record. A type that holds its accuracy in review moves from draft to automatic, one type at a time.

Deployment is the start of an operator’s ongoing responsibility. We are building toward operators that track unresolved work across handoffs and return to it as new information arrives. Progress means more cases brought to the agreed outcome within policy, with less manual coordination. Each increase in responsibility must preserve the review and escalation controls below.

Safety is part of the architecture. The step that reads customer text has no tools, network access or credentials, so the worst a hostile message can produce is a bad draft that a person sees. Approval is enforced by a check the model cannot change. Every item either reaches a person or ends in a state somebody chose.

For how this approach compares with systems that improve without labelled examples, read Post 002: Two ways to improve an AI operator.