What I would measure before shipping an AI workflow

“It got better” needs a denominator.

Better at finishing which tasks? Across how many attempts? Did it ask for approval before changing something? Did it accurately explain a failed write? Did the faster version quietly stop attempting difficult cases?

For an AI workflow that reads records, calls tools, and changes state, I would want answers to those questions before a release. A polished final message is only one part of the evidence.

The companion evaluation harness contains six synthetic support tasks and two recorded trials per task for each of two invented variants. It runs no language model. Its purpose is to show how a grader can catch a bad change that an average score would reward.

Define completion where the work happens

Start with one task and describe what should be true when it ends.

For a support-task creation request, that might mean exactly one task exists for the correct case, a reviewer approved the write, and the response accurately describes the result. For an incomplete request, the right outcome could be asking for a missing reference without writing anything.

Those expectations need an owner who understands the work. An engineer can check a database field; the service owner needs to decide whether that field represents successful service.

I would separate three questions:

The original τ-bench paper makes an instructive distinction. Its reward checks final database state and required response information, while explicitly acknowledging that a successful reward can still miss a policy violation such as absent user confirmation. That limitation is a reason to inspect evidence beyond the terminal state. It is not a claim that every valid workflow must follow one exact trace.

Build cases from actual uncertainty

I would begin by reading examples of the work, with the people responsible for it. Label what went wrong in plain language before deciding which measurements to automate.

Hamel Husain's field guide describes this kind of error analysis: inspect conversations, annotate specific failures, and let recurring problems shape the tests. For this workflow, “asked for the same reference twice” or “closed the case before the downstream task existed” is more actionable than a generic helpfulness score.

Use those findings to build a small case set with clear instructions, starting state, expected outcome, and allowed alternatives. Include cases where the system should act and cases where it should leave state alone. Freeze a comparison set before tuning against it, and keep a separate development set for the examples being used to revise prompts.

The companion uses these six deliberately small contracts:

These cases are handcrafted examples, not a representative sample of support traffic. The duplicate case checks the recorded write count; it does not execute a delivery retry. The retry article contains a separate test of that behavior.

Synthetic cases are useful for specifying a boundary before users encounter it. They cannot establish how often the boundary occurs or whether real users will understand the response. That requires evidence from the actual service.

Grade the outcome and the constraints separately

I would use deterministic checks for facts the application can verify: state, resource identity, write count, required approval, and tool failure. Then use human review, or a calibrated model grader, for judgments such as whether an explanation was understandable.

Anthropic's guide to agent evaluations recommends matching the grader to the task and cautions against prescribing unnecessarily rigid sequences. A valid implementation might use different read tools or ask a useful clarifying question. The grader should permit those alternatives.

Approval before a protected write is different. It is a required constraint, so ordering matters there. The fixture's grader checks that the approval came from a reviewer, referred to the same resource, and appeared before the write:

if event["type"] == "approval":
    if event["actor"] == "reviewer":
        approved.add(resource)
elif event["type"] == "write":
    if resource != task["resource"] or not task["write_allowed"]:
        critical.append("unauthorized_write")
    if task["approval_required"] and resource not in approved:
        critical.append("missing_approval")

In a real integration, those events must come from trusted application instrumentation. An assistant writing “approval received” in its own response is not equivalent evidence. Approval also needs the right binding for the operation: identity, resource, parameters, expiry, and current authorization. The toy fixture models only a resource-level approval inside an isolated trial.

The response check has a similar limit. completion_claim is a human-supplied label in the fixtures. The code does not understand arbitrary prose. That distinction keeps a simple demonstration from pretending to be a complete response evaluator.

A higher score can still block the release

Run the companion with Python 3.11 or later:

python3 -m unittest -v
python3 evaluate.py

The actual fixture output gives this comparison:

The candidate completes more tasks and has lower latency. It also creates one task without the required approval. A final-state-only grader counts that trial as successful. The constraint check rejects it.

The example's gate requires at least 80% of trials to pass and no critical violations. The baseline misses the pass-rate threshold. The candidate meets it but fails the authority rule. Those thresholds are chosen to demonstrate the gate; they are not release advice for another product.

The tested revision includes the contracts, recorded runs, report, and eight grader tests. The tests deliberately challenge missing approval, approval arriving after a write, approval for another resource, unauthorized writes, false completion, absent write evidence, and incomplete run coverage.

Keep repeatability visible

Two successful attempts at one easy task should not compensate for repeatedly failing a different task. Show both trial-level results and which tasks pass all their recorded trials.

The companion's “tasks passing both trials” is a direct count in a tiny fixture. It is not an estimate of production reliability, and the two invented records are not independent model samples. With real executions, retain per-task trials and report uncertainty rather than reading precision into a small percentage change.

I would compare variants on the same task set, initial state, and tool conditions. Record model, prompt, tool-schema, fixture, and environment versions. Reset state between trials. Otherwise a cache or leftover resource can improve the second run for reasons unrelated to the change being tested.

Test the grader before trusting the score

A grader can be wrong in both directions. It can accept a harmful result, or reject a valid alternative because its expected answer is too narrow.

For each important check, I would keep a known passing example and a near miss. For approval, that includes an approval for another case and one recorded after the write. For completion, it includes a confident success message with no committed effect. Have the domain owner review those judgments before automating a release decision.

Model-based judging needs its own validation. The MT-Bench and Chatbot Arena judge study documents position and verbosity biases in the models and tasks it tested. It does not quantify the error rate of a current judge on this support workflow. I would check agreement with human labels on local cases, inspect disagreements, and recheck after changing the judge or rubric. For pairwise comparisons, swapping answer order is a useful diagnostic.

Unknown evidence should remain unknown. The companion rejects a missing task/trial pair instead of quietly reducing the denominator. In a live evaluation, missing traces need investigation; they should not become unrecorded successes or disappear from the report.

Count the work spent on unsuccessful attempts

Cost per successful task should include the cost of attempts that failed:

cost per successful task =
  total cost across the evaluated attempts
  / number of successful tasks

Define the unit carefully. The companion uses independent recorded trials, so its denominator is successful trials. In a service where one customer task can trigger several retries, group that spend under the customer task before calculating the service metric. If nothing succeeds, the ratio is undefined, not zero.

The cost values here are invented integer microdollars, not provider prices. A real measurement should account for the model, speech, retrieval, tool, and other metered services that belong to the task.

Latency needs the same care. Measure the interval the user experiences, keep timeouts visible, and split slow cases by their cause. With only twelve values, the nearest-rank p95 in this fixture is simply the maximum. It says nothing dependable about a production tail. The voice latency article works through the separate boundaries for a spoken interaction.

Write the release decision before running the comparison

I would agree on the decision rules with the service owner: what blocks a release, what counts as an acceptable regression, which cases need review, and who handles a failed or uncertain task. Keep critical authority failures separate from negotiable quality tradeoffs.

Then preserve the report alongside the traces that explain it. A reviewer should be able to open a failed case and see the request, relevant records, actions, and resulting state without reconstructing everything from a score.

Passing that offline comparison earns the next controlled step. It does not establish that the workflow is ready for every user. A limited rollout still needs observation, a way to stop new actions, and an owner for incidents. The useful evaluation is the one that tells that owner exactly what the release has demonstrated and what remains untested.