How Do You Test an AI Workflow That Says Done but Gets It Wrong?

Test the result outside the AI's account of what happened. Define the correct outcome, the changes it may make, and the evidence required before it can claim completion. Then deliberately introduce conditions under which an apparently successful conversation can leave the work unfinished, incorrect, or unauthorized. A second model saying the response looks good does not establish that the requested business operation occurred.

I'd begin with the sentence your user is likely to trust: “I've completed that for you.” Which system can prove it? If the answer is another generated paragraph, you have a gap in the evaluation before you have a score. A fluent explanation can conceal a missing transaction, an update to the wrong record, or a partially completed request. Those are different failures, with different remedies.

This article develops a method for testing that gap. It combines research on agent evaluation, behavioral testing, model judges, and self-correction with an original workflow acceptance framework. The examples are hypothetical; I haven't tested a client's system or measured a model here. The cited experiments establish results in their reported settings, not the capabilities of every current product. My concern is the evidence you need for a bounded release decision.

Define completion before designing the prompt

NIST's Generative AI Profile describes confabulation as confidently presented erroneous or false content (Autio et al., 2024, §2.2). False completion is the operational case I want to isolate: the application represents a task as finished when its completion conditions haven't been met. It can arise from model output, orchestration logic, stale observations, or the interface's interpretation of a tool response. It isn't useful to assume every instance has the same cause.

I would write a completion contract with four separately evaluated properties. Answer correctness concerns what the application tells the user. Outcome correctness concerns the authoritative business state. Constraint compliance concerns which actions were permitted throughout the attempt. Report fidelity concerns whether the final message accurately describes what the evidence establishes. A response can satisfy one property while failing another. A correct balance displayed after an unauthorized account lookup remains unacceptable.

Take a fictional support assistant authorized to cancel one eligible order. Completion might require that the designated order belongs to the verified customer, its status permits cancellation, the cancellation commits, any required confirmation is recorded, and the final response describes the actual state. If refund settlement occurs later, “cancellation accepted; refund pending” may be accurate while “your refund has arrived” is false. The contract must distinguish those milestones before anyone labels the example.

Ambiguous requests need a different contract. “Cancel my recent order” may identify several orders. Asking which one is intended can be the correct next step; choosing one without justification is not successful autonomy. Define acceptable clarification, refusal, escalation, and partial completion as explicit outcomes. Otherwise the evaluator rewards confident guessing and penalizes the behavior you wanted.

This is consistent with the AI RMF's emphasis on evaluation under conditions similar to deployment and documenting generalization limits (Tabassi, 2023, MEASURE 2.3–2.5). The four-part contract is my proposed implementation, not a NIST certification scheme. You still need the process owner to settle what “completed” means when the business rules themselves are unclear.

Give the evaluator access to evidence the agent cannot invent

An evaluation needs an oracle: a defensible way to decide whether the expected result occurred. For a transaction, that might be a read from the authoritative test database plus a record of executed operations. For a sourced answer, it might be an expert-approved set of propositions tied to a frozen document version. One universal similarity score cannot serve both purposes.

WebArena evaluates functional task completion, including checks on resulting content rather than requiring one exact sequence of actions (Zhou et al., 2024, §3.2). The original τ-bench paper compares the final database state with an annotated goal state in simulated user-agent conversations (Yao et al., 2024, §3). Those designs provide useful precedents for checking effects. The τ-bench authors also explicitly note that a passing reward can miss a policy violation, such as acting without required confirmation. That limitation matters: a correct final state is only part of a defensible acceptance test. Neither benchmark constitutes evidence that your business workflow works.

I'd keep expected outcomes in a fixture the agent cannot edit. Start with a known state, run the application, and compare the resulting state with permitted outcomes. Inspect protected records as well as the target. An assistant that cancels the right order and also changes another customer's address has not passed merely because the requested cancellation is present. When several correct outcomes exist, encode the acceptable set instead of demanding one incidental representation.

Final state also cannot reveal every prohibited action. An agent might expose restricted information and subsequently restore the database to its original values. For that reason, outcome checks should be paired with an execution record generated by the tool boundary, not reconstructed from the assistant's narration. Record the operation, principal, resource, parameters needed for evaluation, result, and correlation identifier. Minimize sensitive values and restrict access to the trace.

I would test the evaluator itself with known faults: the wrong order changed, an extra side effect, an omitted subtask, and a fabricated completion message. It should reject each for the intended reason. Also include valid alternative solutions to expose an overstrict oracle. These are proposed tests, not results reported here. An evaluator that simply reproduces the application's assumptions can give you two agreeing implementations of the same mistake.

Build cases around state changes and uncertainty

A large prompt collection is not necessarily a useful case library. Each case should specify the starting state, user objective, relevant policy, permitted actions, expected observations, acceptable terminal outcomes, and failure consequence. That lets you explain why a failure matters and reproduce the conditions that caused it.

CheckList distinguishes minimum-functionality tests, invariance tests, and directional-expectation tests for NLP behavior (Ribeiro et al., 2020, §2). I would adapt that logic to workflow evaluation. A minimal case isolates one cancellation with clear authorization. An invariance case changes harmless wording while preserving the intended operation. A boundary pair changes a decision-relevant fact, such as whether the order has shipped, and requires a different action. These workflow cases are my extension of the method, not experiments from the paper.

Keep the pairs interpretable. If you change the user, order, policy version, and tool availability simultaneously, a changed result tells you little about which condition mattered. Isolated changes support diagnosis; compound cases then test realistic interactions. Both are useful, but they answer different questions. A good evaluation needs coverage of ordinary work as well as deliberately difficult cases.

I would divide the library by operational consequence: routine completion, missing information, prohibited action, unavailable dependency, partial execution, and disputed evidence. Include cases where the correct behavior is to finish and cases where stopping is required. A system can appear safer simply by refusing everything; a system can appear productive by silently ignoring difficult requests. The case mix must expose both strategies.

Protect a release set from routine prompt tuning. Kapoor et al. (2024, §§4–5) identify inadequate holdouts and benchmark overfitting as problems in agent evaluation. My application here is to separate development examples from a versioned acceptance set and retain genuinely new variants. Once a failed acceptance case becomes a development example, keep it as a regression test but don't pretend it remains unseen evidence. Record the transfer and replenish the held-out coverage.

Make the tool fail after something has happened

Testing only clean tool errors misses the cases I would be most concerned about. A request can fail before execution, after execution but before acknowledgment, or after only part of a compound operation. Those conditions require different responses. A timeout is an observation about the communication path; by itself, it doesn't establish whether the intended state change committed.

In a controlled test environment, I would arrange for an order cancellation to commit while its acknowledgment is withheld. The assistant should reconcile the state through an approved read or another documented recovery path. Blindly repeating a consequential operation may create an extra effect unless the service's idempotency contract prevents it. Claiming failure without checking may also mislead the user. The expected behavior depends on the actual API contract, which the fixture must represent.

ReAct makes interaction with external observations part of the agent trajectory, interleaving reasoning and action (Yao et al., 2023, §2). That supports examining the observation-action loop, but it doesn't guarantee reliable recovery. My test would vary the observations themselves: a stale read, a delayed completion, a conflicting status, an empty response, and a successful transport response containing a business-level rejection.

I would also distinguish detection from diagnosis. If a run ends incorrectly, preserve the observations available at each decision point before changing the prompt. An unavailable fact calls for better observation or an uncertainty path; an available fact ignored by the model may call for different orchestration; an unauthorized action accepted by a tool requires an enforcement fix. These are hypotheses to investigate against the trace. Rewriting the prompt after every failure can obscure a defect elsewhere in the system and leave the same failure reachable through different wording.

The table below is a hypothetical design for a cancellation workflow. It contains expected checks, not measured outcomes.

Injected condition	Evidence to inspect	Acceptable reporting boundary
Cancellation commits; acknowledgment is lost	Authoritative order status and operation identifier	Confirm only after reconciliation; otherwise state uncertainty
Cancellation commits; required notification fails	Order record and notification status separately	Explain partial completion and the pending step
User names two orders but authorizes only one	Authorized scope and both order records	Change only the authorized order; clarify the other
Policy changes during the attempt	Policy version and decision timing	Follow the specified transition rule or escalate
Read tool returns stale pre-cancellation state	Version or freshness evidence, where available	Avoid treating an old observation as proof of current state

Reset fixtures between independent trials. Otherwise one run's successful cancellation can make the next run look successful without doing any work. For deliberate continuation tests, preserve state explicitly and label the test as a resumed attempt. Also define a maximum observation window. A workflow that never reaches a terminal state must not disappear from the denominator because the evaluation runner stopped waiting.

Use model judges for judgments they can support

Some acceptance conditions are straightforward to automate. Others involve explanation quality, completeness, or whether an escalation message gives a human enough context. A model judge can help scale those assessments, provided you measure the judge's own behavior rather than treating its score as ground truth.

Zheng et al. (2023, §§3.3–4) examine position, verbosity, and self-enhancement biases in LLM judging, alongside agreement with human preferences in their benchmarks. The relevant lesson is conditional: a judge can be useful while still being sensitive to presentation and evaluation design. Preference agreement in that study does not validate a judge's ability to determine whether a particular financial or administrative operation occurred.

I'd calibrate against a domain-reviewed set containing convincing wrong answers, terse correct answers, justified uncertainty, and misleading statements of completion. Review errors by class, especially false acceptance of consequential mistakes. High overall agreement can coexist with failure on the few examples that determine whether you should release. Keep the judge's prompt, model version, evidence supplied, scoring rules, and adjudication decisions with the evaluation record.

Where a judge compares two responses, reverse their order and investigate inconsistent decisions. Where it assesses correctness, supply the authoritative evidence rather than asking it to infer hidden state from prose. It should be able to return “insufficient evidence.” That outcome needs a defined route to human adjudication or an unresolved result; it must not silently become a pass.

Independence is a property of the evidence and review process, not merely a different model name. Two models reading the same incorrect summary may agree for the same reason. For state-changing operations, I'd use direct state checks wherever possible and reserve model judgment for the portions that genuinely require interpretation. If you also need to examine malicious instructions in retrieved material, the separate analysis of retrieved content and agent authority addresses that trust boundary.

Treat self-correction as a candidate improvement, not a verdict

“Ask the model to check its work” is a reasonable technique to test. It is an inadequate definition of acceptance. The important question is whether the correction receives new, reliable evidence and whether it improves the outcomes that matter under the available time and cost budget.

Self-Refine uses a model to generate feedback on its own output and iteratively revise it, reporting improvements across its evaluated tasks (Madaan et al., 2023, §§2–3). Huang et al. (2024, §§2–5) examine intrinsic self-correction in reasoning without external correctness feedback and identify limitations, including degraded performance in evaluated conditions. These studies differ in tasks, prompts, metrics, and experimental setups. Reading either as a universal statement about all present-day AI systems would overstate the evidence.

I would compare the complete candidate workflows on the same held-out cases: the original application, the application with self-review, and, where feasible, a version with an external verification step. Record correct-to-incorrect revisions as well as repairs. A net gain can conceal newly introduced errors in an important category. If a model changes a correct answer after being challenged, that is a different failure mechanism from never finding the answer.

The comparison should include latency, tool use, and review burden. Giving one workflow several attempts and another only one may answer a budget question rather than an architectural one. Kapoor et al. (2024, §§2–3) argue for evaluating agent accuracy alongside cost. I'd therefore make the intended deployment budget part of the experiment. The selected workflow must earn its additional effort through improved task outcomes, not a more elaborate-looking transcript.

Separate reliability from the chance of one lucky success

Repeated trials reveal behavior a single demonstration cannot. The original τ-bench distinguishes pass@k, success in at least one of k independent trials, from pass^k, success in all k trials, averaged across tasks (Yao et al., 2024, §3). These metrics answer different questions. A workflow that sometimes succeeds after several attempts is not necessarily one that behaves consistently when a user relies on it once.

Retries are particularly awkward when actions have side effects. For an offline drafting task, selecting the best candidate may be legitimate if the selection method is also available in deployment. For a transaction, unsuccessful attempts may already have changed something. You cannot discard those attempts from the record and report only the successful ending. Evaluate the retry policy as part of the workflow, including how it detects failure and prevents unintended repeated effects.

I'd report three separate measures. Task success is the proportion of evaluated cases satisfying the completion contract. False-completion incidence is the proportion of all evaluated cases where completion is claimed but the contract is unmet. False completion among claimed successes uses only cases where the application claims success as its denominator. The last measure speaks directly to whether users can trust “done,” but it must be shown with the claim rate and overall task success.

Suppose, purely hypothetically, an application claims completion on 80 of 100 cases and 8 of those claims are wrong. False completion is 8% of all cases and 10% of claimed successes. Those numbers alone don't establish total task success: the remaining cases need evaluation too. If there are no completion claims, the conditional rate is undefined, not zero. Reporting that distinction prevents universal abstention from looking like perfect reliability.

Some outcomes become observable only later. A request may be accepted immediately but rejected during subsequent processing. I'd distinguish the evaluation window for acknowledgment from the window for terminal completion, then report cases still awaiting adjudication. If the product promises only submission, evaluate that promise accurately. If it promises completed processing, a successful submission response is insufficient. Delayed labels also affect monitoring: an apparently healthy dashboard can simply contain transactions whose failures haven't arrived yet. Compare cohorts with comparable observation windows, preserve unresolved cases, and update their status when authoritative evidence becomes available.

Use separate rows for high-consequence failure categories. I would not average an unauthorized action into the same score as a slightly awkward explanation. Predefine which failures stop release, which require a narrower scope or compensating control, and which are tolerable defects. Severity assignments need reasons tied to the workflow; an attractive weighted score cannot supply those reasons afterward.

State what a clean test run cannot establish

Zero observed failures is encouraging, but the sample size and selection process determine what you can infer. For an illustrative fixed failure probability p across n independent, identically distributed trials, the probability of observing zero failures is (1 − p)^n. Setting that probability to 0.05 gives the one-sided 95% exact upper confidence bound, p = 1 − 0.05^(1/n). This is the zero-event specialization of exact binomial confidence-limit reasoning; NIST documents the general method (National Institute of Standards and Technology, n.d.).

With 100 such trials and zero failures, the upper bound is approximately 2.95%. With 1,000 it is approximately 0.299%. These are calculated examples, not measurements of an AI system. A frequentist confidence bound also does not mean there is a 95% probability that this particular unknown rate lies below the displayed value. Its interpretation depends on the repeated-sampling procedure and assumptions.

Those assumptions are substantial. Handpicked adversarial cases do not constitute a random sample of ordinary traffic. Hundreds of variations on one template do not establish broad workflow coverage. Repeated attempts during one shared outage may be correlated. Continued tuning against the test set changes the procedure. Applying the formula mechanically in these settings can create an appearance of precision without the corresponding evidence.

I would present two complementary views: a risk-driven coverage account explaining which failure mechanisms were challenged, and statistical estimates only for a defensible sampling design. Keep strata visible when customer types, task classes, or policy conditions have materially different consequences. Additional examples help only if they add relevant information. They cannot repair a missing oracle or an untested authorization boundary.

For a rare but unacceptable action, zero observed failures may be a release gate without being proof that the action is impossible. That distinction should remain explicit. Preventive controls, constrained permissions, and reversibility may carry more of the assurance burden than a feasible number of sampled conversations.

Include the reviewer and the operating environment

A human approval step creates another workflow to evaluate. I'd test whether the reviewer can see the original request, the actual proposed action, the supporting evidence, and any uncertainty before the decision becomes consequential. A person asked to approve a generated summary without access to the underlying state cannot independently resolve the failure this article is concerned with.

Construct approved exercises containing incorrect proposals, correct proposals, and cases that genuinely need clarification. Measure what the reviewer detects, what gets approved, and how long unresolved cases remain queued. Use appropriate organizational authorization and participant safeguards. These are proposed acceptance exercises; no human-subject study is reported here. The interface should support rejection and escalation without making approval the only practical way to finish the task.

The test environment also needs a documented relationship to deployment. A mocked payment tool can be useful for fault injection, but it doesn't establish the behavior of a real payment service. A simplified user simulator may omit interruptions, misunderstandings, or changing instructions. Record those differences and identify which require a separate integration or operational check. Otherwise a carefully controlled test can be mistaken for evidence about conditions it intentionally excluded.

Sculley et al. (2015, §§3, 6–7) describe data dependencies, configuration issues, and changes in the external world as sources of machine-learning system debt. For this workflow, I would version the model configuration, prompts, tool schemas, permission model, business rules, retrieval snapshot, fixtures, and evaluator together. A change to any of those may invalidate part of the release evidence even when the model name stays the same.

After release, monitor signals that can reveal silent failure: reconciled transactions, user corrections, reopened requests, incomplete subtasks, and discrepancies between claimed and observed completion. Decide who investigates and what disables further autonomous action. Retain enough evidence to diagnose failures without turning evaluation logs into an uncontrolled copy of sensitive records. The companion article on privacy across the RAG lifecycle examines that data-management problem in more detail.

Make a release claim small enough to defend

A useful release decision states which tasks the system may perform, under which permissions, with which evidence of completion and which fallback. It identifies the tested configuration, excluded cases, unresolved questions, critical failure results, sampling limitations, and accountable decision-maker. NIST's AI RMF calls for participation by internal experts outside front-line development and/or independent assessors in regular assessments (Tabassi, 2023, MEASURE 1.3). That review should challenge the acceptance contract as well as the score.

The resulting decision need not be a binary choice between full autonomy and abandonment. Suppose the evaluation shows reliable preparation of cancellation requests but unreliable recovery after ambiguous tool responses. A narrower candidate could prepare the request while requiring a reviewer to resolve uncertain state before execution continues. That candidate needs its own acceptance evidence: the reviewer must receive the relevant facts, and the application must actually stop at the boundary. Merely adding an approval label to the existing interface does not establish that the failure path has changed. I would record the original failure, the changed authority, and the specific retest required before widening the scope again.

There is a fair objection that this approach costs more than a quick benchmark comparison. For a low-consequence drafting assistant, a proportionate evaluation may indeed be much lighter. I would still check whether the output meets its purpose, but I would not demand transaction-level evidence for a system that cannot transact. The burden should follow the consequence, uncertainty, and available recovery path. Narrowing the task can be a better investment than building an elaborate evaluation around excessive authority.

This framework also has limits. An oracle can be wrong, a scenario library incomplete, and a reviewed policy internally inconsistent. Human adjudication can introduce disagreement rather than remove it. The method does not establish universal safety, legal compliance, or immunity to previously unseen failures. Its value is more concrete: it makes the release claim inspectable and exposes where evidence is still missing.

If your AI reports success but you cannot independently show what it completed, request a briefing on AI workflow assurance. Bring one bounded workflow and the decision you need to make. I would help define its completion contract, select consequential failure cases, and specify the evidence required for release. That gives you a practical basis for deciding where automation can proceed, where a person needs to intervene, and where the system still has to prove its work.