· seed · Updated 2026-10-16
Acceptance criteria and stop rules for an AI pilot
A feasibility pilot should answer one decision, not prove that every part of a product can be built. Before a prototype, state the user task, current baseline, evidence needed to continue, and conditions that stop the experiment. Choose thresholds for this workflow; no single quality score fits every AI use.
Write the acceptance test first
- Question: what uncertainty will the pilot resolve?
- Scope: which user, inputs, output, and human step are included? What is out of scope?
- Baseline: how is the task handled today, and which representative examples can be compared?
- Pass criteria: define quality, safety, workflow fit, latency, and spending limits that matter.
- Owner: who reviews the sample and chooses continue, narrow, retest, or stop?
Include ordinary cases, missing information, edge cases, and a clear “do not answer” path. Keep the evaluation set fixed while comparing a change, and separate measured outcomes from reviewer opinion. If the task or data changes, record a new test rather than comparing unlike runs.
Illustrative pilot decision
This fictional example is a proposed test, not an observed result or universal benchmark. A team tests an assistant that drafts product descriptions from an approved catalog. Before building, it might set a sample of 20 items, require at least 18 drafts to meet its own completeness rubric, allow zero invented product attributes, and cap the pilot at an agreed time and spend. It also records today’s manual process as a baseline. These numbers are choices for this example; another workflow needs thresholds suited to its risk and value.
When to stop, narrow, or retest
Stop if the pilot exceeds its time or spend cap, cannot use required data under approved terms, or repeatedly fails a critical safety or accuracy rule. Narrow the scope when only one input class is unreliable. Retest after a named change, using the same representative set plus a separate check for the newly addressed case. If the sample is too small or unrepresentative, call the result inconclusive instead of a pass.
Feasibility checklist
- One decision question and a bounded user task.
- Current baseline and representative evaluation examples.
- Written definitions of pass, fail, stop, and inconclusive.
- Pre-agreed limits for time, spending, data access, and human review.
- Named owner and next-step decision recorded with the evidence.
For a fuller stop-rule charter, see kill criteria for AI experiments. Evaluation sample design is covered by minimum evaluation sets, and ways to reduce scope are in scope cuts for early experiments.
For experiment design support, see our services.