· 4 min
Kill criteria for AI experiments
Kill criteria are useful only when the team writes them before the prototype becomes emotionally expensive. They set a fair test for whether to continue, narrow the idea, change an assumption or stop. The threshold should fit the task and the cost of failure; there is no universal conversion rate or quality score that every AI experiment must hit.
Write a one-page experiment charter
- Question: what important uncertainty will this test resolve?
- User and job: who will try it, and what are they doing today?
- Evidence: what observation would show that the outcome is useful?
- Quality floor: which errors are unacceptable, and which can a person correct?
- Budget: what are the time, model-spend and data-access limits?
- Owner and date: who makes the decision, and when will the team review the result?
Choose a decision before the demo
Define three outcomes. Continue when the agreed user and quality signals are present and the cost remains within the cap. Change one assumption when the problem is real but a specific input, workflow or model choice failed. Stop when the user need is absent, the required data cannot be obtained responsibly, the output cannot meet the quality floor, or the economics do not fit the intended use.
For example, a hypothetical document-summary test might miss the target because the source files are inconsistent. That is evidence about the input, not automatic proof that summarization is a bad product. The team can decide whether a narrower file type is valuable enough to retest. If the only workable path requires data the organization cannot approve, stop that version instead of relaxing the boundary after the fact.
Review the result without moving the goalposts
Bring the original charter, examples, failed cases, actual spend and reviewer notes to the decision meeting. Separate observations from interpretations. Record the decision, the assumption being changed and the new limit if you authorize another test. One short retest can be reasonable when it addresses a named cause; repeated extensions without new evidence are not a strategy.
The NIST AI Risk Management Framework is a voluntary reference for considering context, measurement and ongoing management of AI risk. Use it to improve the experiment plan, not to outsource the go/no-go call. For selecting a focused use case, see choosing an AI seed wedge; for the investment gate after a positive result, read from seed to build sprint. Explore services or contact.