How to Test Non-Deterministic AI Features

We hear the same complaint from teams adding their first AI feature. The tests that worked for the rest of the product start failing at random, because the feature never gives the same answer twice.

A non-deterministic AI feature is one where the same input can produce different outputs on different runs, which is normal for large language models (LLMs) and many other models. That breaks the core assumption of traditional testing, that a given input always yields one expected output, so the tool for the job is an evaluation, or eval, that scores many outputs against criteria.

This post covers why non-deterministic AI breaks normal tests, a seven-step way to test it, how the grading methods compare, the mistakes we see most, and when exact-match tests still apply.

Why Non-Deterministic AI Breaks Traditional Software Tests

Non-deterministic AI breaks traditional tests because those tests compare an output against one correct value, and a model can return many acceptable values. We see teams respond by deleting the tests or by asserting such loose conditions that nothing ever fails, and neither is testing.

Variation comes from several places at once. Sampling settings, model version updates from the provider, changes to the prompt or retrieved context, and small differences in input all shift what comes back. Some of that variation is harmless rewording, and some of it is a wrong answer that happens to look plausible.

The task is to separate the two automatically. That is what evaluations do. OpenAI's documentation defines evaluations, often called evals, as tests of model outputs to ensure they meet style and content criteria that you specify, and compares the approach to behavior-driven development, where you specify how the system should behave before building and testing it.

How to Test Non-Deterministic AI Features Step by Step

To test non-deterministic AI features, we follow seven steps, from writing measurable success criteria to monitoring in production. The steps turn "it seems to work" into a pass rate that can fail a build.

1. Write Success Criteria You Can Measure

Start by writing what a good output looks like in terms a test can check. Anthropic's guidance on building evals says good criteria are specific, measurable, achievable and relevant, and contrasts a vague goal such as classifying sentiment well with a target F1 score on a held-out test set.

For a feature that drafts email subject lines, criteria might include a length range, the presence of the product name, a neutral or positive tone and correct grammar. Each one can be checked on every output, whatever the exact wording.

2. Build an Eval Set From Real and Edge-Case Inputs

Collect inputs that look like real usage, then add the awkward ones, such as empty fields, very long text, other languages, hostile prompts and inputs that should be refused. Anthropic's guidance puts this as being task-specific, designing evals that mirror the real-world task distribution, including edge cases.

Start with a few dozen cases and grow the set every time a bug reaches users. The eval set becomes the product's memory of what has gone wrong before.

3. Test Properties, Not Exact Strings

Replace equality checks with property checks, such as valid JSON, a length limit, the required fields and the absence of content that is never allowed.

Property checks are cheap, fast and deterministic even when the output is not. They should cover everything that can be checked with code before any slower grading method is used.

4. Choose a Grading Method for Each Criterion

Match each criterion to the cheapest method that judges it reliably. Code handles structure and exact categories, a second model can grade tone or relevance against a rubric, and people judge nuance, safety and anything the business cannot get wrong.

The same guidance says to automate when possible and to prioritize volume over quality, since more cases with automated grading usually beat a few hand-graded ones. We keep a smaller human-reviewed sample to check that the automated graders still agree with people.

5. Run Each Case Several Times and Set a Pass Rate

Because outputs vary, one run per case proves little. Run each case several times, record how often it passes, and set a pass-rate threshold for the whole set rather than demanding perfection on every run.

The threshold is a product decision. A drafting assistant can tolerate occasional weak suggestions, while a feature that touches money or medical information needs a much higher bar and human review on failure.

6. Rerun the Evals on Every Model, Prompt or Data Change

Treat prompts, model versions and retrieval data like code. Any change to them reruns the eval set, and the build fails if the pass rate falls below the baseline for the previous version.

OpenAI's guidance calls writing evals an essential part of building reliable applications, especially when upgrading or trying new models. Provider model updates are the change teams most often forget to test.

7. Monitor Behavior in Production

Pre-release evals cannot cover every real input, so keep measuring after launch. Log inputs and outputs where privacy rules allow, sample them for review, and track user signals such as edits, retries and complaints.

Failures found in production go back into the eval set. Over time that loop is what makes the feature dependable, and it pairs naturally with designing AI features that degrade gracefully when the model gets it wrong.

Grading Methods for AI Output Compared

Grading methods differ in speed, cost and how much nuance they can judge, so most eval sets combine them. We use this comparison when deciding which criterion gets which grader.

Anthropic's guidance also recommends using a different model to grade than the one being evaluated, to avoid bias. We apply that rule wherever a model grades another model's output.

Mistakes Teams Make When Testing AI Features

The most common mistakes in testing AI features come from treating the model like ordinary code or like magic. We see these patterns most often:

• Asserting exact outputs, which produces brittle tests that fail constantly and teach the team to ignore failures.
• Relying only on manual spot checks, which do not scale and cannot show if a model update made things better or worse.
• Testing with a small, tidy dataset that misses the messy inputs real users send.
• Trusting automated metrics without ever comparing them with human judgment.
• Testing the model alone instead of the whole feature, including retrieval, prompts and the user interface around it.

Each of these hides risk until users find it. Choosing the right tool for the job matters too, and sometimes the honest answer is that a rules engine is better than a machine learning model because it can be tested exactly.

When Exact-Match Tests Still Belong in an AI Feature

Exact-match tests still belong wherever the feature has a deterministic part, and most AI features have several. We keep ordinary unit and integration tests for parsing, validation, permissions, storage and every piece of code around the model.

Some model tasks are effectively deterministic too, such as classification into fixed categories, where an exact match against the expected label is the right test. Constraining outputs with a schema also turns many open-ended responses into structured ones that code can check.

The NIST AI Risk Management Framework, released in January 2023 for voluntary use to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, organizes the work into four functions, Govern, Map, Measure and Manage.

Measuring is one of them, which is a useful reminder that testing is part of running the feature, not a phase before launch. This is how we approach AI work within product engineering, and it ties into understanding the hidden costs of running AI in production.

Turn Non-Deterministic AI Into a Pass Rate You Can Trust

Testing non-deterministic AI is not about forcing the model to repeat itself. Write measurable criteria, build an eval set from real and edge-case inputs, check properties with code, grade the rest with models and people, run cases several times against a pass-rate threshold, rerun on every change and keep monitoring. We would rather ship an AI feature with a known pass rate than one that only seemed fine in a demo.

If you are planning an AI feature and want a second view on how to test it, the team at Better Software can help you design the eval set.

Frequently Asked Questions

How to Handle Non-Deterministic AI Tests?

Handle non-deterministic AI tests by checking properties and pass rates rather than exact outputs. We run each eval case several times, grade with code where possible and with a model or a person where needed, and fail the build when the overall pass rate drops below an agreed threshold.

Can AI Be Non-Deterministic?

Yes, many AI systems are non-deterministic, meaning the same input can produce different outputs. We see it most in language models, where sampling and model updates change wording and sometimes substance, which is why testing them relies on evaluation sets rather than single expected answers.

What Are Non-Deterministic Tests?

Non-deterministic tests are tests whose result can change between runs without any change to the code, sometimes called flaky tests. We treat them differently for AI features, measuring a pass rate across repeated runs instead of expecting every run to pass or fail identically.

How to Test AI Capabilities?

Test AI capabilities with an evaluation set that mirrors real tasks, including edge cases, and clear success criteria for each. We score outputs with automated checks, model-based grading and human review, then compare results across model versions before deciding which one to ship.

Are LLMs Deterministic or Non-Deterministic?

LLMs are generally non-deterministic in practice, because sampling and serving details can change outputs for the same prompt. We plan for variation even when settings aim for consistency, and we test behavior with repeated runs and pass rates rather than assuming identical answers.