How to Test LLM Applications and AI Agents

The first thing that breaks when you put a language model in production is your test suite. Not because the tests are bad, but because assertEquals assumes the same input produces the same output, and that assumption no longer holds.

It does not hold even when you try to force it.

Temperature 0 does not mean deterministic

This is the single most load-bearing fact in the subject, and it is widely misunderstood.

Anthropic’s glossary states it plainly: “Even with temperature set to 0, the results will not be fully deterministic and identical inputs may produce different outputs across API calls. This applies both to Anthropic’s first-party inference service and to inference through third-party cloud providers.”

OpenAI offers a seed parameter and documents its limits just as directly: the system makes a best effort to sample deterministically, but “determinism is not guaranteed”, and you are told to watch the system_fingerprint for backend changes underneath you.

There is a stronger version of this on current Claude models: temperature is not merely unreliable, it is rejected. Models released after Claude Opus 4.6 deprecate the parameter — 1.0 is accepted for backwards compatibility and any other value returns a 400. top_p and top_k are the same. If your eval harness sets temperature=0 out of habit, it will stop working rather than quietly do nothing.

How non-deterministic, in numbers

Thinking Machines Lab published the concrete measurement in Defeating Nondeterminism in LLM Inference: 1,000 completions at temperature 0 produced 80 unique outputs. The most common appeared 78 times. Divergence typically began around token 103 — meaning the first hundred tokens often match and the difference emerges later, which is exactly the pattern that makes a shallow smoke test pass while a real one fails.

Their explanation is worth knowing because it corrects a common one. The usual folk account blames concurrency and floating-point atomics; the paper points out that modern LLM forward passes contain no atomic adds at all. The real cause is lack of batch invariance — your request is batched with others, batch composition varies with server load, and the kernels produce subtly different results for different batch shapes. With batch-invariant kernels they got all 1,000 completions identical, at roughly 2.1× the latency.

The practical reading: non-determinism here is a property of the serving infrastructure, not something you can configure away from the client.

What replaces assertions

You stop asking “is the output equal to this” and start asking “is the output good enough, often enough.” Concretely, an eval suite:

1. A fixed set of representative inputs, drawn from real failures rather than imagination.
2. A grader per input — code where possible, a model where not.
3. An aggregate score you can threshold.

Anthropic’s guidance on developing test cases gives a design rule that cuts against instinct: “Prioritize volume over quality: More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals.” Their companion piece on evals for AI agents sets a starting size: “20-50 simple tasks drawn from real failures is a great start.”

That framing — volume over refinement, real failures over invented ones — is the opposite of how most teams write their first eval set, which is usually a dozen hand-crafted prompts that feel representative and are not.

Three grader classes, with real trade-offs

Grader	Strengths	Costs
Code-based	Fast, cheap, objective, reproducible, easy to debug	Brittle to valid variations
Model-based	Flexible, scalable, captures nuance	Non-deterministic, expensive, needs calibration
Human	Gold standard quality	Expensive, slow

Those characterisations are Anthropic’s own. The practical guidance that falls out of them is to push as much as possible into the first row — structure the task so a classification, a JSON schema, a regex or a tool call can be checked mechanically — and reserve model grading for genuinely open-ended output.

LLM-as-judge: the biases are measured

Using a model to grade a model works, and its failure modes are documented rather than hypothetical. The reference is Zheng et al., MT-Bench and Chatbot Arena, NeurIPS 2023.

Position bias. When the two candidate answers were swapped, GPT-4 gave a consistent verdict only about 65% of the time. GPT-3.5 managed 46.2%, Claude-v1 23.8%. The paper’s own summary: only GPT-4 was consistent in more than 60% of cases. If your judge sees answers in a fixed order, a third of your verdicts may be about ordering.

Verbosity bias. Against a “repetitive list” attack — the same content restated at greater length — GPT-3.5 and Claude-v1 were fooled 91.3% of the time. GPT-4 failed 8.7%.

Reference-guided grading helps a lot. On maths problems, supplying a reference answer cut the grading failure rate from 70% to 15%.

Self-preference — state this one carefully. The paper reports GPT-4 favouring its own answers by about 10 percentage points and Claude-v1 by about 25, and then explicitly cautions that “due to limited data and small differences, our study cannot determine whether the models exhibit a self-enhancement bias.” The number circulates widely without that caveat. Quoting one without the other is how a tentative finding becomes an internet fact.

The mitigations follow directly: prefer pairwise comparison to open-ended scoring, randomise answer position, supply reference answers where they exist, and calibrate the judge against human labels before trusting it. Anthropic adds a useful escape hatch — instruct the grader to return “Unknown” when it lacks the information to decide, rather than forcing a verdict.

The framework landscape

All of these exist and are real; maturity varies more than the marketing suggests.

Framework	Licence	Status
promptfoo	MIT, open-core	Very active
DeepEval	Apache-2.0, open-core	Very active
Inspect (UK AI Security Institute)	MIT	Very active
Arize Phoenix	Elastic-2.0 — source-available, not OSI open source	Very active
Braintrust	Commercial hosted (autoevals is MIT)	Active
LangSmith	Closed backend, MIT client	Active
Ragas	Apache-2.0	Dormant — last release January 2026
Evidently	Apache-2.0, open-core	Decelerating
OpenAI Evals (OSS)	MIT	Effectively unmaintained since 2024

Two things worth flagging, because they change decisions.

Arize Phoenix is not open source in the OSI sense. It is Elastic License 2.0 — source-available. That distinction matters if your organisation has a licence policy, and it is routinely described as “open source” in write-ups.

OpenAI’s hosted Evals platform is being retired. OpenAI notified developers on 2026-06-03; it goes read-only on 31 October 2026 and shuts down 30 November 2026. Their own deprecation notice points users to promptfoo as the migration path — a vendor naming an independent OSS project as the successor to its own product is an unusually clear signal about where this tooling is settling.

Getting eval results into CI

Here is the gap, and it is larger than you would expect for a category this well funded.

Framework	JUnit XML	CTRF
promptfoo	Yes, native	No
DeepEval	Via pytest passthrough	No
LangSmith	Via pytest passthrough	No
Ragas, Phoenix, Evidently, Inspect, Braintrust	No	No

promptfoo is the only one with native support, and it works by file extension rather than a format flag:

promptfoo eval -o results.json -o results.junit.xml

Two traps. .xml is not the same as .junit.xml — promptfoo’s own docs note that plain .xml is not a JUnit-compatible CI format. And this is recent: JUnit output landed in version 0.121.10 in May 2026, so anything written before that will tell you it does not exist.

DeepEval and LangSmith run under pytest, so --junitxml=report.xml works — but that is pytest’s reporter, not a feature of the eval tool, and it reports test outcomes rather than eval scores.

Nothing in this category emits CTRF, which is a genuine shame, because CTRF’s retry and per-attempt fields map neatly onto exactly what eval runs produce — repeated attempts at the same task with varying outcomes.

Where this connects to ordinary testing

The vocabulary is different but the underlying shape is familiar. A single eval run is a sample, not a verdict. Whether your application actually regressed is a question about the distribution across runs — which is the same reason a single test report cannot tell you whether a test is flaky. Non-determinism in a normal suite is a defect to be traced; here it is a design property to be measured, but in both cases the answer comes from history rather than from any one execution.

The difference in practice is that a flaky test is a problem and a varying eval score is the ground truth. You are not trying to eliminate the variance, only to know it — which means running each case more than once. Anthropic names the two standard measures: pass@k, the likelihood of at least one correct answer in k attempts, and pass^k, the probability that all k succeed. Which one you gate on says a lot about your application: a code generator can live with pass@k, a customer-facing classifier probably cannot.

For transparency: Qualflare is our product, a test observability layer that reads results from CI and analyses them across runs. It has no eval-specific features today, and this post is not a pitch for one — the honest position is that the eval ecosystem’s CI reporting story is immature, and promptfoo’s JUnit output is currently the bridge. Where our own view is relevant is the broader map of what AI does and does not do in testing, in AI in software testing: what actually works.

Frequently asked questions

Can you make LLM output deterministic with temperature 0?

No. Anthropic’s documentation states plainly that even with temperature set to 0 the results will not be fully deterministic, and that this applies to its own inference service and third-party clouds alike. OpenAI offers a seed parameter but documents it as best-effort, saying determinism is not guaranteed. On Claude models released after Opus 4.6 the temperature parameter is deprecated outright — any value other than 1.0 is rejected with a 400 error.

What replaces assertions when testing LLM applications?

A graded eval suite. You fix a set of representative inputs, then score the output rather than comparing it for equality — with code where the task allows (exact match on a classification, JSON schema validation, a regex, a tool call), and with a model where it does not. Scores are aggregated into a pass rate you can threshold, which is what makes the result usable as a gate.

Is LLM-as-judge reliable?

Reliable enough to be useful, with measured biases you have to design around. The MT-Bench work found GPT-4 gave consistent verdicts when the two answers were swapped only about 65% of the time, and that GPT-3.5 and Claude-v1 were fooled by a repetitive-list attack in 91.3% of cases. Reference-guided grading cut math grading failures from 70% to 15%. Use pairwise comparison, randomise position, and calibrate against human judgements.

Do eval frameworks integrate with normal CI test reporting?

Barely. Of the major frameworks, only promptfoo emits JUnit XML natively, via an output path ending in .junit.xml, and that landed in May 2026. None of them emit CTRF. DeepEval and LangSmith can pass through pytest’s own JUnit reporter because they run under pytest, which works but is a pytest capability rather than a feature of the tool.

How do you stop model updates from silently changing your results?

Pin to a specific model snapshot rather than a floating name. OpenAI recommends pinning production applications to dated snapshots. Anthropic notes that on current models a dateless ID is itself a fixed snapshot rather than a pointer — but warns that serving infrastructure around the model can still change, and that an infrastructure update is the most likely cause when behaviour shifts on a stable model ID.