How to Test AI Features: QA for LLM-Powered Products in 2026

The Quick Answer

This article is about testing a product that has AI inside it - a chat assistant, a summarisation feature, a document extractor, an agent that takes actions on a user's behalf. It is not about using AI to help you test; that is a separate subject, covered in our guides to AI in software testing and AI test case generation.

The core problem: Traditional test design assumes the same input produces the same output. Language models do not work that way. Run the same prompt twice and you may get two different, both acceptable, answers - so an assertion that compares output to an expected string fails on correct behaviour and passes on nothing useful.

The shift required: You stop asserting exact equality and start asserting properties, on a dataset, measured as a rate. A single test case becomes a statistical question: across two hundred representative inputs, how often does the feature produce an acceptable answer, and has that rate dropped since the last version?

This article covers what to assert when output varies, how to build an evaluation set that works as a regression suite, how to use a model to grade another model's output without fooling yourself, how to test for hallucination and adversarial input, and what to do when the model or prompt changes underneath you.

Why LLM Features Break Traditional Test Design

Four properties of these features invalidate assumptions that ordinary functional tests rely on.

• Non-determinism. The same input can produce different outputs across runs. Even with temperature set to zero, provider-side changes, batching, and hardware differences mean identical output is not guaranteed.
• No single correct answer. For a summarisation or drafting feature, many outputs are equally valid. Correctness is a range, not a value.
• The dependency changes without you. A hosted model can be updated, deprecated, or silently adjusted. Your code did not change, your tests did not change, and your feature now behaves differently.
• Failures are plausible. A conventional bug produces an error, an empty screen, or an obviously wrong number. An LLM failure produces a fluent, confident, well-formatted answer that happens to be wrong - which is far harder to notice and far more damaging when it reaches a user.

The practical consequence is that a handful of hand-written test cases cannot tell you whether the feature works. You need a dataset large enough to produce a rate, and you need to compare rates between versions rather than pass or fail individual runs.

What to Assert When the Output Varies

Exact-match assertions are usually wrong here, but that does not mean nothing is assertable. Most LLM features have a layer of strictly checkable properties around a layer of fuzzy ones.

The most valuable early win: Assert the structural and constraint layer strictly and in CI. A large share of production incidents in LLM features are not subtle quality regressions - they are malformed JSON breaking a downstream parser, an output that blew past a length limit, or a response in the wrong language. Those are ordinary deterministic bugs wearing an AI costume, and they are cheap to catch.

Building an Evaluation Set

The evaluation set - the golden dataset - is the central asset of AI feature testing. It plays the role a regression suite plays elsewhere, and its quality determines whether you can tell an improvement from a regression.

What Goes In It

Each entry needs an input, enough context to reproduce the call, and a definition of what acceptable looks like - a reference answer, an expected label, a set of required elements, or a rubric. Aim for a spread across four categories: representative real inputs, drawn from actual usage rather than invented; edge cases, including empty, very long, multilingual, and malformed inputs; known failure cases, every production issue converted into a permanent entry; and out-of-scope inputs, where the correct behaviour is a graceful refusal or a clarifying question.

How Big

Big enough that the measurement is stable between runs. In practice a few hundred entries is often sufficient for a focused feature, and the test of adequacy is empirical: run the same version twice, and if the score moves more than a point or two, the set is too small or too noisy to detect real regressions.

Keep It Honest

Two disciplines matter. Hold a portion back - if every entry is used during prompt development, the prompt gets tuned to the dataset and the score stops predicting production behaviour. And grow the set from production continuously; every user-reported failure becomes an entry, which over time turns the eval set into an accumulated record of everything the feature has ever got wrong.

Practically, an eval set is a test suite with inputs, expected properties, and per-entry pass criteria - which is what test case management already models. Keeping the entries there rather than in a loose spreadsheet gives you version history, ownership, and a run-by-run record of which specific inputs regressed.

Using a Model as Judge

For the fuzzy assertions - semantic equivalence, tone, helpfulness - the practical grading option at scale is another model. It works, with limits you need to know.

The rule that makes it usable: A judge is only trustworthy once you have measured it against human agreement. Have people grade a sample of a hundred or so outputs, run the judge on the same sample, and compare. If the judge agrees with your reviewers most of the time, its scores are a usable proxy. If it does not, fix the rubric before trusting a single number it produces.

Known biases worth designing around: judges favour longer and more confidently-worded answers regardless of accuracy; they favour outputs from the same model family; they are sensitive to option order in comparisons; and they score inconsistently when asked for a number on a wide scale. Mitigations are straightforward - use a narrow scale with explicitly defined levels rather than "rate 1 to 10", ask for a specific property per call instead of overall quality, randomise ordering in pairwise comparisons, and prefer a different model family for judging than the one under test.

Do not use a judge where a deterministic check exists. If you can validate a schema, compare against a ground-truth label, or check for a required string, do that instead - it is cheaper, faster, and not subject to any of the above.

Testing for Hallucination and Grounding

Fabricated content is the failure mode that most damages user trust, and it is the one least likely to be caught by casual testing, because fabricated output is usually well-written.

For features that answer from supplied sources - retrieval-based assistants, document Q&A, summarisation - grounding is directly testable. Build a set where you control the source material exactly, then check three things: every claim in the output is supported by the source; nothing plausible-but-absent has been added; and when the source does not contain the answer, the feature says so rather than filling the gap. That third case deserves its own subset of the eval set, because "I don't know" is correct behaviour that models are reluctant to produce and teams rarely test for.

Specific traps worth including as permanent entries: questions whose premise is false, so the correct response is to challenge it rather than answer; requests for a detail the source genuinely omits; requests for citations, checked for whether the cited passage actually exists and says what is claimed; and inputs about entities that do not exist, where the failure mode is a confident invented description.

Adversarial Input and Prompt Injection

Any feature that puts untrusted text in front of a model needs adversarial testing, and this is testing your own product's resilience rather than an offensive exercise.

The category to cover most carefully is instruction injection: content that arrives as data - a user message, an uploaded document, a web page the feature retrieves, a support ticket - but is written to be read as instructions. The risk is highest in features that pull in third-party content, because the attacker does not need access to your application to place text where your model will read it.

Practical cases to hold in the eval set:

• Instructions embedded in supplied data - a document or retrieved page telling the model to disregard its task or to reveal its configuration
• Attempts to extract the system prompt or the contents of other users' context
• Scope escape - persuading a support assistant to give legal, medical, or financial advice it should decline
• Requests for actions beyond the feature's remit, especially where the feature can call tools or take actions on a user's behalf
• Encoding and obfuscation - the same attempts expressed in another language, in unusual formatting, or split across turns
• Confused-deputy cases - input that tries to make the feature act with its own permissions on behalf of a user who lacks them

The architectural point: Prompt-level defences reduce success rates; they do not eliminate them. Anything that must not happen needs enforcement outside the model - authorisation checks on every tool call, allow-lists for actions, output filtering, and human confirmation before irreversible operations. Test both layers, and treat a successful injection that was blocked by the authorisation layer differently from one that reached the action.

Regression Testing When the Model or Prompt Changes

The unusual property of these features is that behaviour can change when you change nothing. Three triggers require a full evaluation run.

A prompt edit. Prompts are code and deserve the same treatment: version control, review, and an eval run before merge. A wording change intended to fix one behaviour routinely degrades another, and without a dataset you will not see the trade.

A model or version change. Never swap models on the assumption that a newer or larger one is better for your task. Run both against the same eval set and compare per-category, not just overall - upgrades commonly improve average quality while regressing a specific behaviour such as refusing out-of-scope requests or honouring a format constraint.

Provider-side drift. Because a hosted model can change under a stable version label, run the eval set on a schedule against unchanged code. A score drop on unchanged code is the signal that something moved upstream - and it is the only way you will find out before users do.

Compare per-entry, not only in aggregate. An overall score that holds steady while twenty specific inputs flipped from pass to fail is a regression the aggregate hides. Tracking results per entry across runs is exactly what a test run history and reporting give you - each eval execution becomes a run against the same case set, and a per-case success rate across those executions tells you which specific inputs are failing or unstable rather than only how the overall score moved. Pin model versions explicitly in configuration, record the version alongside every result, and keep the last known-good score so a comparison is always available.

Where Manual QA Still Matters

Automated evaluation scales but is blind in specific ways, and the gaps are exactly where reputational damage comes from.

Exploratory testing of an AI feature is unusually productive, because a human probing a conversational interface finds failure modes no scripted dataset anticipated - the tangent that derails the assistant, the multi-turn sequence where it loses the thread, the phrasing that triggers an inappropriate response. Every finding becomes a permanent eval entry, so the exploratory work compounds instead of being repeated.

Three areas need human judgement rather than a judge model: tone and appropriateness in sensitive contexts, where a technically correct answer can still be badly wrong; the end-to-end experience, including latency, streaming behaviour, error states, and what the interface does when the model is slow or unavailable; and the fallback path, which is often the least-tested part of an AI feature and the one users hit on a bad day. Test what happens when the model times out, returns malformed output, or refuses - the feature should degrade into something usable rather than a spinner or a stack trace.

Conclusion

Testing a feature with a language model inside it is not conventional functional testing with more patience. The assumption that one input yields one output is gone, and with it the exact-match assertion that most test design rests on.

What replaces it is workable and largely familiar. Assert the deterministic layer strictly - schema, constraints, extraction against ground truth - because that is where a surprising share of real incidents live. Build an evaluation set from real usage, edge cases, past failures, and out-of-scope inputs, and treat it as your regression suite. Measure rates rather than individual passes, and compare per entry between versions so a masked regression cannot hide behind a stable average. Calibrate any model judge against human agreement before believing it. Test grounding, refusal, and adversarial input deliberately, and enforce the things that must not happen outside the model rather than inside the prompt. Re-run everything when the prompt changes, when the model changes, and on a schedule for when neither did.

All of that depends on treating eval entries as managed test cases with history. QA Sphere holds them as versioned test cases, executes them as repeatable test runs, and scores them in reporting per test case across those runs, so the specific inputs that started failing under a new model version stand out instead of being averaged away. See pricing or book a demo.