How to Test AI-Generated Code: A QA Playbook in 2026
In this article
The Quick Answer
AI coding assistants and agents now write a large share of production code. In Sonar's 2026 State of Code survey, developers estimated that AI accounts for 42% of the code they commit. The same survey found that 96% of developers do not fully trust that code to be functionally correct, and only 48% always check it before committing.
That gap - more code, written faster, checked less - lands on QA.
What changes: AI-generated code is not worse in every way. It is usually tidy, compiles, and handles the happy path well. What changes is where it fails: plausible-looking logic errors, missing edge cases, invented assumptions about how your system works, and tests that confirm the code rather than the requirement.
What it means for testing: those failures look different from human mistakes, so they need a deliberately different approach - aimed at intent, edges and independence rather than at more of the same checks.
This article covers why agent-written changes need their own treatment, the defect patterns to look for, the review rules that matter most, a checklist for agent-written pull requests, the metrics worth tracking, and how to give your agents your real test cases so they stop guessing.
If you are testing a product that has an LLM inside it, that is a different problem - see How to Test AI Features. This article is about testing the code that AI writes for you.
Why AI-Generated Code Needs a Different Kind of Testing
Three recent research findings explain the problem better than any opinion.
Volume went up and stability went down. The 2025 DORA report, based on responses from nearly 5,000 technology professionals, found that 90% of respondents use AI at work and that AI adoption now improves delivery throughput - but still has a negative relationship with delivery stability. Teams ship more, and more of it breaks.
Defects per change went up. CodeRabbit's analysis of 470 open-source pull requests found that AI-co-authored PRs contained about 1.7 times as many issues as human-only PRs, with logic and correctness problems among the largest gaps. CodeRabbit sells code review, so treat the exact ratio with care, but the direction matches what most QA leads report.
Security improves slowly. Veracode's October 2026 update of its GenAI Code Security research found that the tested models still chose an insecure implementation 38% of the time, down from 44% earlier in the year. Functionality, in the same research, is close to perfect. The code works. It is just not always safe.
Put together: AI shifts effort from writing code to verifying it. If verification capacity stays where it was, defects that used to be caught in review now reach your test environment - or your users.
Where AI Code Breaks: The Defect Patterns to Look For
Human developers make tired mistakes: typos, forgotten cases, copy-paste errors. AI makes confident mistakes. Knowing the difference tells you where to point your testing.
| Pattern | What it looks like | How to catch it |
|---|---|---|
| Plausible but wrong logic | An off-by-one in a date range, a discount applied before tax instead of after, a sort that is stable in tests and not in production | Test against the business rule, not the implementation; use boundary values |
| Missing edge cases | Empty lists, nulls, time zones, currency rounding, concurrent edits, very long input | Equivalence partitioning and boundary analysis; exploratory sessions aimed at edges |
| Invented assumptions | The agent assumes an API returns sorted data, that a field is always present, or that a config value exists | Contract tests and integration tests against real dependencies |
| Silent error handling | Broad try/catch blocks, default values that hide failures, retries that mask a broken call | Negative tests, fault injection, checking logs for swallowed errors |
| Duplicated logic | A second, slightly different implementation of something your codebase already does | Review against existing modules; regression tests on both paths |
| Self-confirming tests | Generated tests that assert whatever the code currently returns | Review the test intent; write acceptance tests from the requirement first |
| Insecure defaults | String-built SQL, missing input validation, permissive CORS, secrets in code | SAST in CI and security test cases for every endpoint the change touches |
The last two deserve emphasis. A green test suite means much less when the same model wrote both the code and the tests, and insecure defaults rarely show up in functional testing at all.
Rule One: Never Let the Same Agent Write the Code and Its Only Tests
If an agent writes a function and then writes the tests for it, the tests encode the agent's understanding of the problem. When that understanding is wrong, the tests are wrong in exactly the same way, and they pass.
This is the single most important rule for agent-written changes, and it is easy to break because generated tests look thorough. Forty assertions with good names feel like coverage.
What works instead:
- Write acceptance criteria and test cases before the code. The requirement, expressed as concrete test cases, is the independent check the generated code has to pass. This is shift-left testing with a new reason behind it.
- Treat generated unit tests as the developer's tests, not QA's. Useful for refactoring safety, not proof of correctness.
- Keep a human-owned layer. Acceptance, integration and end-to-end tests that trace to requirements should be written or reviewed by someone who did not write the code.
- Read the assertions, not the test names. A test called
rejects_expired_couponthat asserts the response is not null tests nothing.
Review the Intent Before the Diff
Code review for AI-generated changes tends to fail in a predictable way: the reviewer reads a clean, well-formatted diff, finds nothing wrong with the code as written, and approves it. The problem is that the code solves a slightly different problem from the one in the ticket.
Reverse the order:
- Start from the ticket. What should change for the user? What must not change?
- Read the prompt or task description the agent was given, if your team records it. Ambiguity in the prompt becomes a wrong assumption in the code.
- Then read the diff, looking for where the implementation's assumptions differ from the requirement.
- Check what the diff does not touch. Agents often update the obvious call site and miss the second one.
This is also where a QA engineer adds the most value in an agent-heavy team. Developers review syntax and structure well. QA brings the question "what happens when..." - which is precisely the question AI-generated code has not asked itself.
A Testing Checklist for Agent-Written Pull Requests
Use this as a PR template section or a definition-of-done item for any change where an assistant or agent wrote a meaningful part of the code.
Before merging:
- Traceable requirement - the change links to a ticket with acceptance criteria written before implementation
- Independent acceptance tests - they trace to those criteria and were not generated from the code
- Boundaries covered - empty, null and maximum inputs, plus boundary values
- Failure paths tested - every external call has a test for timeout, error response and malformed data
- No hidden failures - no new broad exception handling that swallows errors
- No duplicated logic - the change reuses existing modules where they exist
- Scans reviewed, not just passed - SAST and dependency scanning results were read by a person
- Dependencies checked - new packages were added deliberately, exist, and are maintained
- Security cases - changes to auth, input handling or data access have dedicated test cases
Before release:
- Regression scope updated - affected areas are included in the regression run; see Regression Testing
- Exploratory coverage - at least one exploratory session covered the changed area
- Safe rollout - high-risk changes ship behind a feature flag; see Testing in Production
Not every item applies to every PR. The point is to make the verification step explicit, because with AI the writing step no longer forces anyone to think it through.
Point Exploratory Testing at the Edges
Scripted tests confirm what someone thought to specify. AI-generated code fails mostly in what nobody specified, which makes exploratory testing more valuable, not less.
Charters that work well for agent-written features:
- "Explore the new export with unusual data - empty accounts, very large accounts, non-Latin names - to discover handling gaps."
- "Explore the edited checkout flow across time zones and currencies to discover rounding and date errors."
- "Explore the new API endpoint with malformed, missing and oversized fields to discover silent failures."
- "Explore what the change touched indirectly - other screens that use the same service - to discover regressions."
Thirty focused minutes on edges will usually find more in agent-written code than an hour of re-running the happy path the agent already tested.
Track AI-Authored Changes Separately
You cannot manage what you cannot see. If your team does not know which changes were substantially AI-written, it cannot tell whether AI is improving or eroding quality.
Light-touch ways to get the signal:
- Label PRs where an agent wrote most of the code. Many teams use a commit trailer or a PR label.
- Tag escaped defects by origin during root cause analysis. Was the faulty code AI-written? Did the tests that should have caught it come from the same agent?
- Compare a few metrics by origin over a quarter: escaped defects, reopen rate, time to fix and change failure rate.
- Watch for test inflation. Rising test counts with flat or falling defect detection usually means self-confirming tests. More tests do not automatically mean better quality - see Testing Myths #1.
The aim is not to blame the tool. It is to find out where AI helps your team and where it needs a tighter verification loop.
Give Your Agents Your Test Cases
Much of what goes wrong with AI-generated code comes from missing context. The agent does not know your business rules, your edge cases or what "done" means for your product, so it fills the gaps with plausible guesses.
Your test case library already holds that context. Giving agents access to it changes the workflow in a useful way:
- Before coding, the agent can read the existing test cases for the area it is changing and see which behaviors are already specified.
- While coding, a developer can reference a specific test case when asking the agent to implement or automate it.
- After coding, the agent can draft new test cases for the change - which a human then reviews, so the independent check stays independent.
- After a run, the agent can summarize what failed and which areas are affected, without anyone copying results into a chat window.
QA Sphere supports this directly. The QA Sphere MCP server connects Claude, Claude Code, Cursor, Codex, Gemini CLI and VS Code to your workspace, with read-only or standard access and OAuth where the client supports it. It is included on every plan. For coding agents working in the terminal, the QA Sphere CLI agent skill lets an agent list projects, create test cases, open and close runs, upload results and pull reports.
One caution: give agents read-only access first. Let them read your requirements and test cases freely, and keep a human approval step for anything that changes them.
Common Mistakes
- Treating a green build as proof. If the agent wrote the tests, green means the code agrees with itself.
- Reviewing the diff instead of the intent. Clean code that solves the wrong problem passes review easily.
- Skipping negative tests. AI is strong on the happy path and weak on failure handling - test the failures.
- Accepting new dependencies without checking them. Agents add packages freely, and some do not exist or are not maintained.
- Letting test counts stand in for coverage. Hundreds of generated tests can cover very little behavior.
- No record of what AI wrote. Without labels, you cannot learn whether AI is helping or hurting quality.
- Cutting QA because developers are faster. Faster writing means more to verify, not less.
- Banning AI instead of adapting. Teams that forbid the tools usually just lose visibility into how they are used.
Conclusion
AI has made writing code cheap. It has not made knowing whether the code is right any cheaper - if anything, it has made that harder, because generated code looks finished long before it is correct.
The response is not more testing everywhere. It is testing aimed where AI code actually fails: requirements written as test cases before the code, an independent layer of tests the agent did not write, review that starts from intent, exploratory sessions pointed at the edges, and a record of which changes were AI-written so you can see what the tools are doing to your quality over time.
The teams getting the most from AI coding are the ones that treated verification as the new bottleneck and invested there. Their test cases became the source of truth both humans and agents work from.
QA Sphere gives you that source of truth: a versioned test case library, test runs that show exactly what was verified for each release, reporting that tracks quality over time, and an MCP server so your AI assistants work from your real test cases instead of guesses. See pricing or book a demo.
Written by
QA Sphere TeamThe QA Sphere team shares insights on software testing, quality assurance best practices, and test management strategies drawn from years of industry experience.



