AI in testing: useful tool or expensive autocomplete?

I tried one of those AI testing tools last month. I gave it a login page URL and asked it to generate test cases. Here's what I got back:

1. Verify that the login page loads successfully.
2. Verify that the user can log in with valid credentials.
3. Verify that an error message is displayed for invalid credentials.
4. Verify that the password field masks the input.
5. Verify that the "Forgot Password" link is functional.

Thanks. I could have written that list in 90 seconds without AI. And I would have also included: What happens if the email has leading/trailing whitespace? What's the max length on the password field? Does the session persist across tabs? What happens if you hit login twice quickly? What about SQL injection in the email field?

The AI didn't generate test cases. It generated a table of contents. The generic, surface-level, obvious checks that any tester would think of first. The value of a good tester is in what they think of *next* — the edge cases, the abuse cases, the weird interactions between features that break in production.

The "AI-powered" illusion

Here's what most AI testing tools actually do: they take your input, send it to a large language model with a prompt like "generate test cases for this feature," and format the response nicely. Some add a database layer so you can save the results. Some let you edit the output. But the core intelligence is a general-purpose language model with no knowledge of your product, your users, your architecture, or your history of bugs.

This is expensive autocomplete. It's useful in the same way autocomplete is useful — it saves some typing. But it doesn't think. It doesn't know that your payment system has a recurring issue with timezone conversions, or that your European users hit a specific edge case with VAT calculation, or that the last three bugs in your authentication module were all related to session token refresh timing.

Without context, AI generates plausible-sounding but shallow output. "Verify the system works correctly" is not a test case. It's a platitude.

Where AI actually helps

That said, I'm not an AI skeptic. I think AI is genuinely useful in testing — just not in the way most tools market it. Here's where I've seen it work:

Reading what humans won't. A 40-page requirements document is a chore to read carefully. Humans skim. They miss details on page 23. They gloss over the "except when" clauses that describe edge cases. An AI model can process that entire document and extract every testable condition, every exception, every boundary value. Not because it understands the document better than a human, but because it doesn't get bored. It doesn't skip paragraphs. It processes every sentence with the same attention.

Finding what humans miss. When you've written test cases for a feature ten times across ten releases, you develop blind spots. You test the same scenarios because they're familiar. AI can look at a requirement with fresh eyes — or rather, no eyes and no habits — and surface conditions that a veteran tester might overlook because they've unconsciously marked them as "that never breaks."

Maintaining consistency. If you have 500 test cases written by 8 different testers over 3 years, the format, detail level, and quality vary wildly. AI can normalize test cases to a consistent format, flag vague ones that need more detail, and identify gaps where requirements exist but test cases don't.

Scaling the boring parts. Generating negative test cases (wrong data type, null values, boundary values, max length + 1) is tedious but important. AI is good at systematically generating these variations. A human writes the interesting test cases; AI fills in the combinatorial matrix.

Where AI fails

Business context. An AI model doesn't know that your biggest customer uses the bulk import feature to upload 50,000 records every Monday morning, and that this specific usage pattern is what you need to test most carefully. It doesn't know that your CEO promised a specific workflow to close a deal, and that workflow has to work perfectly by Friday. Context like this determines what matters. AI doesn't have it unless you provide it, explicitly, every time.

Judgment about what matters. You can generate 10,000 test cases for a login page if you test every combination of inputs. But most of those are a waste of time. A senior tester knows which 50 test cases actually matter based on risk, user behavior, and architectural knowledge. AI treats all possibilities as equally interesting.

Understanding failure modes. When a test fails, a human tester can often identify the root cause by recognizing patterns: "This looks like the same timezone bug we had in Q2." AI sees a failed test. It doesn't connect it to your team's history or your system's known weaknesses.

Replacing judgment with volume. The worst thing AI can do is make you feel thorough when you're not. Five hundred AI-generated test cases that cover surface-level scenarios will give you worse coverage than fifty human-written test cases that target known risk areas. Volume is not coverage. Relevance is coverage.

The honest take

AI in testing is powerful when it has context. Your requirements, your domain knowledge, your existing test suite, your defect history — feed these to an AI system and it can do genuinely useful work. It can read your BRD and extract every testable requirement. It can look at your existing test cases and find the gaps. It can generate the tedious negative and boundary test cases that humans skip because they're boring.

Without context, it's a generic test case generator. You could get the same output from a "common test scenarios" blog post written in 2019.

The question isn't whether AI is useful in testing. It is. The question is whether the specific AI tool you're evaluating has access to enough context to be useful for *your* testing. If it's just a prompt wrapper around a language model with no connection to your requirements, your existing tests, or your domain — save your money. You already know how to write "Verify login works."

The tools that will matter are the ones that integrate AI into the workflow where your context already lives: your requirements, your test management, your defect tracking. Not AI as a standalone novelty, but AI as a layer that makes your existing process smarter.

That's a harder product to build. But it's the only one worth buying.