Quality assurance used to be the discipline that could say, with confidence, “yes, it works.” AI has destabilised both halves of that sentence. The checking itself is increasingly done by machines, and “works” has become a moving target now that so much software contains probabilistic features with no single correct answer.
Having led the engineering teams living through both shifts over the past few years, I’d summarise it this way: AI testing is two revolutions wearing one name, and the second one is where teams actually get hurt.
What AI testing means
AI testing covers two distinct things. First, using artificial intelligence in software testing: generating test cases, maintaining automation, prioritising what to test and spotting patterns in results. Second, testing software that itself contains AI: evaluating features whose outputs vary, drift and occasionally fail in ways traditional QA was never designed to catch.
Both are now mainstream concerns. BrowserStack’s 2026 survey of over 250 engineering and QA leaders found that 61% of organisations already use AI across most of their testing workflows. The World Quality Report 2025-26 describes generative AI in quality engineering as having moved from promising pilot to mainstream expectation, with a consistent finding about what works: AI provides the breadth, covering many scenarios at speed, while experienced humans provide the depth, judging correctness, weighting risk and validating edge cases.
Direction one: AI in software testing
The practical gains here are real and, in our experience, front-loaded. AI generates test cases directly from requirements, turning work that took days of scripting into a curation task. Self-healing automation adapts when an interface changes, so a moved button no longer breaks a hundred tests overnight. Risk-based prioritisation directs attention to the modules most likely to fail, based on commit history and production incidents, instead of spreading effort evenly across the codebase.
None of this removes people from testing. It changes what the people do. Test engineers spend less time authoring individual cases and more time designing strategy, curating AI-generated suites and deciding what “good” means for a given product. The teams that struggle are those that let AI author and approve its own tests with nobody accountable in between.
There’s an infrastructure side to this too. Modern products are cloud-native and multi-device, which makes cloud application testing the default rather than a speciality. Testing a consumer app properly means running it against real devices, operating system versions, network conditions and regional configurations, and no company maintains that matrix in a cupboard any more. Cloud testing environments spin up those combinations on demand, and they’re also what makes AI-scale test volume affordable: machine-generated tests are only useful if you have machine-scale infrastructure to run them on.
Direction two: testing software that contains AI
This is the harder revolution, and the one I push teams to take seriously earliest. A deterministic feature can be verified: given this input, assert that output, ship. A probabilistic feature can only be evaluated: sample its behaviour across many inputs, score the outputs against quality dimensions such as accuracy, safety, tone and consistency, and keep doing so after release, because models drift and real usage always finds inputs your evaluation set didn’t.
| Deterministic features | AI-powered features | |
| What “passing” means | Output matches the expected result | Behaviour scores acceptably across quality dimensions |
| When testing happens | Before release | Before release and continuously in production |
| Who decides correctness | The specification | Calibrated human judgment, encoded into evaluation criteria |
| How failure appears | A bug report | Silent degradation, drift or a confidently wrong answer |
The cost of skipping this discipline is now measurable. Applause’s State of Digital Quality in AI 2026 found that while 54.5% of organisations have released AI features, 44.1% deactivated live AI features in the past year because operational costs outweighed user value. That is an extraordinary rate of retreat, and most of it reflects problems that proper evaluation would have surfaced before launch: outputs that couldn’t be trusted, quality that couldn’t be monitored, or costs that were never modelled against real usage.
Human oversight is the constant across everything above. In AI-heavy products, people stop being the executors of tests and become the accountability layer: defining acceptable behaviour, reviewing what the machines flag, and owning the decision to ship.
What the quality bar looks like at consumer scale
Abstract principles matter less than what quality produces, so here is a concrete benchmark from our own work. When Maxeda DIY Group needed their Praxis app rebuilt, our team rebuilt it from the ground up, natively for both platforms, with a new CMS underneath. The app now sits on more than 800,000 devices, holds a 4.6-star App Store rating, and was named App of the Year 2024.
Numbers like that are earned in the unglamorous middle of a project, through quality treated as a continuous engineering practice rather than a gate before release. That’s precisely the discipline this article is about. AI-assisted testing accelerates it enormously, and AI-powered features raise its stakes, but the underlying habit is the same one that has always separated software people trust from software they delete.
Summary
AI testing means two things at once: AI doing the testing, and AI features demanding a new kind of testing. The first delivers speed, coverage and self-maintaining automation, with humans moving into strategy and validation roles. The second replaces pass/fail with continuous evaluation, because probabilistic software can degrade silently and confidently. Cloud testing infrastructure underpins both. And the organisations retreating from AI features today are mostly paying for evaluation work they skipped earlier.
If you’re building or rebuilding a product and want quality assurance designed for how software actually behaves in 2026, probabilistic parts included, our engineering team would be glad to show you how we approach it. Bring your hardest feature. Ideally the one nobody is quite sure how to test.
