Over the last 14 years of building Yethi Consulting and Tenjin – Testing Workbench , I have had my fair share of conversations defending our approaches and helping manage client expectations.
I have had business leaders ask for the moon, informed mainly, I assume, by reading analyst research and hopefully not by social media claims.
Yes, testing is going through a major upheaval. Self-healing, generative test design and agentic execution are real, and I want more of them in our products, not fewer. Building and deploying this technology has also taught me what works on day one and what stays stubbornly hard in year three.
What worries me is the widening gap between the narrative and the enterprises expected to live it. Many a bank, certainly in my customer base, do not own the source code of their core platforms. Their typical journeys crosses five or six applications with different technologies. Operating in these environments is very different from development shops where you have access to source code and your mandate is a single application. And nobody has written down the correct answer.
Analysts largely describe what the technology can do, and they are mostly right. What I rarely see in these reports is where it applies, for whom, under what constraints and at what cost.
This is not an argument against the future. It is an argument for asking better questions about it.
What the analysts get right
Let me be clear about what I am not saying. The direction of travel is not in dispute.
Maintenance is changing. Self-healing and agent-based repair will reduce brittle scripts, and Forrester now evaluates autonomous testing platforms as a category of their own, noting that most organisations still automate only about a quarter of their testing. But repairing a locator is not the same as knowing that the business process has changed. A healed script can keep faithfully testing yesterday’s rule.
The human role is moving from writing and running scripts towards oversight, agent design, risk prioritisation and domain validation. In regulated industries that human stays in the loop for a long time.
Environments are getting better too: device and browser clouds, service virtualisation, synthetic data. They have to, because we now test microservices, many channels, third-party integrations and AI-infused applications. Enterprises are also right to want platforms, skills and governance over perpetual custom scripting, and to judge AI claims by outcomes. Codeless is table stakes now.
But look at what the World Quality Report 2025 found across more than 2,000 executives. 89% are piloting or deploying generative AI in quality engineering, yet only 15% have taken it enterprise-wide, and the average productivity gain is 19%. Integration complexity is the second-biggest barrier, at 64%. That is the gap this article is about. The trend is right. The context is thin.
The missing context
Analyst research tells you what is possible. It says much less about who can adopt it, under which constraints and at what cost. These are the seven gaps I meet most, in order of importance.
1. Blackbox automation is a world apart from Whitebox Automation
Much of the automation narrative is written from the viewpoint of a development organisation: one that controls its source code, pipelines, test hooks and release cycles. Most banks I work with do not. They buy core banking, cards, payments and lending platforms and run them through system integrators. They test through the screen, the API and the database, on the vendor’s release calendar.
Picture a mid-tier bank on a purchased core, customised by an integrator and patched by the vendor every quarter. An agent built to thrive on a repository it can read has neither the code nor the hooks. It gets a screen and a schema nobody explained to it. That is my experience rather than a statistic, but it is the experience of most of my customers.
2. A bank’s value is in the journey, not the application
Automating one application’s next release is a different problem from keeping a journey intact while every system on it changes on its own schedule. A retail loan crosses onboarding, KYC, a credit engine, core banking, AML screening, payments and customer notifications. Each has a different owner, technology and release cycle. The failure that costs money is the perfectly good release in system three that silently breaks the data contract with system five. A demo of an agent working in one application is not evidence of enterprise-wide autonomy.
3. You cannot buy process maturity
Autonomous execution is the outcome of technology, process, governance and discipline working together. An AI platform cannot make up for weak environment management or missing promotion gates. Before the demo, ask four things from their own organizations. How mature are my current testing processes? Do we undertake regression testing and is that a constraint we want to solve? What data constraints slow down regression and how will we solve for them with automation? What environmental constraints do we deal with in test environments that will hinder automation?
4. Executing a transaction is not validating it
Most autonomous testing demos show an agent navigating an application and posting transactions. They rarely show how it finds the right customer/ account, keeps the test repeatable in a shared environment, or, above all, where the expected result comes from.
Was the loan charge right, including every applicable fee? Were the correct rates, value dates and product rules applied in a currency conversion? Will a bond settle correctly if the settlement date falls on a holiday? The screen says “success” in all three cases. Only business rules say whether it is correct.
This is the old test-oracle problem, which researchers have described as a significant bottleneck to greater test automation. Generative AI can propose expected results. It cannot excuse us from trusting them. And there is a quieter risk: teams that no longer see the day-to-day failures quickly lose the ability to triage and fix them, which is exactly the skill needed to judge whether an agent’s “pass” can be believed.
5. Probabilistic is not automatically cheaper
Traditional automation was justified by less repetitive effort, more coverage and lower cost over time. Some of today’s narrative assumes that regenerating a test on every execution is simply better. Every run then carries inference, context, agent steps, validation of what was generated and human review. Gartner says agentic models need 5 to 30 times more tokens per task than a standard chatbot, and that provider savings will not be fully passed on to enterprises. A nightly regression of 5,000 stable payment checks does not need to reason afresh each night. A deterministic script runs it at close to zero cost, and an auditor can follow it.
6. The same AI is not welcome everywhere
Data residency, privacy, banking secrecy and outsourcing rules differ sharply by market, and they decide whether a tool can see production-like data, call an external model or run at all. RBI’s FREE-AI report says accountability cannot be delegated to the model and treats AI delivered by a third party as outsourcing. Singapore’s MAS has published guidelines that expect institutions to test third-party AI with their own data. The WQR’s top barrier to scaling is data privacy, at 67%. Few feature comparisons grade a tool by where and how it can lawfully run.
7. A great demo is not production-grade autonomy
When an agent wrongly passes a financial transaction, who is accountable, and can the bank reconstruct why? Six weeks after a release, a regulator may ask what data the agent accessed, which tool it chose and why it proceeded. If the log cannot answer, the bank carries the finding. Production-grade autonomy needs repeatability, explainability, audit trails, human oversight that has itself been tested, and controlled access to production-like environments. A demo needs none of those. It is worth remembering that Gartner expects over 40% of agentic AI projects to be cancelled by the end of 2027, citing cost and unclear value.
A more realistic way to judge automation
I do not want to leave this as a list of complaints. Here are five questions I would put to any automation technology, mine included.
1. Applicability. Does it fit our architecture, our operating model and our regulatory environment, including the systems we do not own?
2. Reliability. Does it give repeatable, accurate and explainable results, and can the vendor show repeat-run variance and false positive rates on our scenarios?
3. Context. Does it understand our application landscape, our business processes, our dependencies and our test data, and where does it get the expected result?
4. Economics. Does it lower the total cost of quality and show measurable return at our volumes, once inference and human review are counted?
5. Governance. Can we control, audit and take accountability for what it does?
The future these questions point to is not fully autonomous, and it is not purely AI-driven. Deterministic execution carries the predictable, high-volume, repeatable work. Generative AI helps with requirements, test design, data, analysis and the healing of automation. Agentic AI orchestrates complex workflows and adapts as context changes, inside boundaries that are themselves expressed as repeatable tests. People own the business validation, the risk judgement and the accountability. None of that works without the process foundations underneath it.
Autonomy is earned, not purchased
I believe in autonomous testing, and I expect to see more of it every year. But autonomy is not a feature you switch on. It is what you get when technology, process maturity and governance mature together. Buy the best platform you can find, and fix your environments, your promotion gates and your traceability while you do. The banks that will run autonomously first will not be the ones with the boldest tool. They will be the ones that did the unglamorous work.
And the future of enterprise automation is not deterministic versus AI. It is knowing which is which. Use deterministic execution for what you already understand and run every night. Use generative and agentic AI for what is new, changing or hard to specify. Keep humans accountable for what the business means. Then measure the total cost of quality, not the number of scripts generated.
The technology we can trust, govern, integrate and justify will outlast the technology that merely impresses. I would like to hear how you are applying these five questions in your own bank or enterprise.