Software Testing Automation has gained significant acceptance in enterprises in recent years. Many organizations struggle to identify the value drivers in a test-automation purchase and might take decisions based on partial data. 

This article is an attempt to help people understand why automation would make sense, how to understand the various platforms out there, how to balance the economics and present an ROI to ensure proper communication of the expectations.

Let us start by understanding the need for testing automation –

Software is consistent – Any software will consistently perform actions as per the instructions. While the instruction set might be suspect, the software would be consistent irrespective of the day of the week, the temperature outside or any of the factors that impact humans.

Software is exact – Logical structures have no room for common sense.  So, the onus on the developer to understand intent and implications is high.  Given that majority of enterprise software, today consists of a myriad set of options, the chances that a particular combination of options (an option set) might work contrary to intentions is high. 

The software will evolve – Any software that you use is amorphous and would continue to evolve as long as there are customers and/or developers that wish to tinker. Change is typically due to innovation, competition, regulation, or adaptation. 

Software is fragile – New features and options must be delicately introduced, considering the existing options and their usage.  It is very much possible to unintentionally “break” an “option-set” while bringing in some new capabilities. 

Software Testing – Considering these attributes, it is imperative that software is tested – when it is designed, built, deployed, or modified. The discipline of software testing has been evolving in parallel to software development, albeit in the shadows. Considering the repetitive nature of software testing, a lot of research and effort has been spent on reducing the “drudgery” of the repetitive tasks by adopting automation in the practice of testing.  Automated testing is a large part of the public debate and we have explored why exploratory and automated testing needs to coexist in our article here.    

In this article, we would attempt to share how test automation supports the mission of consistently deploying reliable software and how one could explore the ROI of these efforts. This is important as Test Automation is primarily a tool to improve efficiency and not a tool to improve the effectiveness of your tests.  

What Constitutes Test Automation

The practice of software testing essentially is a stimulus-response study. You provide stimulus to the software (“Application under-test” or AUT) and validate if its response meets a predetermined criterion. 

The profession of software testing is to

1. identify various stimulus-response conditions (test-cases) that could be encountered during the use of the software
2. plan and execute a sequence of such test-cases
3. capture and report software behavior,

Test Automation is an aid to a software tester, where the execution and reporting of a certain subset of cases are delegated to the automation software. 

 Automation allows teams to focus on “net-new”

As software gets enriched, the testing burden shifts from testing “new” feature to testing the “existing” features.  In the graph, we have shown 5 sprints where each bar represents newsprint.  The “new” functionality in each sprint is shown in “blue” and the “existing” features from previous sprints are in “grey”. You would notice that while the volume of new features is relatively steady, the volume of “existing” features has grown with every release.    

While leaders are happy to spend time/effort on testing the blue portions, the repetitive testing of the grey portions is a shoe that pinches. Re-testing of functionality that has been available for a few releases represents a unique set of challenges across all stakeholders. From a tester’s perspective, it is repetitive and therefore boring. It is very likely to only get a cursory look-over and not an in-depth review.  From a sponsor’s perspective, it represents an ever-increasing volume of work that often has a low-return element.  Therefore, we risk under-testing and letting defects slip into “deemed-stable” portions of our software.  Test-automation represents a middle path, where the code can be tested for a larger sample of usage conditions at a lower cost-time construct where only the exceptions are picked-up for further evaluation. 

What benefits can I expect? What benefits should I not expect?

The investment in the right automation tool once set up offers potential benefits like:

Coverage: You could achieve a higher test coverage / larger sampling of the software functionality, faster and at a lower execution effort, than what you would get with pure manual efforts. 

Speed:  Automated tests can run faster and round the clock.  These could reduce your overall time-to-test significantly, giving more time to development teams to fix any potential issues. 

Consistency: Test automation tools offer maximum consistency. The same test case runs multiple times without even a minor deviation.

Effort:  Test Automation can be set up to run with little or no manual intervention. 

The Caveats

Setup: Setting up tests on any automation tool is a laborious task.  It requires planning and foresight that only experience would bring. Knowledge of how data changes across time, what variables would need to be dynamically updated and how to validate expected outcomes etc. require experience and time. 

Maintenance:  Once you have set up a large library of cases to be automatically executed, it should be smooth sailing; or so we think!!  The challenge of test automation (and therefore where money is generally made or lost) is how easy it would be to maintain this comprehensive library when your underlying application changes. When the “Application-Under-Test” changes, you would have to identify all aspects of your automation that consequently need a change.  

Test Automation – Understanding Costs & Benefits

Companies are increasingly investing in test automation considering the increased frequency of changes and anticipated cost-benefit. The investment in the right automation tool is worth it, as it offers potential benefits like:

Coverage: Test automation offers high test coverage with nearly 100% results, which otherwise cannot be achieved by manual testing.

Consistency: Test automation tools offer maximum consistency. The same test case was run multiple times without deviation.

Speed: Automation is multiple times greater than the manual testing speed. 

Effort: The manual effort in executing a test automation tool is almost negligible.

Test-automation solutions are varied and bring different categories of over-bearing costs

The cost of the test automation can be categorized as:

• Acquisition cost
• Maintenance cost

Acquisition cost: There are some free test automation tools available in the market. The free versions mostly perform basic functions and are not recommended for thorough end-to-end or regression testing.

To buy an efficient test automation tool, you will require to make some investment. A licensed and open-source automation tool may start with as low as $4000, depending on the brand and functionality it offers. The total acquisition cost is often inclusive of the tool cost and construction cost. Before investing in the tool, it is recommended to do thorough research and understand the right tool that will complement your requirements perfectly.

Maintenance cost: The maintenance cost of automated tests drives expenses up. Maintenance may involve a range of factors:

• Process
• User interface
• Data
• Environment

However, maintenance of automated tests may cost you a bit, statistics reveal the fact that maintenance can be a lot cheaper affair than creating new test scripts. Furthermore, frequent maintenance will keep the system efficient and seamless. Maintenance is a good option while considering the cost model of the project, as manual testing or implementing new automation tools can be expensive and time-consuming.

Test Automation ROI

Now that we have given you an understanding of the costs and benefits of test automation, we go further deep into exploring the ROI of these efforts. Companies are constantly investing in test automation, but investment in test automation without proper calculation of ROI would be a half-baked effort, which is unacceptable for the companies.

At the beginning of this article, we have stated that test Automation is primarily a tool to improve efficiency, and the calculation of ROI determines the efficiency of your test automation process. Test automation is a one-time investment. Configuring the test automation at the initial stage of project inception can maximize the benefits in the long run. By calculating ROI, companies can decide to invest in test automation.

Calculating ROI does not necessarily mean that we would focus on the overall cost, which is required to evaluate the quality of the product. ROI calculation also includes the parameters like late releases, defects overlooked, and lack of quality.

One of the biggest challenges that the companies face is investing a large sum of money for conducting the end-to-end testing of software, which often they are unable to recoup. Once the companies decide what test sets or subsets they must automate, they will generate a high return on investment.

Due to poor decision-making, companies may spend a lot to adopt automation at the later stage of development, and if the project fails, they may incur a high loss. ROI-based automation testing resolves this issue. The Quality Assurance team decides which section must be automated based on the clients’ environment, system, and machine. They can do a cost comparison analysis of manual test runs and automation test runs and determine the types of scripts that can be automated.

Following are the significant metrics, which can be considered while calculating automated testing ROI.

• Measure product quality – Ensuring the overall quality of the product by identifying defects and mean time to detection of defects.
• Test execution speed – Comparing the test run duration percentage between automated and manual testing.
• Thorough cost and resource analysis – Analyzing cost and resources are important as the test automation process involves software and hardware costs, prioritizing defects based on their severity, operational costs, the requirement of new resources, training costs, and more.

With the metrics mentioned above, we can calculate Automated Testing ROI. The cost that companies bear on resources, tools used, setting up the testing framework and more fall under companies’ investments. The net gain that the companies receive over their investments would be the ROI for the automation testing.

Following are the parameters required for performing ROI calculation.

Automating new tests – There are always a certain amount of costs involved in automating new tests. Before initiating the automation of new tests, companies must consider the predictable costs of developing, executing, and maintaining the tests. Based on the test categorization, you would segregate the test to automate or run manually.

Automate regression testing – Adding new features or changes in software would introduce new bugs in the existing modules. It makes regression testing one of the most expensive phases of testing. It is a necessary step to calculate the cost of automating regression testing.

Managing Multiple Environments – As companies develop products, they must be convinced that their product works equally in different environments. Hence, companies set up multiple platforms, devices, and browsers to ensure the success of their products. For organizations, it is necessary to calculate the ROI while setting up test environments.

Minimizing Defect Leakage – There are instances where bugs were detected at the production phase, as they were not resolved at the development stage. It often leads to defect leakage in the production phase, making it an expensive issue for organizations. Calculating ROI is necessary for companies to control such overhaul expenditures in certain instances.

Reusable test cases – Using reusable test cases saves time and effort by avoiding task duplication. When you already have created test cases, you can reuse them in multiple scenarios. Hence this will impact while calculating ROI.

Protecting product knowledge – Companies may still gain control over their systems and software, but they need to take a calculative risk on their resources. If an engineer leaves, the company must look for a replacement and provide training for the job. It may impact the company’s long-term automation testing ROI.

Ways to improve Automated Testing ROI

Instead of automating all the tasks at hand, it is best to automate repetitive tasks. Automating the applications, which are not steady, may not show an accurate result. Moreover, it may increase your team’s effort and affect while calculating ROI.

Automation testing works best for regression testing. Hence, when new tests and integrated with the older ones and applications are tested across various platforms, automation testing can be implemented to save time, effort, and overall costs. Further, if you need to complete the test run in a shorter time, you can run parallel testing like a smoke test.

Changes are introduced frequently in software. Hence, the developers and testers are relying on exploratory testing which is widely used in agile models. The focus of exploratory testing is discovering, investigating, and learning through the process. The testers here own the freedom and responsibility to think and note down ideas before the test execution. As exploratory testing is strategic, it helps in improving automated testing ROI.

The testing process has evolved, and today shift-left testing is a popular step in the software development lifecycle. Shift-left testing allows you to detect defects in the early stage, which helps in reducing effort, time, and money. You could also use test management tools to identify redundant test cases at the early stage of the development process. Finally, keep thorough documentation of the process so that the newly hired resources can pick up where your previous resources have left.

Conclusion

The company adopts automation testing to improve its test efficiency and test coverage. Of all the possible methods, which could be applicable for calculating the automated testing ROI, the company must adopt the strategies, which work best for them and suffice their requirements.

When it is about improving the quality of your product, the right methodology would be to test your applications simultaneously while developing them. If defects are detected at the postproduction stage, the company will suffer monetary loss, which may be unacceptable for any organization. Hence, a little strategy, the right tool, and differentiation between manual and automation testing can successfully help you implement test automation ROI.