Why and how to self-assess test automation maturity

As the name suggests, the paper, which was published in April 2019 in Evaluation and Assessment in Software Engineering (EASE ‘19), analyzes gaps in the available test automation maturity self-assessment frameworks, and proposed a better way forward.

I thought it might be worthwhile to highlight the key findings of the paper and the fact that you can assess your test automation maturity with the survey. In essence, this blog is a checklist of what you should look for when doing a test automation maturity assessment.

Let me labour the point: the study is still on-going! You can still self-assess your test automation maturity and add to the existing body of knowledge about this field while you're at it. Please fill out the EASE survey to self-assess your company’s test automation maturity.

Why is test automation important?

I’m biased – but test automation is the backbone of DevOps because it enables quality at speed. If you do it right, you set up test automation once, it isn’t that difficult to maintain, and testing thereafter takes a lot less effort.

Test automation maturity self-assessment: Why do it?

That test automation is important to DevOps performance is not exactly breaking news.

What is slightly new to the field, however, is the idea that knowing the maturity of your test automation capacity is well worth your while.

We’re not talking DevOps maturity; we’re talking test automation maturity in particular.

The point of self-assessing is simple: it drives down test automation costs and reduces the risk of you making errors that will bite you in the foot later down the line.

What factors determine test automation maturity?

A proper test automation self-assessment instrument covers a lot of previous blind spots for companies implementing test automation. Here’s an overview of the areas you should be looking at. It’s not comprehensive: for that, you’ll have to go straight to the source.

Test automation strategy

Unfortunately, 'we need test automation' is not quite enough on the strategy front. A solid test automation strategy includes the following. 

● Goals are set for test automation

● Business case is set for test automation

● Risks are analysed

● Scope is decided: to what degree will automation occur?

● Overlap between test automation and manual testing has been considered

● Gaps and overlap between types of testing and abstraction levels has been considered

● Resources for test automation are identified

● Feedback on strategy is collected from stakeholders

Resources

Who, what, and is it enough? Look out for whether: 

● There are enough people to conduct test automation. This can include experienced testers, managers, consultants, automation experts, etc.

● There are testing tools to support testing activities

● The test environment is set up with the required software, hardware, and test data

Test organization

Who's responsible and does everyone have what they need? 

● Roles and responsibilities are clearly defined

● Communication and problem-solving mechanisms are provided

● Test team has enough skills to perform best automation, including: coding, domain knowledge, and test design techniques

● Insight into cost/profit ratio of test automation

● Test organization has the capability to maintain its chosen test tools

Knowledge transformer

Is your organization growing in its test automation expertise? 

● Good test automation expertise, test tools, and practices are shared

● Training and learning curves are given due time

Test tool selection

This is important. 

● Right tools are selected (constraints, data-driven testing, test code language, test environment)

● Use of existing test tools is maximised

Test environment

This is also important. 

● Requirements are understood

● Test environment is tested

● Test environment support is provided

Test requirements

Here too, a process is necessary. 

● Test automation requirements processes defined

Test design

No re-inventing the wheel here. 

● Specific test design techniques are used to create test cases using different types of testing

● Design patterns are reused

● Static and dynamic measurements are performed on test code

Test execution

It's not just a matter of pressing 'go'. 

● Test cases are prioritised for the automation in order to meet the test execution schedule

● Pre-processing tasks are finished up before execution

● Post-processing tasks automatically performed after the execution

Verdicts ie. results

Test data needs to be harnessed to the max. 

● Stable test oracles determine the pass or fail of a test

● Status and progress of testing can be monitored

● Different test results are integrated to create a big picture of test data

● Useful test results are communicated to stakeholders

● The dashboard is adapted to each stakeholder

Test automation process

Development cycle is a keyword here. 

● The process is stable and controllable

● Test automation runs in parallel to the relevant development cycle

● Test automation process supports other processes

The software being tested

Test automation is not a matter of course. 

● Software itself is mature enough to conduct test automation – is it testable?

● Software is performant enough ie. the execution speed is enough to conduct test automation.

Test automation metrics

More on data. 

● The right metrics are used to measure test automation

● Important attributes are identified: effectiveness, thoroughness, reliability, maintainability

● Improvement areas identified through measurements

● Test team members get feedback about their performance

The hallmarks of great test automation

I can’t say this piece is getting any shorter, but before I sign off, here are some generic points about what you should be aiming for with your test automation program, alongside the specifics mentioned above. As with all the content in this blog, this too comes from the academic paper in question.

Portability: How easy is it to run automated tests in a new environment with different hardware?

Maintainability: Is your test automation easy to maintain, especially regarding ease of managing and updating the test environment?

Reliability: Are the tests reliable and accurate? Are the tests resistant to trivial change?

Usability: Are the tests easy to understand by a wide range of people including managers and leadership?

Efficiency: Is the right amount of cost and human effort being used?

Functionality: Does your testing improve ROI ie. product quality?

Now go take that test automation maturity self-assessment

Now that you made it to the end of this piece, you impressive human you, please do take a look at the test automation maturity self-assessment survey. It’ll shed some light on your test automation practices at the very least.