Writing automated tests can be a challenging task due to the particularity of this kind of coding. If your plan is to use them inside a Devops software development process, which uses intensive automation, both for testing and infrastructure as a code, it’s important to make tests run fast and reliably. Unreliable tests have their own definition, as described at Google Testing Blog, “Flaky tests are tests that exhibit both a passing and a failing result with the same code”. Those types of tests can ruin a pipeline, causing false positives and leading the QA team to draw an incorrect conclusion of the test suite pass rate.
UI tests provide an approach that follows the user behavior, so you will assert the functionality of the entire software from the users perspective, providing a concrete business value, as opposed to testing if your function that generates a JSON is in the correct format, for example. This is the main point of the Google Testing Blog article, “Just Say No to More End-to-End Tests”, which discusses the practical aspects of UI testing, describing them as flaky, hard to point to the root cause and, hiding small bugs inside big bugs as most exceptions are suppressed and only custom warnings are exhibited in the UI.
To facilitate a change in this paradigm, Martin Fowler presents the Test Pyramid, a great visual metaphor directing you to think about the different layers of testing, and how many tests to distribute relatively in each layer. This pyramid was first mentioned in Mike Cohn’s book, “Succeeding with Agile” however, Martin comments that this is overly simplistic and can, therefore, be misleading.
Let’s see the modern concept of the Test Pyramid, updated by Martin:
The pyramid’s shape is useful to demonstrate two points about this approach: firstly, the right amount of tests to be implemented in each layer, so in the bottom, there are more tests compared to the top level. Secondly, it’s possible to expose the resource usage to run each layer, starting with unit tests in the bottom, that run quickly with no third party apps, compared to UI tests that are in the top and use a browser and Selenium Server (example). The distribution of the categories are completely aligned with these two metrics, let’s elaborate:
Unit tests are easy to write and run at very low cost. When writing them, time should be taken to assert functionality and narrow down cases of whether: the function works with the anticipated/expected input, and how it deals with unexpected/invalid inputs. At the component test layer, it’s time to forget small functionalities covered on unit tests and assert business rules in the software.
For integration tests, the coverage must be further than the expected cases in the components communication and be capable of understanding a components error messages and when to return exceptions that can be treated by the application. Don’t forget to stub or mock what is not being tested in this scope. If you are testing database communication, you don’t need to reach the real third-party APIs, for example.
The End-to-End must test the main user flows into the application. These tests must be fewer in quantity than previous tests since they already cover a large part of the application; it’s not necessary to duplicate tests.
As a closing note:
Testing applications using only UI tests have huge long term costs and cause uncertainty due to their flakiness. The Test Pyramid is a good practice for your Devops Software Development process because it costs less, and is fast and reliable.