Previously I wrote an article titled The Right Balance Between Different Types Of Test. This article in intended to act as a complementary piece that will help us to make a better choice when it comes to adopting a testing strategy. In this post we’re going to first see what different types of tests are there to choose from. We also ging to discuss and help you understand what the unique characteristic is in each type of test. That in turn is going to help us to choose a type of test that best suit our project. Because different types of tests are best suited for certain testing certain aspect of the project. Also, the stage of development, our resources, the time we can spend are also determining aspect that changes what types of tests is best suited for our QA efforts.

So, in this post first I’m going to introduce what are those different types of tests. After that we’re going to see their pros and cons. That will help us to know in which type of project of environment they are best suited. After knowing more about different types of tests, now we can think about what our objectives are when writing tests. More specifically, what aspects in our testing helps us to get the most out of our tests?

What are the different types of automated tests available when choosing a testing strategy?

Unit tests

Unit tests are low level tests and close to the source of the app. They are useful in testing individual methods and functions of the classes, components, or modules used by your app. Unit tests are generally quite inexpensive to automate and can run quickly by your continuous integration process.

Integration tests

Integration tests cab help making sure the different modules or services used by your app work correctly together. For instance, it can be testing the interaction with the database or making sure that microservices work together as needed. Integration tests are more expensive to run in general as they require multiple parts of the application to be up and running.

Functional tests

Functional tests test the business requirements of an app. They only verify the end result of an action and do not check the internal implementation of the system when performing an action.
There’s an overlap between integration tests and functional tests, in that they both require multiple components to interact with each other. The real difference is that while integration test may verify that you can query the database, the functional test would expect to get a specific value from the database based on our product requirements.

End-to-end tests

End-to-end tests simulate a user’s interaction with the program in an environment that includes the entire app. It might be as easy as launching a webpage or checking in to ensure that different user flows function as required. Additional sophisticated scenarios, such as confirming email notifications and online payments, can also be examined. Although end-to-end tests are useful, they can be expensive to execute and challenging to manage when automated. In order to quickly discover breaking changes, it is generally advised to rely more on lower-level testing (unit and integration tests) and have a small number of critical end-to-end tests.

Acceptance testing

Acceptance tests are official testing that determine whether a system complies with business needs. They concentrate on reproducing user actions while executing the complete application during testing. They can, however, take a step further and assess the system’s effectiveness, rejecting improvements if particular objectives are not achieved.

Performance testing

Performance evaluations assess a system’s ability to handle a specific workload. These evaluations aid in determining an application’s dependability, speed, scalability, and responsiveness. A performance test, for instance, can look at response times while a lot of requests are being processed or see how a system responds to a lot of data. It may assess whether a program satisfies performance standards, find bottlenecks, gauge stability during high traffic, and do much more.

Smoke testing

Smoke tests are simple tests that examine an application’s fundamental operation. They are designed to be quickly carried out, and their purpose is to provide you with the confidence that the main system components are functioning as planned. Smoke tests can be helpful immediately following the creation of a new build to determine whether or not more expensive tests can be run, or immediately following a deployment to ensure that the application is functioning properly in the newly deployed environment.

Our Focus in This Article

For the moment we’ll put our focus on three different types of testing that is most familiar to most developer and are used the most. That is unit testing, integration testing and end to end testing. In the subsequent section we’ll see what the limitation and strength are of each type of test. But the same principle that allows us to analyze the effectiveness of aforementioned test can also be used to choose many other different types of tests based on the project requirements.

Strength and weaknesses of different types of tests when choosing a testing strategy

Unit tests

Unit tests Strength

• Running unit tests makes it possible to quickly identify the effects of new changes, which speeds up development and debugging
• A module’s flaws can be found early in the development process, As a result, addressing the flaws costs far less
• Increases testing efficiency and improved resource usage since testing of a module may begin without having to wait for other modules to finish, which helps us build better code
• Makes testing of a module on its own individually possible
• Allows for exhaustive testing that concentrates on specific functionality
• Serve as documentation or a specification for our code
• Provide us with immediate feedback regarding the internal logic of our code
• Assists us in testing the architectural and code standards, such as ensuring that the front-end resource translations from the back end are used and are not left orphaned
• Good for testing the logic of our code, things like pure functions, libraries and functionalities with limited dependency from outside, for example a math library or date time library

Unit tests weaknesses

• Unit testing cannot identify problems with interoperability or integration between two modules.
• Complex errors in the system involving several modules cannot be caught by it.
• It is unable to test non-functional characteristics like usability, scalability, system performance overall, etc.
• The functionality of an application or adherence to its business requirements cannot be guaranteed by unit testing.
• Unit tests can’t explain how the components interact.
• You can’t write a test you didn’t think of. In other words, there are no tests for the cases you don’t have.
• Tightly connected code cannot have unit tests written for it (not easily)
• If a programmer didn’t consider an edge case while coding, he probably won’t when writing the unit tests (designing good test cases is more important than writing them)
• If they are not properly designed, they will be exceedingly difficult to maintain (excessive testing of internal implementation as opposed to caring about the result)

Unit tests are not only about testing

We can see that unit tests are not only about testing our software. But also enforcing code quality and best practices.​ That’s why we shouldn’t only think about unit tests only in terms of testing. But rather a tool that can increase quality, readability and maintainability of our code.​

Another point to bear in mind is that the advantages of unit testing only happens if the person who wrote them care about writing good unit tests.​ But writing them still is better than not writing them.​

“Imperfect tests, run frequently, are much better than perfect tests that are never written at all”. _Fowler​

Unit tests have a lot of flaws but that does not mean the alternatives don’t have bigger problems.​

Integration tests

Integration tests Strength

• In comparison to unit tests, it does not have as many coupling restrictions.
• Use fewer resources than tests from beginning to end
• Because it is not as thorough as end-to-end testing, it is simpler to identify the source of the issue or bug.
• Although it can test the interface between several modules, it can also be used as a unit test in situations when a strongly connected system prevents unit testing.
• It begins in the early stages, perhaps before all the modules are available. Without having to wait for the system as a whole to complete, each module can be tested independently.
• May apply to situations that unit tests did not address.
• Finding errors in the linkages and interfaces between the modules is made easier.
• It works best for apps that have a complicated infrastructure and plenty of moving pieces as opposed to those that have a sophisticated logic.
• Good at testing underlying infrastructure and architectural issues, such as whether a message travels over the network as we anticipate or whether an API provides the desired result with data.
• End-to-end test scenarios perform more slowly than integration tests.

Integration tests weaknesses

• Not as easy to setup as unit tests and not as easy to run
• It costs more to perform and is slower than unit tests.
• Inability to conduct early-phase testing
• To make sure the integrated system functions properly, testing should be done thoroughly taking the environment into account in addition to the integration linkages.
• The complexity of managing integration testing arises from a number of elements, including the database, platform, environment, etc.
• Integrating two separate systems created by two different organizations is difficult because it is unclear how modifications made to one system will affect the other. Therefore, it is important to have a solid understanding of the modules that are being integrated.

Specific types of integration tests (such as big bang, top down, bottom up) can have their own advantages and disadvantages, but we look at integration tests pros and cons from general point of view.

End-to-end tests

End-to-end tests Strengths

• Tests conducted with users in mind
• If the internal implementation changes, we won’t need to adjust our tests as much (low test churn)
• High level of confidence, meaning we can believe the feature works if an end-to-end test succeeds, and the case is well built.
• Lowers the quantity of production faults discovered

End-to-end tests Weaknesses

• Slow execution and high infrastructure need for a development or staging environment
• Flaky, brittle, and easily broken
• Finding the root cause of a failing end-to-end test is painful and can take a lot of time. They are also more difficult to debug because they can’t identify the reason for a failure. For instance, if the login process fails, you won’t know whether the frontend or the backend is at fault or even which specific section of the code is at fault.
• Failures can occur that are not directly related to our code, such as when we use an API or component from a third party.
• Fail due to insufficient resources on the server it was running on
• Developers must wait longer to determine whether a fix was successful.
• Because the developer feedback loop is delayed, it is harder to repair errors and takes longer to give value to customers.

Creating one test pyramid per project for our testing strategy

We could come up with a generic way of allocating our resources to different types of testing, but that wouldn’t be something realistic. Because each project will have a different type of complexity and each type should be analyzed based on its own characteristic. Each project should be analyzed based on its own need and a testing strategy be created for it. Even a project that already have a testing strategy, these kinds of needs might change in time, so reevaluation of testing strategy for previous projects also might be a good idea.

The spectrum of tests

There are many different types of tests with different names. Some of them even overlap with each other. But I think a more useful way to think about them is to think about them in terms of how many components are involved in the test.

Our goals when selecting a testing strategy

What we want is a testing strategy developed on project-by-project basis. It should maximize the test confidence while minimizing the test churn, the run cost, the difficulty of setting up various cases, and the difficulty of pinpointing issues. Also ease of test is something that we should consider too, in the subsequent section I’m going to explain what I mean to each of these aspects.

• Confidence: How confident we are that when a test passes, it means that the feature actually “works”. The more components involved, the more confident we are on the test
• Test Churn: How likely it is that when we refactor and modify a piece of code, we need to also modify the tests. The more components involved, the less you need to modify the existing tests. It’s because you’ll frequently test flows at a higher level. It won’t matter if the implementation’s specifics change. On the other hand, if you write tests for every function, you might need to change the way the tests are written for every refactor.
• Run Cost: The expense involved in conducting the test. The cost in this case may be time, system resources, or even money. When there are more components, the cost is often higher because the test takes longer to start up and more CPU/RAM/Disk resources need to be assigned.
• Case Variations Setup: How difficult it is to write up tests that account for all possible case variations. It is simpler to set up different instances with fewer components (e.g., using mocks). On the other side, setting up different cases becomes more difficult the more components there are.
• Pinpointing Issue: How difficult it is to identify the specific flaws that are to blame for your test failing. Finding the problematic components—and there may be more than one—becomes more difficult as the number of components increases. You can quickly find the bug if you only test one component.
• Ease of test: The types of tests we’re choosing should be able to be performed on the specific type of project at hand. For example, if we have a legacy application with a lot of tightly coupled components, performing unit test on it is difficult if not impossible. But we can do integration testing and end to end testing.

Here’s the summary of our goals when select a testing strategy:

We want a testing strategy developed on project-by-project basis with the following characteristics:​

• Maximizes the test confidence​
• Minimizes the test churn
• Minimizes run cost
• Minimizes the difficulty of setting up various cases​
• Minimizes the difficulty of pinpointing issues
• Maximizes ease of test​

Now that we came up with these goals, and we also know a little about the how different types of tests handle these cases, we’ll do an exercise that compares and contrast 4 different types of projects.​

Sample exercise to determine testing strategy for a project

We will be going to perform an exercise to analyze which is the best type of test for a given project. We’re going to describe the project, what it does and its limitations, and then we’ll see what type of test is best suited for testing this project. We’re going to assign a number to our satisfaction with a specific aspect that we discussed above. Please note that this number is representative of our satisfaction with that aspect and is not directly related to that aspect. Also, note that there’s no exact science on assigning such values. It comes down to your estimate. Also please note that we can combine different types of tests to get the best result out of it.

Old Legacy inventory application

This application is consisting of an API, Server side, Database and a front end (Number of components 4) written in MVC razor pages. This application has a lot of tightly coupled components which cannot be isolated and tested on their own.

As we can see the most suitable test for this type of project is integration tests. The reason is that this project has a lot of coupling between its components, so unit testing them are extremely difficult. If we look at the ease of test aspect in the table above, we see that it only receives 1/10.

Date and Time Library

This is a math library written for the sole purpose of being used by other applications. This library cannot be served on its own. All the functionalities in this library are pure functions which are only dependent on their input and independent of outside sources.

Some projects can be fully tested only with one type of test, in this case unit test. This applies to the project with high level of logic in form of pure functions.

Shopping cart with loosely coupled components

This application is an e-commerce application with a shopping cart. The user can add products into its shopping cart and then checkout. It consists of an API, Client side, Server side, Database (Number of components 4). The components of system are not tightly coupled, and everything can be tested in isolation if need arises.

As we can see for this type of project again integration test would have the most return on our time investment. It’s partly because the app has a lot of moving parts. But unit tests can also be valuable because the ease of test is high due to lack of coupling in our components.

HR Candidate Management System

This application is a candidate management system used by people in HR. It can create interview for candidate, search potential candidates, see their resume etc. It consists of multiple APIs, Client sides, Server side, Database, Elastic Search, Redis. Different part in the APIs and server side are relatively coupled but not entirely.

We can see that here also writing integration tests have the most return in our time investment. That because we have many components involved in the application that needs their interaction tested. There are a lot of infrastructural concerns that each on their own can cause flaw in the system. So, the best testing strategy for this type of project is using a combination of integration test and end to end tests.

Key takeaway

I think the most important thing to learn from this article is that each project should be analyzed on its own. We can’t just select one type of testing strategy and apply it, but we should come back time to time to the project and re-analyze it. Also, it helps if each project has its own specific testing pyramid, designed and analyzed based on the criteria’s that are important to us.

Further Reading

The Practical Test Pyramid

Just Say No to More End-to-End Tests

The Right Balance Between Different Types of Test

Summary

In this post, first we saw why choosing a good testing strategy is important if we want to maximize the benefit a testing strategy can provide. Then, we saw what are the different test types that can be leveraged in our QA efforts. We also saw in detail the different pros and cons of those tests based on their inherent characteristics. After that we set some ideals or goals in our testing that helped give our efforts some direction. After that we followed a set of exercises to analyze multiple projects to see what kind of test is the most suited one based on the goals that we set in the previous section.