The Right Balance Between Different Types Of Test

Testing strategy is one of those topic that can get very tricky to figure out at the start of project. What I mean is every project depending on the type, needs a specific type of test more than others. For example if a project contains a lot of complicated logic and algorithm then unit testing is what we can rely the most on.

But what if our project is mainly consists of web pages with a lot of coordination in the controllers? In these situations unit testing shouldn’t be the majority of our test cases. Because there are no complicated logic involved. Often times a piece of code is so small, that it is unlikely that any problem is going to be encountered when it’s used. In an Asp.Net MVC example, anything happening in controllers are going to depended on other services. Which in turn are not that complicated. So in these situations integration test might be more worthwhile. In this post I’m going to go through these different type of tests from a developer perspective and compare their characteristics to see which one of these are suitable for when.

Unit Test

What type of code does not benefit from unit testing?

Unit test is the least expensive type of test. The resources spent on running the tests are minimal and it’s very easy to setup. But this type of test is not as useful if it’s not used in a very focused and specific way. In other word, it is not useful for applications that have a lot of coordination code with minimal logic. A good example is a Asp.Net MVC application, in this type of application most of the code in our controllers are coordination code. This type of code marshal the request to different views and different responses. But the logic is minimal, more important than that is that in order for the controller action to do its things, it needs to collaborate with other parts of our application.

But in these situations, testing that the controller’s action return something is not enough. Because the data that goes through that action is mocked. So we have no guarantee that these components actually work together correctly. We only know that one specific component work, and we don’t know anything about how these individual components interact with each other.

expect(umbrellaOpens).toBe(true)

tests: 1 passed, 1 total

**all tests passed** pic.twitter.com/p6IKO7KDuy

— Erin 🐠 (@erinfranmc) July 10, 2019

What kind of code benefit the most from unit testing?

I would say most of the time codes with the least dependency on outside like pure functions. But again a simple function that is pure does not need unit tests. Another type of code is algorithms and complex business logic. Also applications that don’t have strong typing might benefit from unit tests (There are better alternative for that kind of scenario). Another benefit with unit tests are their focused feedback which can help us understand what exactly went wrong. As opposed to integration tests and end to end tests that are less fine grained.

But still there are a lot of other opinions on this matter. Some people completely ditch the unit testing phase because they believe it has the least amount of return on investment. I don’t agree with this sentiment in general, but it also depend on the type of application.

Another approach is TDD which we write test before actually implementing the feature and let the test drive the design. But this type of approach is more about design and less about testing the application. Also TDD does not advocate creating brittle tests or using too much unit tests to reach 100% coverage. Those are the things that make our unit tests less effective. On the other hand there are people who think TDD can drive our design to bad places etc.

How unit tests can be abused?

Another important thing is that we shouldn’t unit test implementation detail. This most of the time comes in the form of interaction testing and verifying that some method calls another method and it passes something inside the method. Which in my opinion is a waste of time. By doing that not only we don’t prove that our code work, but also make our test very brittle and unmaintainable. This behavior is called over specification, here a post from Roy Osherove that goes into more detail about this phenomenon. Also it can come in the from of testing private methods by making them public. If an application does that we’re going to see a lot of public methods that didn’t receive any call outside the class that use them.

Also unit test is the most abused type of test. Because it’s used a lot of times to reach the 100% code coverage which is useless. The reason why is that to achieve 100% code coverage we need to test some parts of our code that are dead simple and basically do not need any test.

Integration Test

Where integration testing can be beneficial?

Integration test is the sweet spot between unit test and end to end test. While unit test does not prove that a piece of functionality works, integration test can give us more confidence in that regard. But if an integration test fails, it does not tell us why exactly it failed. It only show us the symptom and not the root cause of the problem. To get to the root of the problem more effectively, we’re going to need unit test and proper logging.

A lot of people think our test should be mostly consist of integration tests. That’s because the integration test is not as costly as end to end test but still provide us with more confidence that a feature works.

Downsides of integration testing

But the downside to integration test is that it only test the current behavior that a piece of code is used for. As opposed to unit test that tests all aspects of our code with more focus. Because it’s slightly detached from testing the behavior only. Suppose we want to use the same component in other part of our system. If we test all aspects of the code under unit test, we can be more confident that the code is going to behave as it should if we use it in other parts of the app. But integration test cannot do that as well as unit test does.

Also there’s the cost associated with creating accurate dependencies and data for integration test. Another big issue with integration test is that it does not help up as much as unit test when it comes to refactoring. When we refactor there are going to be more unit test failing than integration test. That can help us to understand if we break something. But one can also argue that unit tests are bad exactly because of that reason, they fail and make refactoring more difficult. But in my opinion if we write our unit tests correctly, they can be more help than burden.

End-to-End Test

Benefits of end to end testing

End to end test is the most expensive type of test, but it can also be the most crucial. It’s the type of test that proves all the component are working and the feature can be considered as done. With this type of test we can create a real world scenario that test all layers and component of our application. Also this is the test that can bring the perspective of customer into our test. More specifically the ideal end to end test is written in a way that mimics how the application is used by the user. But we can’t do that when unit and integration testing because the test is too focused on the various aspect of the system.

Also this kind of test helps us expand the test coverage to all aspect of the application. What if a third party API call failed or a database call or some other type of infrastructural problems. Another important aspect is the user interface of the app that is also tested by end to end testing.

Downsides of end to end tests

But there are a lot of problems associated with overusing it. First of all this type of test is very resource intensive. That means we can’t run it as often and have a quick feedback loop which can add to development time. The second problem is that it’s very difficult to find the root cause of failure with only end to end test. Anything can be responsible for the failure, from the front end part of our app to the back end and APIs etc.

Also there are problems associated with the test focus. Because the breadth of it is very wide, failures can happen that was not directly related to our code like when we have a third party API or a component. Not to mention that this kind of tests can be flaky at times. For example the test can fail because the server which its run on didn’t have enough resources or you need to manually change the test behavior for specific part of the app to load/function.

This type of test might tell us something is not working at a very high level, but it does not help us much with finding what the issue is exactly. Add to it the fact that sometimes smaller bugs are hidden behind bigger bugs. So although I admit that end to end test is very useful, they have their own downsides and limitations.

In the case of finding bugs we need to think smaller, but how much smaller exactly? In my opinion the ideal size lies somewhere between unit and integration test. But I’m more leaned toward more integration than unit tests, but that can be different across projects.

Test’s Return On Investment

Test return on investment is about maximizing the benefit we receive from performing different type of tests. This figure can be different from project to project. But as a general guide line, Google suggests 70% unit tests, 20% integration tests, and 10% end-to-end tests. This can be shown as a pyramid.

But it is highly debatable that this pyramid can be used in every situation. Each project needs to come up with its own test pyramid that maximizes the return on investment given its nature. Here’s the link to the google article that proposes this pyramid.

But there are other people who propose a different distribution between tests. One of the most interesting one is “The Testing Trophy” which Kent C. Dodds proposes.

Here’s his blog post with an accompanying video that he goes into more detail about why this approach is the best. The static part at the bottom can be considered as things we setup to analyze our code like linters and analyzers. Also worth noting that the author of this testing trophy looks at things from JavaScript/front end point of view. I’m not sure I can agree with any of the proposed test pyramids/trophy. I think it goes without saying that there can’t be a hard and fast rule which everyone can follow in this regard. But the real question remains, how can we figure out what is the right balance for us? I’ll try to answer this question in the next section.

Effect of test quality/approach on return on investment

The type of tests we write is not the sole determining factor when it comes to return on investment. But also how we write our test can effect it as well. For example in the case of unit testing when we test the implementation detail of a method as opposed to its behavior, we reduced the return on investment. Because the implementation is the most likely thing to change in our code but the code final behavior is less likely to change. So by focusing too much on the internal implementation we risk creating brittle test which needs to be fixed. This can steal time away from tests that are more vital.

So we need to also strategize about what type of problem we want to catch. Analyzing what problem a test solve is important because it can be multiple problem. For example the unit test can be written as a tool for the developer to catch bugs early on. In this case it might not be able to prove that the actual application is working. But in another project that have a lot of logic in some library, the unit test actually proves that the code works and this is what we rely on. Analyzing these points help us write our unit tests in a very specific ways that are suited for the problem at hand. By doing that we prevent writing tests that are excessive and unnecessary considering the problem that we’re trying to solve.

Finding the Right Balance

First of all, I think coming up with a general guide line for balanced testing in an application is not an accurate process. So any type of pyramid or trophy etc is subject to change as the project moves forward. The most important thing in my opinion is to understand which type of test is suitable for which type of code/scenario. By doing that we direct our focus on the testing process rather than coming up with a inflexible testing diagrams. By doing that our testing approach is going to shape into some kind of diagram on its own.

What type of test should be written for what type of code?

A good argument for choosing between tests are made by Steve Sanderson in this blog post. Here’s a diagram that help us visualize the cost benefit analysis.

Here’s how he describe different type of code in relation to the above diagram.

Complex code with few dependencies (top left). Typically this means self-contained algorithms for business rules or for things like sorting or parsing data. This cost-benefit argument goes strongly in favor of unit testing this code, because it’s cheap to do and highly beneficial.

Trivial code with many dependencies (bottom right). I’ve labelled this quadrant “coordinators”, because these code units tend to glue together and orchestrate interactions between other code units. This cost-benefit argument is in favour of not unit testing this code: it’s expensive to do and yields little practical benefit. Your time is finite; spend it more effectively elsewhere.

Complex code with many dependencies (top right). This code is very expensive to write with unit tests, but too risky to write without. Usually you can sidestep this dilemma by decomposing the code into two parts: the complex logic (algorithm) and the bit that interacts with many dependencies (coordinator).

Trivial code with few dependencies (bottom left). We needn’t worry about this code. In cost-benefit terms, it doesn’t matter whether you unit test it or not.

In my opinion for most applications the end to end test should not be the majority of our tests and it should be used more carefully. Here, our decision is more about deciding between unit test and integration test. This diagram shows that trivial code and coordination code do not benefit much from unit tests. But more complex pieces of logic and algorithm benefit the most from unit tests.

A real world example : Asp.Net MVC application

As an example consider a Asp.Net MVC application with a service layer containing business logic. In this case I think unit testing the controllers are not beneficial. Assuming that controllers do what they suppose to do that is coordination. If our controllers contain a lot of code and do a lot of things, then the situation is different (better to refactor the controllers and move the logic into separate services). On the other hand unit testing business logic might be beneficial if they are not too trivial.

If they are trivial, we can integration test them along with our controllers. Also we need to keep in mind that unit testing does not mean testing implementation detail here. I saw in a lot of code bases the private method in a class made public in order to be unit tested. Also sometimes unit tests are written to make up for the fact that we don’t have strongly typing for example. So if we use JavaScript and we don’t have static typing maybe relying on unit test is not a good approach (better approach is to use something like type script). In short if we can put some mechanism in place to help us write less unit tests we should do that so. By doing so we let the unit test be more about proving the logic and the code is correct than to test what should have been caught be some kind of compiler.

A real world example : Visual Studio extension

Let’s now consider a visual studio extension. In my experience a visual studio extension benefit the most from unit tests. The reason is that it has minimal coordination code. Also most often it does not call other APIs over the network. The only thing that can go wrong in terms of infrastructural is the hard drive that the extension might read or write to.

A real world example: algorithm library

Suppose we have a library the help us find the most efficient path to a destination. This application does not have any coordination code. It does not speak to the database or call any API. It mostly consists of pure functions that have path finding logic in them. The only thing that might make sense for this application in terms of testing is unit test.

Summary

In this post, I first introduced different kinds of test from a developer perspective. Then we saw which one of those tests are suitable for what kind of situation. I also discussed how we can find the right balance between those of kind of tests and maximize their usefulness. If you think I missed something please let me know in the comments section below.

Share...

Hamid Mosalla

Programming Adventures