Myth 0: Unit tests are unnecessary if we have integration tests.

One thing I’ve run into a few times is people who have the idea that unit testing is useless or unnecessary if you’re running integration tests. In some cases people even go so far as to say they’re unnecessary if you have functional testing. This is a mistake.

It is true that unit testing is unnecessary. In fact, testing period is rather optional and unnecessary. You can write code without doing any testing at all, package it up, and sell it. People might even buy it. If you’re really lucky it might even work some of the time. You’d be pretty stupid to try it, but speaking strictly of necessity testing is not necessary.

The reason you test at all is because you want to ensure that you have developed a good product. There are all kinds of stages to testing though and even if you decide you will test, strictly speaking no single form of testing is necessary either. The reason you perform any given kind of test is because you want the benefits that it offers.

Here are the kinds of testing that most people engage in:

Acceptance Testing
Verifying that the product does what the customer wants. An acceptance test might be, “remains stable under extreme loads,” though this is rather vague (you’d want to negotiate with your customer what “stable” means and what “extreme load” means).
Functional Testing
Verifying that some aspect of the product does what you think it’s supposed to do. An example of this might be verifying that a user login form pops up when a user tries to access a page that requires authentication and they haven’t yet.
Integration Testing
Ensuring that objects work correctly when tied together. This is more of a development task in my opinion. An example might be testing that a user authentication library’s components work as a user authentication system when used together. Integration testing often encompasses an entire library or component but will often not encompass an entire product.
Unit Testing
Unit testing is a development task. It involves ensuring that specific functions and units within a component work the way they are supposed to.

Many people also differentiate between “regression testing” and normal testing. The only significance of regression testing is that it is about old behavior. Regression tests are simply tests that you wrote for previous releases. Strictly speaking it is unnecessary to run these tests, the only reason you do is to ensure that when you added new behavior and/or fixed bugs, that you did not break old behavior or reintroduce bugs that you’d previously fixed (because of course you’re writing tests for all your bugs, right?).

As you can see, these classifications build up in layers. A unit test tests one single entity in a software component. Integration testing tests the integration of many software entities into a component. Functional testing tests that a set of components implements specific functionality. Acceptance tests verify that that functionality implements the expected product. Omitting any one of these layers and you suffer a lack of the benefits they offer. You’re basically not testing that level of your software product and of course the argument against doing this is the argument for doing testing period.

Lets consider an example to get into mind what it is that unit testing offers that the higher layers do not. We don’t need to go all the way up right now because we want to examine what unit testing itself gives us, so we’ll assume that integration testing is being done. This example will use a user authentication system as an example.

A user authentication system is going to have all sorts of entities that implement it. There may be a persistence entity or set of entities that makes sure user data is kept around for a while. There may be entities like “User”, “Role”, “Group”, “Credential”, etc… There should almost certainly be at least one encryption scheme for relaying passwords from user to the system in order to ensure security.

An integration test of this component might be implemented by writing a scriptable program that logs in a user given a specific account name and password and then verifies that when a specific permission is requested that it is either given or denied. A complete set of these tests will very likely exercise every aspect of the code written to implement the behavior, just as a complete set of functional tests will exercise much or all of the behavior tested by this integration testing system.

Lets say now that you are writing this component. If you’re not doing unit testing here then of course you are writing a significant amount of code before you have any idea how much of it works the way it probably should. You will not know if the permission checking system works until you’ve implemented the entire system and can run an integration test that tries to gain permissions. If you were writing unit tests on the other hand, you could check that the permission checking system is probably working as expected long before you have to create other things like persistence. In fact you might not even need real users, groups, roles, or whatever because you can create fakes of these things to test the checking algorithm.

If you are working on this system, it’s already in place and has integration tests but no unit tests, and you want to swap out the encryption algorithm with something more secure. There are two things here that may very well be missing:

  1. The design may not be well suited to this change.
  2. You have to go through an entire login process just to check that encryption and decryption work correctly

These issues are actually related. Unit testing offers an inherently valuable service beyond simple testing: unit testing encourages a decoupled design because you are required to make your objects reusable in order to be testable as individual units. Integration tests, operating at a much higher level, do not require well thought out design that allows dependency injection, overriding, and use of individual aspects because all of the units that implement these things are available. Writing a unit test for one, solitary unit makes sure that you are keeping everything that is not directly related to the specific behavior implemented by that unit separate from other stuff.

A good design lets you plug and play things together to implement functionality rather than requiring you to track changes across multiple units. When your components become deeply coupled internally it becomes much harder to implement even the most basic changes without causing failures in ways you had no expectation of. Since testing units as units requires a decoupled design of those units, changing one becomes much less risky. What you gain here then by unit testing, and what is not offered at any other level, is a decoupled product with less rot that allows you to respond to change much quicker and cleaner.

The second part of the idea is that you can more quickly develop a single entity because you can test it more quickly. Not having to perform a large change set in order to verify the code you’re currently working on is a huge benefit to productivity. You can work on things that are not ready to integrate due to interface changes or even non-existence of pieces of a component and have some confidence that you are moving forward. Integrating and then debugging is a much slower method due not only to the fact that more must be accomplished before testing is possible, but also because the brain of the developer has to track a whole lot more that way and no matter how brilliant a developer is, there is an upper level to the amount of disparate things the human mind can be thinking about at the same time. When this barrier is broken a developer begins spending a lot more time rediscovering rather than holding the entire thing they are working on in memory. If we were talking about software performance we’d be talking about cache misses and page faults.

In short, if a developer can work on one, specific entity and test it they can then forget utterly about the details of that entity and move forward to another entity or at a higher level of abstraction. Lacking unit tests and only being able to test once integration is possible reduces this ability, especially if the design has coupled due to not being actively discouraged by the requirement for entity reuse in unit tests.

Furthermore, debugging itself is a larger task when faced with an entire integration or more due to the simple, basic fact that there’s a lot more to debug. While the act of logging in a user, attempting to gain a particular permission, and checking to see if you have or not can fail in any number of ways, the act of encrypting or decrypting a password is a much smaller thing to debug. If the fault is in the encryption functionality then finally debugging to that point to find out that a perfectly valid password failed encryption or decryption and then why is a much larger task than testing and debugging this behavior directly. If you fail more than once to implement your encryption scheme, but have to step down to it each and every time you check, or pass through a whole list of successes before the one failing point, then you are adding a whole lot of unnecessary work and time to what is already a complicated endeavor.

Finally, when you do integration testing the combinatorial nature of the problem you are testing explodes quite rapidly. Testing an entire user authentication and permission system is several orders of magnitude larger a problem than testing an encryption scheme or account search function. This creates a similar problem to the developer having to track more information in order to accomplish something before he can test it. Exercising an entire component means you have to work each possible branch in the system. Are you so sure you can even accomplish this? Unit tests on the other hand, being focused to one particular entity, have many less paths of execution to check. You can fully test 5 units much easier than you can fully test a component of those same 5 units. Remember that this an exponentially growing problem; 3 branch points creates 23 paths, not 2*3.

Yes, this means that you are relying on the lower levels to do some of the testing at the higher levels. Testing every path of execution in an entire product is almost certainly outside of practicable reality. It would be akin to solving chess, which is said to require more atoms in the universe to compute. This means that yes, sometimes bugs get through that are caused by the inter-operation of several entities or components because you didn’t execute that exact path in the upper layer tests. When this happens what you do is you take the hard path and debug. When you find the reason for the failure, the component and/or unit that is not working the way it needs to, you write new tests at that layer to check the behavior.

Generally speaking a bug is either caused by an entity that is not obeying the contract it is supposed to, the misuse of an entity that is obeying its contract, or a misunderstanding about what the requirements of the entity were. Debugging then becomes an act of discovery about which of these issues you are facing, where, and how. Then you can discover what you missed in your unit test that failed to account for the complete interface of the entity, what caused your entity under test to misuse another interface, or what needs to change in the requirements for an entity in order to correctly perform the task that is required of it.

In conclusion, by not testing at the unit level, which is fairly well the lowest level of abstraction it is reasonable to test at, you miss a great many benefits offered by unit testing that are not offered by higher layers of testing. You increase the amount of effort that must be accomplished without a harness, thereby decreasing the safety of your code changes. You vastly increase the complexity that your testing must reach in order to fully test your product. You lose the benefit of forced encapsulation and decoupling created by unique requirements imposed by unit testing. Finally, in so doing you greatly reduce the short and long term value of your product and increase the amount of technical debt it will accumulate over time; you vastly decrease the maintainable lifetime of your product, that amount of time in which the continued maintenance of a code-base is less than throwing it away and starting over. In these respects then, every developer, development team, and manager should consider unit testing absolutely necessary no matter how much other testing is being done.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s