One day I had a job interview over the phone. The person on the other end asked me what I thought of unit testing. I explained how I had, on my own, adopted the practice and tried to get my fellow team members to as well but had met with resistance. I further explained how I tried to write them before coding, but lacking a rigorous sort of process or environment I sometimes got lazy…but I always regretted it later. I told him how I thought doing the tests before the code was pretty important, made sure you were thinking ahead, and that when I actually did it I tended to write better code faster. I also explained how even insisting on there being unit tests in projects I lead tended to cause developers to write them after the fact as just another thing they had to do rather than actually getting any use out of the practice. I also explained that it was the first practice to go out the window when the time crunch happened because management didn’t understand or appreciate the practice even though it gained them so much.
The person on the other end agreed with me on everything, said he liked to write unit tests first, etc… Later I was offered the job, I accepted the offer, and couple weeks later I was in the office working on this guy’s team.
There wasn’t a unit test to be found anywhere on a brand new codebase. There wasn’t even any automated integration tests. One developer, desperate for something, had made a client program to connect with this service program, take keyboard input, and spit out to the console. We would log into the development server, shut everything down, copy a bunch of files over, restart stuff, run your client program, copy a line out of a word document that you had personally made for testing, fed it as input, manually looked at the result and say, “Yeah, that makes sense,” and then log into the database to make sure it had done its thing there as well. There were a couple source files that had an, “#ifdef UNIT_TEST,” check that would generate a main function that would take input, call some functions, and put the output on the screen. The architecture itself suffered as well and was primarily a large collection of globals that were manipulated by functions that could be in any module; it was essentially untestable for the most part, which is number one reason why almost no unit tests existed or so I thought.
As I continued my adjustment phase I came to know that this guy actually hated unit testing. What I just described is exactly what he meant by that term and what most of the world calls “unit testing” was new and strange. There was a clear disconnect between what the two of us had been talking about during the interview process.
As we continued down this path for a few months there came a time when we were told to implement a particular version of a scrum-like, lean-like process. The company that was consulting us gave us some training on their “unit tests”. These were composed of a large program they’d developed which would perform socket connections and communicate with services. You’d program it with a scripting language and activate it with a directory searching makefile system that would find these scripts and run them. They would link libraries into executable that could be driven by this program and these scripts consisting of “thin” layers on top of the library to make it driveable in this manner. The engine had regex capabilities and could take the output received from network sockets and check that it matched specific patterns. The same was done with the logging system.
Neither of these is a bad practice, and in fact the later is actually a GOOD practice, it just isn’t unit testing. I’ve been rather surprised how few people seem to know what unit testing is, what it gives them in return, and how to do it effectively. Of course these are things I continue to work on myself, but the first stage that must be reached is knowing what unit tests are and are not. So without further ado, here’s what they are not:
Unit tests are not interactive
A unit test should not be getting input from the user nor providing output that is checked by one. This includes the developer as a user. A unit test should only be printing results, if it even does that much. There is some utility in having intermediate data printed to the screen for when the test fails, especially if you’re using a crappy framework that can’t print the values of asserts but only the expression used in them, but you should be able to turn this off and you really should be using a debugger instead.
A unit test should be a program that runs without interaction, does its tests, checks the results, and expresses PASS or FAIL. That is it. The most useful way for it to express this result isn’t even on the screen in any way but in the return value to the operating system: 0 for pass, non-0 for fail.
Unit tests test a single unit
A test program that has to link to an entire library is not a unit test. Sometimes it may be easier to, but this should certainly not be required by the framework you’re using. This is especially true in languages that lack virtual dispatch or hybrids like C++ where much of your work might be static. In these cases it’s very important that you be able to override and mock behavior by linking with stub versions of functions and classes rather than the real thing.
A unit tests a single, solitary unit. In a large library, composed of many units, you should have many unit tests that compile and run as part of the process of building that library. A library that implements a user, roll, and security session for example is going to have several components in it probably not limited to user, roll, and session. A unit test doesn’t check all of these together, it checks the behavior of only one of the units. Generally you want to mock or stub out the rest to provide as much control and simplicity as possible. This focuses the testing effort on the key piece when actually using the rest of the library means that failures could be caused by units that you’re not intending to test.
Further, having a whole bunch of components that need to be used together in order to build and run a test means that when it comes time to change the behavior of the larger entity you’re working on you will need to make changes across the board before being able to run your test. This is much harder, more prone to error, and encourages hacking. It adds to the decay of a project, further ingraining coupling, etc rather than alleviating it. It is much easier and much faster to perform your changes incrementally, testing as you go, and this can only be done if you don’t have to implement EVERYTHING before you test. This is one of the main benefits of unit tests in my opinion so if you have to link it all together to do your tests you’re severely missing the whole point.
Unit testing should be easy or your lazy developers (like me) won’t be encouraged to do it and will do silly things like write them after the fact, just throw something together that sort of looks like a test but doesn’t really test anything, etc… They’ll work to get the requirement done and that’s it. Do it right though and your developers may actually come to realize that it’s much easier to work with unit tests that try to write everything without harness, that they’re faster and better that way, and go home less frustrated and suicidal or homicidal.
Unit tests are written in the language the unit is in
If you’re writing some complex, or even simple program to drive another program and eating scripts, etc… what you’re doing isn’t unit testing. Although problems can be solved by adding an extra layer of indirection, the use of the unit you’re testing should not be so encapsulated. You want to be directly using this unit, feeding it parameters exactly how it’s normally done and checking the results exactly as they are returned. The only way to do this is to use the language the code you’re testing is written in so that you can call its functions themselves.
If you are not writing in the language that your code is in then there are numerous modes of failure that can be the result. Your test can fail not because the unit is broken, but because all the crap you added to use it is broken. Maybe your script program has a bug in it and is feeding your unit a bunch of bullshit. Maybe the code you used to turn your unit into a program drivable by your scripting engine is full of bugs. You just don’t know. The whole purpose of a <i>unit</i> test is to focus on that single unit of code and check its functionality against expectations. Anything extra is just added fluff that gets in the way.
In fact, you should be pulling stuff OUT of your test suite. You should be mocking functions that your unit uses and injecting them into the system through virtual dispatch or linkage. This way you’re simplifying the system even further to the point where JUST the behavior of the unit is under scrutiny.
If you’re writing some gigantic engine that reads scripts and runs a program then what you’re doing is integration testing. It can temporarily take the place of real unit testing in legacy systems that are too coupled to do it right, but should not form a permanent framework.
Unit tests are written first
I’ve very often broken this rule but it’s really a lazy practice to do so that in the end only causes problems. Unless you are writing your tests first, making them fail before writing the code, you don’t know if the code you’re testing is even BEING tested. You thus don’t know if your tests are giving you anything at all. One of the largest benefits of having unit tests is the ability to refactor implementation details while being fairly confident that you’re not breaking anything. If the unit test checking the unit you’re altering is complete then when you’re done and the test still passes, the rest of the system should behave exactly as it did before you started changing things. This is very, very important to the practice of agile development.
One thing I have done to try and alleviate this issue with post-coding unit tests is to use a code coverage tool. This also though doesn’t really give an accurate picture though because simply executing a block of code makes it “tested” according to the code coverage reporter, but doesn’t mean you’ve checked anything that block has done. It’s very difficult and time consuming to write tests post-facto.
Furthermore, another good thing about unit test writing is the effect it has on the design of your code. Having to make your code testable makes it also reusable, decoupled, and simpler. These things are important for maintainability of course and when code starts getting all tangled up it also gets complicated and fundamentally unreusable…and thus untestable. Writing code without a harness means you have to be very careful about accidentally introducing coupling or you’ll be tearing your hair out trying to write the tests. This is probably the primary reason I’ve seriously regretted it when I got lazy and didn’t do my tests and do them first.
Of course, if you’re not inclined to like SOLID design principles to begin with then you’re not likely going to enjoy unit testing too much. Adopt unit testing though and much of these principles will simply force themselves upon you.
So that’s some of the things I think unit tests are not that I’ve seen go by the term.