nicholascloud.com Existence is Identity, Consciousness is Identification

How not to unit test

I remember when I wrote my first unit test.  It happened at a point of realization -- I had enough experience writing code that the pain of software bugs started to outweigh the joy of software development.  I'd heard about unit tests from friends, read about them on blogs, heard about them at conferences.  When I inevitably wrote my first test, it was pretty frightening.  I understood enough about unit tests to limit each test to a single class method or property, but I did not understand that dependencies would be my undoing.  My early tests traversed all layers "downstream" of the object under inspection, and even queried the database and live web services.  My tests tended to look like this:

Since those early days, I've learned a lot about unit testing, specifically that it helps you break your dependencies so that your code can be tested in isolation.  Testing across software "layers", i.e., testing a unit of code and its dependencies can be considered anti-testing, and actually gives you false confidence in your system when, in reality, it is a fragile cacophony.

1. Tests across layers require an exponential number of tests.

Every developer who writes dependency-traversing unit tests eventually comes to the sickening realization that no matter how many tests he writes, the possible combination of parameters and possible code branches that could be executed in any given test is, well, staggering.  For every new layer, or every new dependency, the number of tests for a given member would be raised to the value of the number of tests for that dependency.  In contrast, if the dependencies of a given unit of code are mocked or stubbed for the purposes of unit testing, the developer only has to write the tests that would test the code immediately under inspection.  In other words, breaking dependencies = additive, keeping dependencies = exponential.

2. Tests across layers require simulated application state.

Any developer maintaining a large config file for his tests knows how much of a pain this is.  If the application relies on:

• configuration values that are read by the application at startup;
• configuration values that are provided by background threads;
• configuration values that are retrieved from database calls;

then code that relies on this state, either directly or through dependencies, must have unit tests that recreate it for every test run.  Again, this can be very prohibitive, especially if the volume of configuration data is vast, or in relatively inaccessible places (e.g., other threads).

3. Tests across layers can give false positives.

A unit test is typically named for the unit of code being tested, for example, CalculateCartTotal_SomeState_SomeExpectedResult().  If this unit test spans application layers the developer who wrote it won't immediately know if an error in the test is a result of a calculation failure, corrupt database data, missing session values, etc.  The test might fail for reasons that have absolutely nothing to do with calculating a shopping cart's total price.  Precious development time is often spent debugging test paths to determine where the offending dependency lies.

4. Tests across layers can give false negatives.

Because of #1, it is possible that all tests for a given method will pass, even though the method itself is flawed.  This can occur because the chance of a developer missing a given combination of state and behavior in a unit test is significant.  This is possible in test suites where dependencies are removed, of course, but is far less likely given the reduced number of possible tests and the ability to simulate known state (by mocking and stubbing) that will produce exact, expected results.

5. Unit tests are for code, not data or connectivity validation.

A common justification I hear for encouraging unit tests that cut across application layers is: "It helps us know if our data is correct."  This justification confuses the role of unit tests and integration tests.  Unit tests are designed to determine if code functions properly, not if a given system can connect or retrieve data from another system.  Integration tests help us to validate those boundaries, but unit tests are only concerned with the behavior of code.  The validation of data should occur as close to the data source as possible, even within the data source if possible.

6. Unit tests across layers are slow.

One of the primary benefits of dependency-ridden unit tests is that they are fast.   A developer can run a suite of unit tests in a matter of seconds, or minutes if the code base is particularly large.  This means that the developer gets instant feedback about changes made to the system.  If unit tests cross data access boundaries, or query live services, performance will drop dramatically.  In a high-pressure deadline-driven environment, this encourages developers to ignore unit tests altogether, or leave them for "later" (you know, the later that never comes).  Without unit tests as a feedback mechanism the developer will revert back to the days of design-by-debugging.

7. Remember: testing the sum of the parts = testing the whole!

So if dependencies are bad in unit tests, how does one test the system in tot0?  If all dependencies are tested in isolation, that is, without reference to, or reliance on, any other testable code, they will work when combined if all unit tests pass.  Software systems should be decoupled anyway -- your code should have a sparse number of direct references to concrete types, and instead, rely on references to abstract classes or interfaces which can easily be mocked or stubbed with mocking frameworks.  Real unit tests help you see where these dependencies are, and eliminate them.

In this example, ClassA has a direct dependency on ClassB, as show by field _b.  When a unit test is written for SomeMethodThatUsesB(), ClassB will directly affect the outcome of the test. In contrast, If ClassA depends on an interface for ClassB:

then the implementation of ClassB can be replaced with a mock object with known state and behavior, that will affect the unit tests for SomeMethodThatUsesB() in a known and controllable way.  Not only is this good unit test design, it is good object design as well (the implementation for ClassB can be extended in your application without breaking classes, like ClassA, that depend on the interface IClassB).

The good news is that writing correct and accurate unit tests will make you a better developer.  The bad news is that people often feel a warm fuzzy feeling if their tests span layers in an application, because they believe it offers better "coverage".  Nothing is further from the truth.  Breaking applications down into discreet units and testing each unit in isolation, under controlled conditions, is the only way to achieve accurate test results and prevent those early AM calls from disturbing your beauty sleep.  Now go forth and test.