Someone wrote to me:
Hey Michael – So I was at the Innotech conference this week in Portland and attended a talk about using mock objects to make unit testing easier. It definitely sounded like a way of getting some test coverage of “big ball of mud legacy code,” but it also threw up a bunch of red flags in my non-engineer brain.
Since I’m not an engineer, I don’t have any experiences to look back on to have an opinion on whether or not using mock objects is a good idea. But I felt like something was missing. Let’s say you have some code you want to test, but you want to isolate it from other layers of the architecture so there are no messy dependencies. So you put in some “perfect” mock objects to test the code with. GREAT! It works. Then you put in your real database or other real module that your code is supposed to work with and it seems to me that it might suddenly cease to work because you didn’t code it to your real world environment.
It feels like having a bunch of clothes tailored to a size six model, because you are also a size six but when you go to try on your new clothes, none of them fit because the fit model didn’t ACTUALLY have the same proportions as you. Sure, it takes more time out of your day to tailor everything to the “real world system” but it seems like that’s the only way to get a real fit and know that your investment will work.
Let me preface my response with the disclaimer that any kind of automated testing is better than what most teams are doing.
One goal of eXtreme Programming is to get the test to run quickly (like in a few milliseconds) so people will run it over and over as they're refactoring code. Running the application in the context of the actual database, actual GUI, actual web environment takes seconds, minutes, or hours. Using mock objects for these parts makes it easier to develop tests, and much easier to run them. So mock objects are a good thing, and skill at using them is worth developing. New development environments for Java and C# make it even easier to build these.
But do functioning components guarantee we have a functioning system?
The second test flight of the Grumman F-14 Tomcat (the huge fighter jet in Top Gun) suffered a hydraulic failure and crashed because of a harmonic relationship between the speed of an engine and the speed of a fuel pump. This combined with the lack of any mechanism to smooth out the pulses caused the resonance that burst the hydraulic lines. All parts (which are actually systems themselves) worked perfectly, but the system failed. The pilots ejected (and survived) 1.3 seconds before the plane crashed.
In software it turns out to be even easier to construct flawed systems out of working components. The most frequent source of the flaws? The presentation layer and interactions with other systems sharing the same database tables, the parts it's most tempting to mock up.
I can't give a blanket prescription for everyone. When I look at the tests that gave us the most bang for the buck in developing control systems for aircraft and spacecraft, they were always the end-to-end tests. I would generally prioritize those above the unit tests. Sure they take longer to rerun. Fortunately we now have continuous integration tools to do ease some of that pain.
This isn't a black and white issue; it turns out there's a continuum between pure unit tests and "system" tests. For example, I describe how to use a unit-testing tool for "system" testing here.
Think about your bathroom floor. Does more crud build up on the tiles, or in the grout between the tiles?
Michael James
Software Process Consultant
http://www.danube.com
| Attachment | Size |
|---|---|
| Mock Objects blog.pdf | 138.25 KB |
I agree that there is a place for mock objects (or Strata tests) in the process, especially if you need to break dependancies, but a recent experience has made me wary of using them extensively.
A major system integration / deployment failed, and one developer held up his hands and said "it's not my problem, my tests passed with the mock objects...". I was unable to respond politely.
I guess, like most things, you've got to use your judgement, but I'll try to ensure we can speed up whole-system tests rather than use mock objects in the future: load a smaller test dataset into the database, use a servers/services hosted on the same box.