The Developer Day | Staying Curious



Avoiding Brittle Tests / Testing Output

While unit tests have benefits they can also cause trouble. Having tests to catch software bugs is great but having tests that break whenever the application is at least slightly changed might not be very pleasant. The latter effect is called brittle tests. It may work well for applications which change rarely but may be counterproductive for applications that change rapidly. Test brittleness can be caused by a variety of implementation details. This post aims to describe few of these details and explain ways how brittle tests can be avoided.

Deciding how detailed the tests should be

It’s important to have an at least general idea what tests should test and what should be left untested. Imagine having to functional test a web application UI displaying a form made of various input fields populated with values coming from the database. Quite a few things could be tested. Are all the values displayed? Are all radios, check boxes, drop-downs properly selected? Are validation messages displayed and are they correct? Are all labels displayed and correct? Are attached javascript events working? Can the form be submitted and is the data passed to the underlying layer? Is the confirmation message displayed?

The more things there are to test more likely that the tests will break not because of a bug but of a minor change. It’s important to pick only the important battles to fight. Even though it’s possible to test a lot of things it may not be practical to do so. It would certainly be possible to run a spelling checker on every displayed word but if it’s not critical to the application it may not be worthwhile to do so. For example testing javascript integration requires use of Selenium. To work with continuous building it would require a Selenium RC server to run all the browsers. Tests recorded by a selenium recorder may be brittle to a slightest HTML structure change unless designed very carefully. While selenium would provide the ultimate functional testing power it might be overkill for a simple web application. Decide what is critical to your application, which things are more likely to break than others and test those things only. Adapt to reoccurring software problems by adding additional tests.

Testing output not implementation

When developing unit tests the most effective way to test is by testing the output of method calls instead of testing the internal implementation. For example testing a simple multiplication function which multiplies a and b is straightforward. More sophisticated units which rely on other units require use of mocks. If possible it’s best to avoid testing that a mock was used or how many times a mock was called and what kind of data it was passed. Otherwise the test is tightly hooked to the internal implementation and is more likely to break when it changes. It comes to the first principle deciding how detailed a test should be. If you are fairly comfortable that the code is less likely to change or break or it’s less critical, hooking deep into the mocks might be avoided. Imagine having to test the following piece of code:

class Notifier
    public function __construct(Zend_Mail $mailer)
        $this->_mailer = $mailer;
    public function notify()
        $this->_mailer->setBodyText('This is the text of the mail.');
        $this->_mailer->setFrom('[email protected]', 'Some Sender');
        $this->_mailer->addTo('[email protected]', 'Some Recipient');
        return $this->_mailer->send();

In this case the mock is the _mailer. All it’s method calls could be mocked and tested against that they are called only once and are passed the correct data. In turn that would make the test more likely to break whenever this function is changed. Instead it may be enough to test that function notify() returns true whenever send() returns true. On other hand such a test might seem not sufficient enough and more hooks may be required. For example adding a test for addTo() function call. Or if the functionality is extremely critical an integration test could be created to test that an actual message was sent to the mail server with the correct header and body.

Final Words

In the end it’s a challenge of trying to find the the acceptable balance between testing application functionality and avoiding having too many brittle tests. Try to identify what’s important to your application, and test those things only, prefer testing output of method calls over hooking deeply into implementation. Let your tests work for you and not against you.

RSS Feed

1 Comment for Avoiding Brittle Tests / Testing Output

Dave | April 7, 2011 at 11:03 PM

This is a really good point and I think it’s one of the greatest challenges facing the TDD approach. A lot of classes don’t work like black boxes that take some input and return some output. They have side effects on other objects and entities in the system. I don’t know of way to test a method like the one you wrote above that reasonably exercises the class without requiring tests to have WAY to much implementation knowledge and thus become brittle. It’s likely that just checking whether or not send() was called isn’t enough to call that class sufficiently unit tested.

This seems to be one of those problems that nobody wants to even acknowledge much less try solve.

Leave a comment!



Find it!

Theme Design by