The Developer Day | Staying Curious

CAT | Testing



PHPUnit email integration testing using Sendmail

One of the problems when doing functional or integration testing is testing that emails are being sent out with a correct header and body. One such scenario could be a controller action which sends a password reset confirmation email and redirects to another action.

A common way to solve such a problem is to configure the local MTA to store the test emails on the file system. The following shows how this could be done using sendmail. First create a sendmail alias by editing a file located at /etc/mail/aliases and adding a line bellow other aliases:

test-mail: “| cat > /tmp/test-mail”

This tells sendmail that all incoming emails to test-mail will be written (not appended) to /tmp/test-mail. Sendmail needs to be restarted for the changes to take effect.

sudo /etc/init.d/sendmail restart

Depending on the situation it may be necessary to add the user who is going to be reading emails (for example apache) to the mail group.

sudo /usr/sbin/usermod -G mail apache

Now using PHP it should be possible to do this:

$ok = mail('test-mail', 'Hello world!', 'I am an email.');
echo file_get_contents('/tmp/test-mail');

Further PHPUnit could be extended to add the following method to the base test case class:

public function assertEmail($attributes, $emailFilePath, 
$message = '', $delta = 0, $maxDepth = 10, 
$canonicalizeEol = FALSE, $ignoreCase = FALSE)
    $mailParser = new Company_Product_MailParser;
    $mailData = $mailParser->parseFile($emailFilePath);
    foreach ($attributes as $attribute => $value) {
        $constraint = new PHPUnit_Framework_Constraint_IsEqual(
            $mailData[$attribute], $delta, $maxDepth, $canonicalizeEol, $ignoreCase
        $this->_test->assertThat($value, $constraint, $message);
    if (is_file($emailFilePath) && is_writable($emailFilePath)) {

The mail parser class name explains itself:

class Company_Product_MailParser
    public function parseFile($mailFilePath)
        $emailBody = file_get_contents($mailFilePath);
        $attributes = array(
            'to' => '',
            'from' => '',
            'date' => '',
            'subject' => '',
            'body' => ''
        foreach (array_keys($attributes) as $attribute) {
            if($attribute  == 'body') {
                if (preg_match("/\n\n(.*)/", $emailBody, $matches, PREG_OFFSET_CAPTURE)) {
                    $offset = $matches[1][1];
                    $attributes[$attribute] = quoted_printable_decode(substr($emailBody, $offset));
            } else {
                if (preg_match("/" . ucfirst($attribute) . ": (.*)\n/", $emailBody, $matches)) {
                    $attributes[$attribute] = $matches[1];
        return $attributes;

Important notice. Sendmail may not immediately send the email and it may take a few seconds for the file to appear. It may require you to add a sleep for a few seconds before the email file appears. If you find a way how it is possible to make sendmail send an email immediately please let me know.




Avoiding Brittle Tests / Testing Output

While unit tests have benefits they can also cause trouble. Having tests to catch software bugs is great but having tests that break whenever the application is at least slightly changed might not be very pleasant. The latter effect is called brittle tests. It may work well for applications which change rarely but may be counterproductive for applications that change rapidly. Test brittleness can be caused by a variety of implementation details. This post aims to describe few of these details and explain ways how brittle tests can be avoided.

Deciding how detailed the tests should be

It’s important to have an at least general idea what tests should test and what should be left untested. Imagine having to functional test a web application UI displaying a form made of various input fields populated with values coming from the database. Quite a few things could be tested. Are all the values displayed? Are all radios, check boxes, drop-downs properly selected? Are validation messages displayed and are they correct? Are all labels displayed and correct? Are attached javascript events working? Can the form be submitted and is the data passed to the underlying layer? Is the confirmation message displayed?

The more things there are to test more likely that the tests will break not because of a bug but of a minor change. It’s important to pick only the important battles to fight. Even though it’s possible to test a lot of things it may not be practical to do so. It would certainly be possible to run a spelling checker on every displayed word but if it’s not critical to the application it may not be worthwhile to do so. For example testing javascript integration requires use of Selenium. To work with continuous building it would require a Selenium RC server to run all the browsers. Tests recorded by a selenium recorder may be brittle to a slightest HTML structure change unless designed very carefully. While selenium would provide the ultimate functional testing power it might be overkill for a simple web application. Decide what is critical to your application, which things are more likely to break than others and test those things only. Adapt to reoccurring software problems by adding additional tests.

Testing output not implementation

When developing unit tests the most effective way to test is by testing the output of method calls instead of testing the internal implementation. For example testing a simple multiplication function which multiplies a and b is straightforward. More sophisticated units which rely on other units require use of mocks. If possible it’s best to avoid testing that a mock was used or how many times a mock was called and what kind of data it was passed. Otherwise the test is tightly hooked to the internal implementation and is more likely to break when it changes. It comes to the first principle deciding how detailed a test should be. If you are fairly comfortable that the code is less likely to change or break or it’s less critical, hooking deep into the mocks might be avoided. Imagine having to test the following piece of code:

class Notifier
    public function __construct(Zend_Mail $mailer)
        $this->_mailer = $mailer;
    public function notify()
        $this->_mailer->setBodyText('This is the text of the mail.');
        $this->_mailer->setFrom('[email protected]', 'Some Sender');
        $this->_mailer->addTo('[email protected]', 'Some Recipient');
        return $this->_mailer->send();

In this case the mock is the _mailer. All it’s method calls could be mocked and tested against that they are called only once and are passed the correct data. In turn that would make the test more likely to break whenever this function is changed. Instead it may be enough to test that function notify() returns true whenever send() returns true. On other hand such a test might seem not sufficient enough and more hooks may be required. For example adding a test for addTo() function call. Or if the functionality is extremely critical an integration test could be created to test that an actual message was sent to the mail server with the correct header and body.

Final Words

In the end it’s a challenge of trying to find the the acceptable balance between testing application functionality and avoiding having too many brittle tests. Try to identify what’s important to your application, and test those things only, prefer testing output of method calls over hooking deeply into implementation. Let your tests work for you and not against you.

, , , Hide



Benefits of Testing and Types of Testing

Software development produces products as any other industry. It is an interesting issue that products produced by the software development industry are not always as well tested as in other industries. One of the main reasons is due to the fact that in most companies software is manually tested by the same developers who built it.

Developers are not necessarily good testers. Good software developers while sharing some traits are different from good testers.  A good developer may be excellent at software design but won’t notice spelling mistakes. Developers tend to take the testing path that usually works while cutting corners on special cases. It is also known as happy testing.

Most reasons why software is tested manually boil down to:

  • Not knowing how
  • Not having the time
  • Maintaining legacy code

Hardly any of these reasons are valid excuses. Truth be told automated testing is not easy. It is a skill that has to be learned and doesn’t come naturally. It’s a skill that takes years to master. It may seem that automated tests take too much time and at first it will. Due to the same fact that it’s a skill and takes time to master. In some way it is like skiing. Before going down a steep mountain racing the wind one has to learn how to otherwise it’s an unpleasant struggle climbing down with your skis across the shoulder. Automated testing won’t do miracles but when done properly it may increase productivity by 10 - 20%.

There are many types of software testing with their own benefits and downsides. Bellow are the three most important and common types of testing.

Unit Testing

Unit testing is about testing the smallest individual parts of an application. For example instead of testing the whole engine of a car all individual pieces are tested separately as units. All dependencies are replaced with stubs or mocks or fakes. Unit tests are meant to be lightning fast and execute thousands of tests in a few seconds therefore they shouldn’t connect to the database, web services or send emails. Unit tests are most effective when executed very frequently during development to inform the developer of any broken parts allowing to fix the problem immediately without losing line of thought.

Integration Testing

Integration tests ensure that all of the application’s parts work correctly when they are assembled in a simulated production environment. An example of an integration test can be a batch job executing against a database with test data. Integration tests are a lot slower than unit tests and usually work best integrated with a continuous build system. While integration tests are not good at identifying broken parts they are excellent at testing if an overall group of components is correctly wired together and produces the desired outcome.

Acceptance Testing

Acceptance testing also known as functional testing or QA testing involves testing a complete system that is usually identical to the user’s anticipated system. Acceptance tests can be automated or may be carried out by a QA team. In agile software development terms an acceptance test tests a business story. For example a simple story may be “As a manager I am able to list invoices, filter them by name and date and approve or reject them” An acceptance test would test that scenarios mentioned in the story can be done on a completed system. Automated web application development acceptance tests can be carried out by selenium that simulates recorded browser actions or by asserting HTML structures. Acceptance tests shine in proving if the user story scenarios can be completed but are usually not very good at pinpointing the nature of a problem. They tend to be slow and complicated to set up. Even testing a relatively simple back office application would require to set up an up to date test database, provide datasets for every possible scenario, provide a way to test sending out emails and their contents.

Software development testing is a vast topic that deserves more attention from the the ever growing industry. Automated software testing is often times disregarded as boring, time consuming or ineffective. Mainly because of lack of knowledge and skill to make it work for you and not against you.

, , , , Hide

Find it!

Theme Design by