synchronicity

Tests Happen: How Metaprogramming Avoids The Drudgery Of Testing

01/24/2012@14:27:30

Wikipedia defines the mythical man-beast known as metaprogramming thusly:

Metaprogramming is the writing of computer programs that write or manipulate other programs (or themselves) as their data, or that do part of the work at compile time that would otherwise be done at runtime.

Everyone has heard about it, but few understand it. What makes it worse is that, like calculus, it's a very useful technique whose relationship to real-world, everyday usage has been cloudy, at best. It's been the exclusive domain of people who solve hard problems for so long, that many problems that benefit from its application have required mountains of code to solve without it, and that's a shame.

This guide is not going to define everything there is to know about metaprogramming. What it will do is give a practical, everyday example of how to apply metaprogramming to solve part of a common problem - testing - in a way that is compelling for a variety of reasons:

1. it produces a larger body of tests from a smaller body of test code
2. it produces better test coverage from less test code
3. it allows user interface tests to scale linearly with database models
4. it requires virtually zero maintenance once its implemented
5. it is based entirely on the standard unittest module.

The example I'm about to share is based on python, but the techniques involved are applicable to just about any programming language that supports reflection. Again, from wikipedia:

Reflection is the process by which a computer program can observe (do type introspection) and modify its own structure and behavior at runtime.

The project I'm using to demonstrate this technique uses python, flask, and an Object Relational Mapper called PeeWee ORM. The admin user interface in the example below comes from flask-peewee. The example is just under 100 lines of code, but less than 30 of those actually apply to the metaprogramming technique we're interested in, namely lines 51-82 toward the end of the test program.

The models module defines some simple database models for our app. For instance:

class HttpMethod(db.Model):
    verb = CharField()
    active = BooleanField(default=True)
    created = DateTimeField(default=datetime.now)
    updated = DateTimeField(null=True)
    deleted = DateTimeField(null=True)

These models have administrative interface built up in another module called admin. The model is derived from peewee and the admin is derived from flask-peewee. The admin module simply takes a model and registers it in the admin interface:

admin.register(HttpMethod)

This, in turn gives us an admin interface for the model in the web browser:

And that admin interface is exactly what we need to test. I discovered the hard way that doing bad things in my code would break the admin interface, so I set about designing a way to avoid having to worry that I had broken the interface. The technique here is specific to these tools, but with a little munging, can be used across a much broader array of problems - any problem in which the user interface to be tested is predictable. Note: these tests only scratch the surface. They merely verify that the page loads returning a status code of 200. They could go way beyond that, but I've left out a lot of further detail so as not to obfuscate the core message.

Back to our code. Lines 51-54 define a test case class for our Admin UI but note that there's only a single method defined which, according to the specs our unittest class expects, is not really a test. That is, it doesn't begin with "test". Without the remaining code, this class by itself would run zero tests.

The magic really happens in the two global methods that follow. In line 57, add_admintest takes a class and a uri as parameters and attaches a test method for the uri parented by the class parameter based on the base_test method.

If you dissect the list comprehension at line 77, you'll see that we use the inspect modules getmembers function to load all of the classes that our models define. The problem with this is that the models include some inherited members that aren't actually exposed by the admin UI, so we have to remove those classes that we don't care about before buildling our tests from the model classes. This is done via the found_ignore function at line 67 which takes a string representing the class name and compares it to a list of classes we want to ignore and tells us whether to ignore it.

Once our list comprehension has generated a list of URI candidates for our admin tests, the loop at line 79-81 calls add_admintestcase and attaches a member function to the AdminTestCase class. We then call add_admintestcase once at line 82 for the parent-level admin interface.

Finally, at line 85 the unittest module enumerates all of the TestCase classes with methods ending in "test" and runs all of these methods on our behalf when we execute:

python metatests.py

Using python's coverage module is just as easy:

coverage run metatests.py

and the coverage report reveals that we've covered 60% of our codebase with less than 100 lines of test code. I think that's a good ROI from a not-so-brainbending investment in metaprogramming test code.

(env)[watson@watson-thinkpad metatests (master)]$ coverage report
Name          Stmts   Miss  Cover
---------------------------------
admin            59     30    49%
api              15      0   100%
app              10      2    80%
app_wan          82     63    23%
auth              6      0   100%
db_setup         25      1    96%
main             31      2    94%
metatests        53      3    94%
models          152     21    86%
send_thx         22     15    32%
test_config       7      0   100%
views           148    106    28%
---------------------------------
TOTAL           610    243    60%

And here's the full listing for metatests.py:

import unittest
import tempfile
import test_config
TESTING = True
from main import app
import flask
import main
import base64
import json
import inspect
import models

class BaseTestCase(unittest.TestCase):
    """Base class from which all  test classes are derived."""
    uri = None

    def setUp(self):
        """Set a tempfile sqlite3 database and a few flags."""
        self.app = app.test_client()
        main.create_tables()
        main.create_data()

    def tearDown(self):
        pass

    def assert_response(self, response, status_code=200, data=''):
        """Wrap asserts and make sure that colons have been escaped."""
        self.assertEqual(response.status_code, status_code)
        self.assertIn(data, response.data)

    def login(self, username, password):
        return self.app.post('/private/login/', data=dict(
            username=username,
            password=password
        ), follow_redirects=True)

    def logout(self):
        return self.app.get('/private/logout/', follow_redirects=True)

    def open_with_auth(self, url, method, username, password):
        return self.app.open(url,
            method=method,
            headers={
                'Authorization': 'Basic ' + base64.b64encode(username + \
                ":" + password)
            }
        )

class AdminTestCase(BaseTestCase):
    def base_test(self, uri):
        response = self.app.get(uri, follow_redirects=True)
        self.assert_response(response, 200)

def add_admintest(cls, uri):
    name = uri.replace('/', '')
    def inneradmintest(self):
        self.base_test(uri)
    inneradmintest.__name__ = 'test_%s' % name
    inneradmintest.__doc__ = 'Test that %s returns 200.' % uri
    setattr(cls, inneradmintest.__name__, inneradmintest)
    return inneradmintest.__name__

def found_ignore(st):
    '''Ignore the inherited classes that don't appear in the admin interface.'''
    li = ('Field', 'Database', 'Improperly', 'Q',
          'Arbitrary', 'Base', 'Subscriber', 'Model')
    for l in li:
        if st.find(l) > -1:
            return False
    return True

# for each class in models, create an admin interface test
uris = ['/admin/' + member[0].lower() for member in
        inspect.getmembers(models, inspect.isclass) if found_ignore(member[0])]
for uri in uris:
    add_admintest(AdminTestCase, uri+'/add/')
    add_admintest(AdminTestCase, uri+'/export/')
add_admintest(AdminTestCase, '/admin')

if __name__ == '__main__':
    unittest.main()