Using Travis CI for testing Django projects

A couple of weeks months1 ago in my post about using tox with Django projects, I mentioned using Travis CI as well as tox for testing.

There’s plenty of documentation out there for Python modules and Django applications, but not so much guidance for testing a complete Django project. I can only speculate that this is because testing projects normally involves more moving parts (e.g., databases) as opposed to applications which are supposed to be self-contained.

A good example here is the cookiecutter templates of Daniel Greenfeld – one of the authors of Two Scoops of Django. His djangopackage template contains a .travis.yml file, yet his django[project] template doesn’t. Since many consider these templates best practices, and Two Scoops as a Django “bible”, perhaps I’m wrong to want to use CI on a Django project?

Well (naturally) I don’t think I am, so here is how to do it.

Travis CI has a whole bunch of services you can use in your tests. The obvious ones are there e.g., PostgreSQL, MySQL, and Memcached. For a more complete system, there’s also Redis, RabbitMQ, and ElasticSearch. Using these you can build a pretty complete set of integration tests using the same components you will in a production environment.

For the purposes of testing capomastro2, we only need a PostgreSQL database 3. The first step is to say we want PostgreSQL available during tests:

services:
– postgresql

Now we need to create the database, using the postgres user provided by Travis CI:

  psql -c 'create database capomastro;' -U postgres

The next part is to configure our Django project to use this database. Fortunately our project already provides a sample local_settings.py that is configured for connecting to PostgreSQL on localhost without a password, so all we need to do is modify this file to use the same postgres user:

  cp capomastro/local_settings.py.example capomastro/local_settings.py
  sed -i -e 's/getpass.getuser()/"postgres"/g' capomastro/local_settings.py

Finally we can call the Django syncdb (and migrate because like everyone else, we use South) to setup our database:

  python manage.py syncdb --migrate --noinput

All of this is done in the before_script hook in Travis CI:

before_script:
– cp capomastro/local_settings.py.example capomastro/local_settings.py
– psql -c 'create database capomastro;' -U postgres
– sed -i -e 's/getpass.getuser()/"postgres"/g' capomastro/local_settings.py
– python manage.py syncdb –migrate –noinput

We can now execute the full test suite for the project, using the same database we use in development and production!

For reference, here is the complete .travis.yml file:

language: python
services:
– postgresql
python:
– 2.7
install:
– pip install -r dev-requirements.txt
before_script:
– cp capomastro/local_settings.py.example capomastro/local_settings.py
– psql -c 'create database capomastro;' -U postgres
– sed -i -e 's/getpass.getuser()/"postgres"/g' capomastro/local_settings.py
– python manage.py syncdb –migrate –noinput
script:
– python manage.py test


  1. Wow, this post has been sitting in drafts for quite a while! 
  2. master builder“ 
  3. A lot of people deploy against PostgreSQL, but develop and test against SQLite for speed and convenience. This will eventually bite them. 

Using tox with Django projects

Today I was adding tox and Travis-CI support to a Django project, and I ran into a problem: our project doesn’t have a setup.py. Of course I could have added one, but since by convention we don’t package our Django projects (Django applications are a different story) – instead we use virtualenv and pip requirements files – I wanted to see if I could make tox work without changing our project.

Turns out it is quite easy: just add the following three directives to your tox.ini.

In your [tox] section tell tox not to run setup.py:

skipsdist = True

In your [testenv] section make tox install your requirements (see here for more details):

deps = -r{toxinidir}/dev-requirements.txt

Finally, also in your [testenv] section, tell tox how to run your tests:

commands = python manage.py test

Now you can run tox, and your tests should run!

For reference, here is a the complete (albeit minimal) tox.ini file I used:

[tox]
envlist = py27
skipsdist = True

[testenv]
deps = -r{toxinidir}/dev-requirements.txt
setenv =
    PYTHONPATH = {toxinidir}:{toxinidir}
commands = python manage.py test

django-getenv – use environment variables in your Django settings

So I made a thing for using environment variables within your Django settings: django-getenv.

Although the code itself is trivial (and, to be honest, not coupled with Django in any way), because I was re-using the same code in multiple projects I decided – if only to make my life easier – to made it into a standalone module and package.

Enough of the what, let us get onto the why.

I’m a big fan of The Twelve-Factor App, and although I can’t or don’t always follow its tenets in every app I write, I do my best.

One of the tenets is “Store config in the environment”:

The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.

Although Django has a range of configuration options, it does not lend itself well to using environment variables out of the box.

Jacob Kaplan-Moss has addressed part of this with django-dotenv which lets you use a Foreman-style .env file to populate your environment with the settings contained within that file.

django-getenv is another piece of the puzzle, helping you use those environment variables in your Django project settings. As well as simplifying accessing these variables, it will also convert boolean, integer and float values to their native Python types.

The module is BSD (3-clause) licensed, available on Pypi, and the project is hosted on Github.

Using Django flatpages content in other (class-based) views

The flatpages application that ships with Django is both (intentionally) simple and useful. In previous projects I’ve found myself both extending the models (to add data like meta tags) and duplicating the whole application (when I wanted to add a whole bunch of extra details, multiple content sections, or a WYSIWYG editor).

In a recent project I had the need to editable (by administrators) content in other views, and I turned to flatpages again for my solution.

What I did was to create a class-based view mixin to find the flatpage for a given URL (defaulting to the one defined in request.path), and include the resulting object in the context (once the title and content had been marked safe of course):

from django.contrib.flatpages.models import FlatPage


class FlatPageMixin(object):
    """
    Retrieves the FlatPage object for the specified url, and includes it in the
    context.

    If no url is specified, request.path is used.
    """
    url = None

    def get_context_data(self, **kwargs):
        if not self.url:
            self.url = self.request.path

        context = super(FlatPageMixin, self).get_context_data(**kwargsG)
        try:
            flatpage = FlatPage.objects.get(url=self.url)
            flatpage.title = mark_safe(flatpage.title)
            flatpage.content = mark_safe(flatpage.content)
            context["flatpage"] = flatpage
        except FlatPage.DoesNotExist:
            context["flatpage"] = None

        return context

Then you just need to ensure a flatpage exists for the expected (or specified) URL, add the mixin to your class-based view(s) and use {{ flatpage.title }} and {{ flatpage.content }} in the template(s).