Marcin Swierczynski's blog

In the Java world there was always a problem with dependencies. All the jars we had to mange by hand – quite awful. Thankfully there is a Maven project which helps us in requirements management. But what about Python? Is there any way to handle necessary libraries? Of course, there is! It is a tandem of virtualenv and pip.

Virtualenv is a tool to create isolated Python environments. I’m sure you can imagine a situation when one project needs a library in version x, while another one needs the same library but in version y. Then, you can’t install both libraries to your default site-packages directory. Another example: let’s assume that new version of Django has been released and you want to test your application with this version. It’d perfect to create independent environment for the new release, wouldn’t it? So, this is a place for virtualenv.

Pip is simply the Python packages installer. It can be called easy_install successor. Here, you can read pip and easy_install compare. From our point of view, pip has one extremely useful feature. It can save all requirements information to the text file and use it later to download necessary libraries. The example file is listed below.

Django==1.2.1
Fabric==0.9.1
PIL==1.1.7
South==0.7.2
...

OK, but how can we use it together? Here comes how. Let’s assume, we’ve virtualenv installed. Then, just do the following steps.

1. virtualenv --no-site-packages environment_name – this command creates new environment with given name. The no-site-packages option ensures that this environment won’t inherit any libraries previously installed in your system.
2. cd environment_name – virtualenv has created a new directory, so we’ve to go to it
3. Now, we’ve to install all requirements into a new environment. As a bonus, virtualenv contains pip package and puts it into each environment. So, all you’ve to do is bin/pip install package_name. We need to repeat this command for each library our project depends on.
4. It’d be useful to export all the dependencies into to the file to use it in the future. To do that just run bin/pip freeze > requirements_file.txt
5. You can use this file to build your environment later, perhaps on another machine. It’s simple: bin/pip install -r requirements_file.txt
6. It’s almost done. The last thing is to activate our new virtual environment. We need this to run all subsequent commands within its context. Use source bin/activate and notice that your command prompt got a prefix – an environment name. To deactivate the environment, simply write deactivate.

Of course, the possibilities of virtualenv and pip are a lot more sophisticated. For example, you can create your own bootstrap script with virtualenv to setup a whole web application. On the other hand, you can use pip to download dependencies from version control systems directly or even create your own libraries bundle to use it when you’re offline. The use case I’ve described above is just an introduction. But I hope, it is a useful one.

Last week I’ve completed an interesting task within our project: building internal link-shortening system based on persisted objects, not on constant URLs. The main requirement was to get as loosely-coupled design as possible. In the perfect world the “shortenable” classes should know absolutely nothing about link-shortening mechanism. How to get such a flexibility in Django? I’ve used the ContentTypes framework with generic relations and it works pretty well!

First of all, we need to prepare the model class which will keep references to every “shortanable” object and its shortening key which will be used in URLs. The key-related part is quite easy, but what about the former requirement? How it is possible to keep a relation to an object of type we don’t even know? The answer is: ContentTypes framework. It gives us an ability to get an identifier of every class within Django-based application. So, we’ve got all we need now. Just take a look into a small code snippet to see this feature in action:

class ShorteningKey(models.Model):
    """ Contains a key for a generic object used in short link resolving """
    key = models.SlugField(unique=True)
    content_type = models.ForeignKey(ContentType)
    object_id = models.PositiveIntegerField()
    content_object = generic.GenericForeignKey()

Quite neat, isn’t it? Now it is possible to get a shortening key for a given object. We’ll do that with help of the following method.

    @classmethod
    def get_shortening_key_for_instance(cls, instance):
        """ Return ShorteningKey object associated with given model object.
            May throw DoesNotExist exception """
        type = ContentType.objects.get_for_model(instance)
        return ShorteningKey.objects.get(content_type__pk=type.id, object_id=instance.id)

The last thing to do is to generate a unique key for every object. As I’ve mentioned, it was extremely important to keep “shortenable” classes as clean as possible, so hard-coding shortening key generating method within the scope of each relevant class wasn’t a good idea. Instead of this, I’ve used Signals. Now, we have to write key-generation method and associate it with post_save signal. Why do we use the post_save? Because we need persisted instance to use it within ContentTypes framework. The described function is listed below while process of attaching it to the signal will be shown later.

def generate_shortening_key(sender, **kwargs):
    """ Generates shortening key for given object """
    instance = kwargs['instance']
    new_instance_created = kwargs['created']

    if new_instance_created:
        attempt = 0
        while attempt < 100:
            try:
                shortening_key = ShorteningKey(content_object=instance, key=generate_key())
                shortening_key.save()
                break
            except DatabaseError:
                attempt += 1
        else:
            logging.warning("Cannot create link shortening for " + str(instance))

We'll also need a function which will delete unnecessary key after object deletion. It's quite simple and similar so I won't put its code here.

The whole thing is almost ready, we just need to provide a function which is responsible for retrieving shortened URL.

def get_shortened_url(instance):
    """ Returns shortened url to given instance. If there is no shortened url,
        it tries to return full url to object. If there is no one, it returns empty string.
        This function is dynamically attached to any class with link shortening enabled """
    try:
        shortening_key = ShorteningKey.get_shortening_key_for_instance(instance)
        return shortening_key.get_shortened_url()
    except ShorteningKey.DoesNotExist:
        try:
            return instance.get_absolute_url()
        except (TypeError, AttributeError):
            return ''

It will be perfect if the function would be a member of each "shortenable" class, won't it? We will attach it dynamically to each of them in the same loop which is responsible for signals attaching.

LINK_SHORTENING_ENABLED_CLASSES = (Class1,Class2,...)
for cls in LINK_SHORTENING_ENABLED_CLASSES:
    post_save.connect(generate_shortening_key, sender=cls)
    post_delete.connect(delete_shortening_key, sender=cls)
    cls.get_shortened_url = get_shortened_url

That's all! We've got fully featured link-shortening system with loosely-coupled classes. You can easily adjust this example and make it appropriate for your particular needs. For example, you can build a comment system, just like Django team did

Code coverage analysis could be very valuable if you want to do reliable testing. 100% coverage doesn’t give you a guarantee that everything will be fine, but still – it’s extremely useful tool. How can we use it in Django-based application?

First of all, we’ll need an awesome module called coverage.py and written by Ned Batchelder. I recommend to install it on a development machine instead of just attaching it to the project, because it can give us a huge performance boost — almost 600% during my tests.

Our next step could be a test runner concept. It is a quite good approach, because we haven’t to worry about running code coverage analysis — we can just start our tests, as we usually do. I’ve found a great code snippet with such a runner. But what if we don’t want to run analysis every time we test our application? Code coverage can be quite time-consuming while the first rule of TDD says — tests should be as fast as possible. With this thought in mind, I’ve decided to write my own Django custom command. It is based on a snippet I’ve mentioned and its aim is to run tests with code coverage analysis using a separate command. It is also able to generate pretty HTML reports (see example).

The main part of this command is a Command class.

#imports

class Command(test.Command):
    args = '[app_name ...]'
    help = 'Generates code coverage report'

    option_list = test.Command.option_list + (
            make_option('--format', '-f', dest='format', default='txt', help='Change report output format (html or txt, default: txt)'),
            make_option('--directory', '-d', dest='directory', default='.',
                        help='Change html report output directory. Default: current directory'),
        )

    def __init__(self):
        self.cov = coverage.coverage()
        self.coverage_modules = []

        self.cov.use_cache(0)

        try:
            from south.management.commands import patch_for_test_db_setup
            patch_for_test_db_setup()
        except ImportError:
            pass

    def handle(self, *test_labels, **options):
        self.__run_tests_with_coverage_analyse(test_labels, options)

        if test_labels:
            self.__add_selected_applications_to_report(test_labels)
        else:
            self.__add_all_aplications_to_report()

        if self.coverage_modules:
            self.cov.report(self.coverage_modules, show_missing=1)
            if options['format'] == 'html':
                dest_path = os.path.join(options['directory'], 'coverage_report')
                self.__generate_html_report(dest_path)

        self.cov.erase()

    def __run_tests_with_coverage_analyse(self, test_labels, options):
        self.cov.start()
        super(Command, self).handle(*test_labels, **options)
        self.cov.stop()

    def __add_selected_applications_to_report(self, test_labels):
        for label in test_labels:
            # Don't report coverage if you're only running a single
            # test case.
            if '.' not in label:
                app = get_app(label)
                self.coverage_modules.extend(self.__get_coverage_modules(app))

    def __add_all_aplications_to_report(self):
        for app in get_apps():
            self.coverage_modules.extend(self.__get_coverage_modules(app))

    def __generate_html_report(self, dest_dir):
        print "Generating HTML report in %s..." % dest_dir
        self.__delete_directory_content(dest_dir)
        self.cov.html_report(self.coverage_modules, directory=dest_dir)

    def __get_coverage_modules(self, app_module):
        """
        Returns a list of modules to report coverage info for, given an
        application module.
        """
        app_path = app_module.__name__.split('.')[:-1]
        coverage_module = __import__('.'.join(app_path), {}, {}, app_path[-1])

        #ignore external modules/applications
        module_path = coverage_module.__path__[0]
        if 'webapp/apps' not in module_path:
            return []

        return [attr for name, attr in
            getmembers(coverage_module) if ismodule(attr) and name != 'tests']

#some other stuffs

As we can see in lines 7-11, it has two additional options:
• -f or --format which can take a html value to generate an HTML report
• -d or --directory which can take a path to destination directory where an HTML report will be saved to

Either we decide to generate an HTML report or not, we’ll always get a text report in the console. Moreover, the default localization of this report is current directory, so we haven’t to declare a -d parameter at all.

Line 21 is very important too. You’ve to call this function, if you use South migrations in your application. Otherwise your tests will fail.

Obviously, this command works just as classic test command. You can call all tests with ./manage.py code_coverage. You can also run tests for selected modules, ex. ./manage.py code_coverage app1 app2.

I hope it’ll be helpful. I’m also open to any ideas.