Easier A/B Testing with Template Inheritance

There is source code for this post on github if you want to play around with it.

A common problem when writing the UI layer for your web site/application is that you'll need to substitute out chunks of some of your pages temporarily to test whether a new version works better than the existing version.

The first solution people often come to for this problem is to use CSS and if/else blocks based on whether a test is active. That has its uses, but it starts breaking down when you want to make larger or more structural changes. It's also extremely easy to accidentally leave the code in, making it harder to read and understand for future developers.

This post is about a method I developed for making these larger changes with a minimal impact on the primary templates.

I'll be using the Mako templating library in this post since I think it's the best, but something similar should be possible in any library that lets you define where to find templates at runtime and override blocks.

The setup

In most modern templating languages there is the concept of a block. If you have a "base" template which defines blocks you can have another template inherit from the base and override just the blocks it cares about.

For the rest of this entry we'll work with a tumblr-like blogging service with multiple media types for posts. The templates are layed out like this in the project:

templates/  
    base.html
    post.html
    about.html
    posts/
        text.html
        video.html
        ...
    ...

In Mako we'll find templates using this code TemplateLookup(directories=['templates']) to allow us to find templates to render. Your templating system probably has something similar. The following are the bodies of the templates we're going to be using. You can find them and a script to play with the system in a github repo here. You'll probably want to look at them later, but feel free to skip past them for now.

## templates/base.html
<html>  
<body>  
    <div class="header">
        <%block name="header">DEFAULT HEADER</%block>
    </div>
    ${next.body()}
    <div class="footer">
        <%block name="footer">DEFAULT FOOTER</%block>
    </div>
</body>  
</html>  
## templates/post.html
<%inherit file="/base.html"/>  
<%block name="header"><h1>${title}</h1></%block>  
<%block name="footer">  
    <div class="comments" data-comment-id="${comment_thread_id}"></div>
</%block>  
<%block name="post_body">  
<div class="post-wrapper">  
    ${next.body()}
</div>  
</%block>  
## templates/posts/text.html
<%inherit file="/post.html"/>

<%block name="text_post_body">  
<div class="text-post-body">  
    % for p in text_paragraphs:
        ${p['text']}
    % endfor
</div>  
</%block>  

This is great. We can make all kinds of pages and have them share a common header, footer, and navigation. The post page can handle the chrome around posts but still allow the the media types to define the html for how they are rendered. It can get much more complicated with a real website, but the main thing to note is that there is a three file inheritance chain: text -> post -> base.

Making changes

Let's say that we want to try out a new type of text post body that we think users will like more. All we have to do to make that change is create text_new.html and select that to render instead of text.html. To do an A/B test we just do an if/else with our testing framework when choosing what to render. We can even have text_new.html inherit from text.html if we only want to replace a small part of the page.

This gets more complicated if we want to update a template that isn't the last one in an inheritance chain. Let's next say that we want to change the design around our post types next in an A/B test using a new post.html. Now we run into a problem. We could select the template to use to render a text post, we can't choose at runtime what our posts inherit from. Mako–and I believe this is true generally–doesn't allow you put an if/else around an <%inherit> tag, so you're stuck. The obvious but gross solution here is to create a parallel set of pages with the different inheritance chain text -> post_new -> base.

I discovered I could handle this problem by using the template lookup mechanism in a clever way. In the initial setup we told mako to look for templates in the templates directory. We're going to move our files into a configuration which is a bit more useful:

templates/  
  default/
    base.html
    post.html
    about.html
    posts/
      text.html
      video.html
      ...
    ...
  post_chrome_test/
    post.html
  text_post_test/
    posts/
      text.html

Our main templates have moved under default and the templates for two tests we're interested in are in two other directories. When we set up the template lookup right before we render we'll use something which adds in any tests we're running:

dirs = ['templates']  
dirs.extend(get_active_ab_test_names())  
dirs.append('templates/default')  
lookup = TemplateLookup(directories=dirs)  

What this means is that when we look up a template in python code, using an <%include> tag, or with a <%inherit> tag we will check each location in order. All that's left to make this work is defining the new_post_chrome_test version of post.html

## templates/post_chrome_test/post.html
<%inherit file="/default/post.html"/>  
<!-- post chrome test. Add "by $user" to the title and remove  
     comments from the footer -->
<%block name="header"><h1>${title} by ${user}</h1></%block>  
<%block name="footer"></%block>  

How it works

The key to making all of this work is that the template above inherits from default/post.html rather than just post.html. The logic of how this works is a bit tricky the first time you see it, so I'll walk through it.

  1. A request comes in for a text post.
  2. We set up the template lookup based on the A/B tests of the current user, it contains ['templates', 'templates/post_chrome_test', 'templates/default']. Note that it does not contain templates/text_post_test since that test is not active for the current user.
  3. We look up posts/text.html from each location in order and we find it under templates/default. When we find it, we try to render it.
  4. This template inherits from post.html. When the system looks that template up it finds it under templates/post_chrome_test (because that occurs before the default in the list).
  5. The postchrometest version of post.html inherits from default/post.html. When the system looks that up it finds it under just templates (which we put first so that "absolute" templates like this would work).
  6. The default post inherits from base.html. In this case we find that under templates/default, but it could have ALSO been under a test if the file was defined under one.

Using this scheme any template at any level of the inheritance chain can be overridden once. As long as they aren't overriding the same file you can have any number of tests running at once. Cleanup of old tests is also very easy. For failed tests you delete the template directory for the test and everything is gone. For successful tests you manually merge the changes from the test into templates/default. There's no chance that the remnants of old tests are left around to cause trouble in the future.

If you're using something like this, let me know. I haven't done UI development in a few years but I'd be very interested to hear about similar systems people have put in place.