Ubuntu Cloud Images now available for Google Compute Engine

I'm really pleased to announce that Ubuntu is now a supported platform on Google Compute Engine. There are currently images available for 12.04 LTS, 14.04 LTS and 14.10. You can read more about the launch and how to get started in utlemming's blog post, Google's announcement or the Google Compute Engine documentation.

Setting up a Stellar trust line

Recently Stellar, a decentralized protocol for sending and receiving money in any pair of currencies, was announced. I decided to have a play around (because why not?). A core concept to Stellar is that of a gateway; an individual or organisation that takes a deposit (in 'real' currency) and credits you on the network.

Coinex are a New Zealand-based gateway, and are the first gateway for Stellar; they put up a challenge to be the first person to use them as a gateway (which I won!). To do this, you have to say that you trust that gateway to hold a particular amount of currency; in this case I wanted to publish the fact that I trust Coinex to hold up to $20 NZD for me.

To do this, you'll need to interact directly with the API (Stellar clients are still playing catch-up), and submit the following JSON:

{
    "method": "submit",
    "params": [
        {
            "secret": "<SECRET>",
            "tx_json": {
                "TransactionType": "TrustSet",
                "Account": "<your Stellar address>",
                "LimitAmount": {
                    "value": "<the amount you trust the issuer to hold>",
                    "currency": "<the currency the issuer will hold>",
                    "issuer": "<the issuer's Stellar address>"
                }
            }
        }
    ]
}

You can find <SECRET> on your Stellar settings page; everything else will obviously depend on what you're trying to do.

WARNING: POSTing your secret to live.stellar.org is potentially insecure; you would be better off signing requests locally; a blog post on this may be forthcoming.

You'll then need to POST this to the API:

curl -X POST https://live.stellar.org:9002 -d'<JSON>'

There's a bug in Stellar that means that newlines and/or indentation cause problems, so you'll need to make the above JSON just a single line and slot it in to the above command line.

Once you run that, you're done!

Push It!

Earlier today, Alex Couper presented me with a challenge on a captainhook issue:

I want to play "push it" by salt n pepa on every push.

Challenge accepted.

If you are using a sufficiently recent version of git (i.e. one which supports the pre-push hook) and have youtube-dl, mplayer and curl installed, then dropping this in to .git/hooks/pre-push for a repo will play Push It (in its entirety) on every push:

#!/bin/sh
curl -s "$(youtube-dl -g http://www.youtube.com/watch\?v\=vCadcBR95oU)" | mplayer -really-quiet -cache 8192 -vo null - > /dev/null 2>&1 &

PyCon UK 2013 Notes

Over the weekend, I was at PyCon UK 2013, which was fantastic! I took notes of some of the talks I went to, and the links to my Evernote notes are below:

• Tim Golden: Ten Things Coming in Python 3.4
• David Winterbottom: A Smorgasbord of Testing
• The Test People: Breaking Things for Money
• Rob Collins: Test-Driven Madness

For more PyCon UK follow-up goodness, check out the conference wiki.

Fixing nose.tools pylint errors

What is nose.tools?

If you've engaged in any test-driven Python development, you've probably considered using nose as your test runner. One of my favourite parts of nose (among its many excellent parts) is the nose.tools module. This not only contains a number of helpful tools (documented here), but also all of the assertion methods on unittest.TestCase converted to PEP-8 compatible functions (e.g. self.assertEqual becomes assert_equal).

These assertion functions are useful for a few reasons: readability, line length, PEP-8 compliance. Most importantly, though, is the fact that nose allows you to write test functions and test classes which don't subclass unittest.TestCase. The only way to access these assertions, therefore, is through nose.tools.

The assertion functions in nose.tools are generated at runtime from unittest.TestCase and put in to the nose.tools namespace. This has a couple of consequences: firstly, any new assertions in the standard library will immediately appear in nose.tools (a word of warning: as nose doesn't implement the assertions, you need to pay attention to stdlib unittest assertion changes). Secondly, this means that they don't appear anywhere in the source code if you go looking for them. This will become relevant shortly.

What is pylint?

If you've engaged in any team Python development, you've probably considered pylint to help you maintain code quality. pylint uses static analysis to find problems in your code. Static analysis just means that it looks at your code as text, rather than executing it (which would be dynamic analysis). The problems it finds range from bad syntax to poor style; it can even work out when you've tried to import a non-existent symbol from another module.

What's the problem?

This last feature is really useful most of the time; it has saved me from pushing broken code innumerable times. However, it does cause one major headache: because the nose.tools source code doesn't contain the PEP-8 assertion functions (remember, they're dynamically generated!), pylint's static analysis can't find them; every single time you import one of them you get a pylint error (E0611, to be precise). If you're using nose as part of your continuous integration to check your pylint errors, this can be a major source of pain. Luckily, pylint has a solution: disabling checks.

Fixing with disabled checks

pylint allows you to disable checks in a couple of ways: you can disable them globally (i.e. none of these warnings should ever be displayed), or based on scope (i.e. this module, class, function or line should never have warnings displayed). So, for example, an import line could look like:

from nose.tools import assert_equal  # pylint: disable=E0611

Problem solved! Your pylint run will no longer flag this line up as problematic. There are a few issues with disabling checks, though. Firstly, you have to remember to do it. Secondly, it crufts up the screen when you're trying to read, and gives you more information that you need to process. Finally, and often most crucially, you can accidentally change the scope of your disabling. Consider the following lines at the top of a module:

# pylint: disable=E0611
from nose.tools import assert_equal, assert_items_equal, assert_greater

This does solves the warning (and avoids violating PEP-8's line limit), but it has an unintentional consequence: E0611 is now disabled for the entire file. pylint will no longer catch this obvious error!

# pylint: disable=E0611
from nose.tools import assert_equal, assert_items_equal, assert_greater
from mcok import Mock

Fixing with a pylint plugin

Luckily, there is a better way of solving this problem than disabling individual checks everywhere. pylint provides a plugin interface so that people can implement their own checks and plugins. Using this, we can also modify the way that the static analysis looks at the nose.tools module. In this pylint plugin, I have implemented a function which transforms the abstract syntax tree which pylint uses when looking at nose.tools to inject all of the functions which are generated at runtime:

from nose import tools

function_template = """
def {}(*args, **kwargs):
    pass
"""

def nose_transform(module):
    funcs = ''.join(function_template.format(func_name)
                    for func_name in tools.__all__)
    fake = ASTNGBuilder(MANAGER).string_build(funcs)

    for func_name in tools.__all__:
        if func_name not in module.locals:
            module.locals[func_name] = fake[func_name]

Just follow the installation instructions in the gist, and all of your pylint nose.tools problems will disappear!

BrightBox munin plugin

We're signed up with EE for broadband, and they provide us with a BrightBox router. I wanted to see what connection speeds we were getting over time, so I wrote a munin plugin which would graph it for me.

You can find the source, installation instructions and sample output on GitHub.

Zero-Hour Contracts: Letter to my MP

Earlier this evening, I read the Guardian article about Sports Direct's zero-hour policy for their part-time staff. After my wife pointed out that a bitchy outburst on Twitter wasn't going to change anything, I decided to write a letter to my MP, Kate Hoey, using the excellent WriteToThem service.

If this issue (or any other) winds you up as much as it does me, I would recommend using some of your self-righteousness to write a letter to your MP. How I did so is enclosed below.

Dear Kate Hoey,

You may recently have read in The Guardian about Sports Direct's policy of keeping all of their part-time employees on zero-hour contracts[1].                                                                                      
                                                                                                                                                            
I feel like zero-hour contracts are unacceptable, as they leave employees with no sick pay, holiday pay or guarantee of future earnings. This sort of security is something that I feel an employer should provide their employees with.

I would appreciate hearing your thoughts on this matter which I consider very important.

Yours sincerely

Daniel Watkins

[1] http://m.guardian.co.uk/business/2013/jul/28/sports-direct-staff-zero-hour-contracts

Sushi Go! Unboxing

A few months ago, I supported Sushi Go! on indiegogo. Sushi Go! is a set collection card game, similar in mechanic to 7 Wonders (one of my favourite games).

Today, it arrived! I haven't had a chance to play it yet (as I'm writing this at work), but here are some photos.

The Box

The box is very small, and very pretty. I'm pretty sure this is the cover design which I voted for, but I am obviously in no way biased.

Opening the Box

To give you some idea of the size, the cards are slightly smaller than standard playing cards (but big enough for my large hands to handle comfortably, I think!).

The Rulebook

The rulebook is smallish but, again very attractive!

Everything Side-by-Side

Some of the Cards

All in all, I'm a very happy camper!

Version 3 of sphinx-git Released

I've just pushed the latest release of sphinx-git to PyPI.

sphinx-git extends Sphinx, adding a directive which allows you to embed a git changelog of your documentation within the documentation itself.

sphinx-git is developed on GitHub where you can find an example project. Issues, comments or pull requests welcome!

Using inotifywait To Run Your Tests When Your Code Changes

Recently, I tweeted about my desire for a change-aware test runner:

I would like a test runner that detects when I have changed tests/code, and runs the relevant tests.
— Daniel Watkins (@Odd_Bloke) January 31, 2013

I haven't quite come up with that (though I do now have a half-finished blog post prognosticating on it), but I do have a solution which covers some of the bases: I determine what tests I want to run, and the test harness runs them whenever my code changes.

This "test harness" is actually a bash snippet which relies on inotifywait, a command-line program which blocks until it detects an inotify event on the files it is watching, and is incredibly simple:

while inotifywait -e close_write -r $CODE_DIRS --exclude=".*sw[px]"; do
    $TEST_COMMAND
done

This while loop is simple. When the inotifywait command stops blocking (and with exit code 0, which it will unless something unusual has happened to the files you're watching), we run $TEST_COMMAND. You can put whatever you want there, so you could limit the number of tests you run that way.

Now let's break that inotifywait call down. $CODE_DIRS can be any number of directories or files that you would like inotifywait to watch. -e narrows down the events that we should unblock on; we don't want to run the tests every time we open a new file. close_write triggers on (to quote the man page):

file or directory closed, after being opened in writable mode

vim triggers this when I save, so it works for me. You might also want to listen for modify (file or directory contents were written), move (file or directory moved to or from watched directory) and create (file or directory created within watched directory). Multiple -e options are comma-separated.

-r, as with many commands, tells inotifywait to watch directories recursively. This will mean that all directories under those that you specify will trigger your tests. I normally just run this command pointing at the top-level directory of my project. It's worth noting that -r applies to all directories that you pass.

Finally, --exclude takes a regular expression of files to exclude from your watch. As a vim user who hasn't configured swap files to be stored out of my tree, I want to ignore them (otherwise my tests run every time I open a file because vim writes out a swap file).

A quick disclaimer: as this uses inotify, this will only work on platforms that support it (which, I think, is only Linux). Mac users might want to examine this StackOverflow question. Windows users might find this answer useful.

UPDATE: +Murali Suriar has helpfully pointed me to kqwait on IRC, which will help out any Mac/BSD users.

Removing .pyc Files: A Coda

A few days ago, I wrote a blog post detailing a git hook that would automatically remove .pyc files when checking out a different branch. I received a variety of feedback, which I will outline here.

Firstly, +Jeff Mahoney provided a more efficient implementation of the git hook in a comment on the original post. I haven't tested this, but it might provide a more efficient implementation if you need it.

Secondly, there are a number of helpful comments on the reddit post, including a fine-tuned git/xargs command.

Finally, and most importantly, a number of people (both in the blog comments and on reddit) pointed out the PYTHONDONTWRITEBYTECODE environment variable.  Setting this to anything will mean that Python doesn't generate .pyc or .pyo files, completely circumventing the problem that I was trying to solve.

Git Tip: Get A Warning When A File Has Changed

Every project has one¹. The file. If that file changes, you need to know, because if you don't then your life is going to be very unpleasant in very short order.

At Hogarth, that file is called development.conf. It is, roughly speaking, a dump of the way our various elements and components are wired together², and as it's completely unmergable only one person can be working on it at any one time (something which we're working to fix). If you want to work on it, you need to make sure that you have the latest version in the database, so that when you dump out your new version, it includes all previous changes.

At various times, we've all been caught out by missing an update, and had to completely redo (often complex) changes just to incorporate a (often minor) previous change. One of my colleagues (Patrick), having just been caught out by this for (I think) the first time, suggested that it would be useful to get a warning when this changes. He reasoned that as we use git to manage all change to our codebase, it would be natural to write a git hook which did this for us.

We had a look together at the list of git hooks, and couldn't really see anything appropriate. post-merge seemed like what we wanted, but we couldn't work out how we could determine what had actually changed. So Patrick went off to do some real work, and I turned to the ever-reliable #git IRC channel on Freenode. ojacobson suggested a solution, which works beautifully.

To understand how this works, you'll need to know about the git reflog. +Alex Blewitt has a good introduction here. If you do one thing as a result of this article, educate yourself about the reflog. It's incredible!

The important take-away from that article is that:

git diff "HEAD@{1}"

will show you the diff between what you currently have in your tree and the commit before the last action that changed your history. Importantly, it treats merges as a single entry, so if you are immediately post merge, running the above command will show you a diff containing everything that changed in that merge, regardless of the number of commits the merge contained.

Pulling these various bits together, with some shell wizardry, gives us:

#!/bin/bash
set -eu

git diff "HEAD@{1}" --name-only | grep config/development.conf 2>&1 > /dev/null
CHANGED=$?

if [ $CHANGED ]; then
    echo
    echo -e "\e[41m!!! development.conf HAS CHANGED !!!\033[0m"
    echo
    echo "You should reload config using (something like):"
    echo "  ./manage.py loadconfig config/development.conf"
fi

We check if anything has changed, and if it has then we print out a BRIGHT RED WARNING MESSAGE. Drop that in .git/hooks/post-merge, make it executable, and you're good to go.

---

1 Well, maybe not every project. If you don't, I'm jealous.  But a lot of "enterprise" or otherwise unloved codebases will have one. And, more pertinently, ours does.^↩

2 You can tell that ours is not a Java project because this is not an XML file.^↩

Git Tip: Remove .pyc Files Automatically

N.B. I've published a follow-up to this here, which includes a way to completely avoid this problem in the first place.

Recently, I've found myself increasingly caught out by stale .pyc files in our project. When I change from our mainline branch to a story branch (or vice-versa), I often find myself with inexplicable test failures because Python is using the .pyc files for no-longer-current code.

Luckily, it's pretty easy to fix this in git, using hooks, specifically the post-checkout hook. To do that, add the following to .git/hooks/post-checkout, and make the file executable:

#!/bin/bash
find $(git rev-parse --show-cdup) -name "*.pyc" -delete

Now, every time you checkout a new branch, all the .pyc files will be cleared out of your git branch.

Django, Meet Nose

At Glasses Direct, we use nose to run our tests, as it gives us all sorts of nice things like test functions and XUnit-compatible output (which Jenkins loves).  As the majority of our projects use Django, we use django-nose to integrate nose into our Django projects.  This gives us all of that nose loveliness when using manage.py test.
The process for setting this up is simple:

1. Install django-nose and nose however you normally do (I would use pip install django_nose nose).
2. Add 'django_nose' to INSTALLED_APPS.
3. Add TEST_RUNNER = 'django_nose.NoseTestSuiteRunner' to your settings.
4. Run manage.py test -s to check that nose is being used (the standard manage.py test doesn't have this option).
5. Read manage.py test -h and nose docs to learn about the exciting things you can do with nose.

I've added a few more nose-related things to my blogging backlog, so I'll get to those eventually.

Deploying Sentry on Heroku

For a recent personal project, I've been using Sentry to monitor errors and suchlike. I was hosting this on my VPS, but the app it is monitoring is hosted on Heroku, so I thought that (a) it would make more sense to have the monitoring In The Cloud (TM), and (b) it would be an interesting exercise with Heroku, which I have been enjoying using a great deal.

It turns out to be pretty simple, and I've made a git repo available to make it even easier.  Just follow these steps:

1. Clone my GitHub repository:

   git clone https://github.com/OddBloke/heroku-sentry.git

2. Generate a secret key and set it as SENTRY_KEY in sentry.conf.py. See point 11 here for a good example of how to generate a good key.
3. Install the Heroku Toolbelt.
4. Sign up for a Heroku account.
5. Log in to your Heroku account from the CLI:

   heroku login

6. Navigate to your clone of the git repo and create a new Heroku app:

   heroku apps:create

   This command creates a heroku remote in the git repo, which we will use in a minute.
7. Take the URL it spits out (which will be something like http://floating-earth-1234.herokuapp.com/) and set that as SENTRY_URL_PREFIX in sentry.conf.py.
8. Commit your changes:

   git commit --all -m "Personalise."

9. Push your repository up to Heroku:

   git push heroku master

   If all goes according to plan, this will cause Heroku to deploy, and you will see the output as it installs Sentry and all of its requirements. Finally, it will tell you that your app is deployed to Heroku, but we're not quite done.
10. We need to configure a database for Sentry to use. We're going to use the new Heroku Postgres offering, which is currently in public beta: 

    heroku addons:add heroku-postgresql:dev

11. The previous command will have given you a database name, something like HEROKU_POSTGRESQL_CHARCOAL. Tell Heroku to use this database: heroku pg:promote HEROKU_POSTGRESQL_CHARCOAL
12. Finally, we need to run the Sentry setup script, create a default user and tell Sentry to use it: 

    heroku run sentry --config=sentry.conf.py upgrade
    heroku run sentry --config=sentry.conf.py createsuperuser
    heroku run sentry --config=sentry.conf.py repair --owner=JustCreatedSuperuser

    You should use the name of the super-user you create with the second command as the --owner argument to the third command.

Voila!  You should now be able to see Sentry by pointing your browser at the URL you used for SENTRY_URL_PREFIX earlier.

If you have any problems, you can check out your logs using heroku logs.

One final note: Heroku doesn't provide SMTP for you, so you'll also have to modify the SMTP settings to point at your own mail server (or play around with the Heroku add-ons that do provide it).

User Fixtures in Django

At Glasses Direct, we are setting up an internal system which needs a simple web UI.  As we use Django for all of our HTTP needs, the admin interface was the obvious, quick solution.  All we need to do is write the requisite model, tie it in to the admin interface and we're done!  Right?  Wrong.

The above description is missing an important part of the puzzle: authentication.  Django's admin interface (sensibly) requires authentication by default.  However, this piece of our system will only be exposed internally, and we don't want to have to manage credentials for all of our internal users (as we are sadly lacking when it come to internal single sign-on).

The obvious course of action is to remove the authentication.  However, this seems to be easier said than done.  Firstly, there is no simple switch to disable authentication. Secondly, even were there, we wouldn't want everyone to have access to the full admin interface, largely because it would be confusing for the target users.  So we can't get rid of authentication.  What we really want is a default user.

A default user is easy enough, you can add it using a User fixture that looks something like this (the easiest way to do this is to create a User object and use the dumpdata management command):

[{"pk": 2,
  "model": "auth.user",
  "fields": {
    "username": "default",
    "is_staff": true,
    ...
  }}]

Voila! After running a syncdb, you'll have a user who can access the admin interface.  Unfortunately, they won't be able to do anything, because they don't have any permissions.  Let's fix that by adding some (again, easiest to do this using dumpdata):

[{"pk": 2,
  "model": "auth.user",
  "fields": {
    "username": "default",
    "is_staff": true,
    "user_permissions": [12, 13, 14],
    ...
  }}]

You can see here that we've granted this user three permissions.  The relevant entries will show up in the admin interface.  We're done!  Right?  Wrong.

Everything will seem to be proceeding happily, possibly for quite some time.  Then, in a few weeks or months, you'll add another model, or app or something and suddenly your default user will have permission to do really weird things.  The problem here is that Django will occasionally regenerate the primary keys of permissions (and other internal objects).  So what are we to do?  After a fair amount of swearing this afternoon, my colleague Ondrej pointed me in the direction of natural keys. With these, you can future-proof yourself against primary key oddities:

[{"pk": 2,
  "model": "auth.user",
  "fields": {
    "username": "default",
    "is_staff": true,
    "user_permissions": [
      ["add_mymodel", "myapp", "mymodel"],
      ["change_mymodel", "myapp", "mymodel"],
      ["delete_mymodel", "myapp", "mymodel"]
    ],
    ...
  }}]

As with the above examples, you should generate this output with dumpdata, passing in the --natural flag on the command line.

To conclude, we've looked at how we can use Django fixtures to give us a default user with a known username and password, with reliable, known permissions. Perfect!


N.B. One option for "auto-authentication" would be to use a middleware class that sets the user on the request to our default user, something along the lines of this.  This component is only meant to be a quick procedure fix, so we haven't taken the time to do that.