Using inotifywait To Run Your Tests When Your Code Changes

Recently, I tweeted about my desire for a change-aware test runner:

I would like a test runner that detects when I have changed tests/code, and runs the relevant tests.
— Daniel Watkins (@Odd_Bloke) January 31, 2013

I haven't quite come up with that (though I do now have a half-finished blog post prognosticating on it), but I do have a solution which covers some of the bases: I determine what tests I want to run, and the test harness runs them whenever my code changes.

This "test harness" is actually a bash snippet which relies on inotifywait, a command-line program which blocks until it detects an inotify event on the files it is watching, and is incredibly simple:

while inotifywait -e close_write -r $CODE_DIRS --exclude=".*sw[px]"; do
    $TEST_COMMAND
done

This while loop is simple. When the inotifywait command stops blocking (and with exit code 0, which it will unless something unusual has happened to the files you're watching), we run $TEST_COMMAND. You can put whatever you want there, so you could limit the number of tests you run that way.

Now let's break that inotifywait call down. $CODE_DIRS can be any number of directories or files that you would like inotifywait to watch. -e narrows down the events that we should unblock on; we don't want to run the tests every time we open a new file. close_write triggers on (to quote the man page):

file or directory closed, after being opened in writable mode

vim triggers this when I save, so it works for me. You might also want to listen for modify (file or directory contents were written), move (file or directory moved to or from watched directory) and create (file or directory created within watched directory). Multiple -e options are comma-separated.

-r, as with many commands, tells inotifywait to watch directories recursively. This will mean that all directories under those that you specify will trigger your tests. I normally just run this command pointing at the top-level directory of my project. It's worth noting that -r applies to all directories that you pass.

Finally, --exclude takes a regular expression of files to exclude from your watch. As a vim user who hasn't configured swap files to be stored out of my tree, I want to ignore them (otherwise my tests run every time I open a file because vim writes out a swap file).

A quick disclaimer: as this uses inotify, this will only work on platforms that support it (which, I think, is only Linux). Mac users might want to examine this StackOverflow question. Windows users might find this answer useful.

UPDATE: +Murali Suriar has helpfully pointed me to kqwait on IRC, which will help out any Mac/BSD users.

Removing .pyc Files: A Coda

A few days ago, I wrote a blog post detailing a git hook that would automatically remove .pyc files when checking out a different branch. I received a variety of feedback, which I will outline here.

Firstly, +Jeff Mahoney provided a more efficient implementation of the git hook in a comment on the original post. I haven't tested this, but it might provide a more efficient implementation if you need it.

Secondly, there are a number of helpful comments on the reddit post, including a fine-tuned git/xargs command.

Finally, and most importantly, a number of people (both in the blog comments and on reddit) pointed out the PYTHONDONTWRITEBYTECODE environment variable.  Setting this to anything will mean that Python doesn't generate .pyc or .pyo files, completely circumventing the problem that I was trying to solve.