Type Slowly

Day log, 3rd November 2010

I didn’t bother with a day log yesterday, as I was at my other job, washing casks and carrying sacks of malt. Today, however, I was back in front of the computer today, with a surprising amout to show for it; I’ve completed my final programming assignment, to create a visualisation of the Mandelbrot set, and have added a few extra features to it - the ability to scroll around the plane and zoom in on areas of interest, for instance. With that wrapped up I’ve started looking seriously at my systems infrastructure coursework - it’s a pretty broad ranging course, starting with a few lectures on databases, followed with a load of stuff on operating systems, concurrency, memory management and networking, and finishing up with compilers, parsers and language design. 

We’re currently looking at how concurrency and multiprogramming are implemented in operating systems - I’ve read through our lecture notes today as well as looking at a couple of papers: Firstly, Gary Peterson’s brief letter on his mutual exclusion algorithm, which is wonderfully concise and informative - it sets out the context of the problem and existing solutions, describes his algorithm, and gives a proof of its correctness in just two pages, while still being incredibly accessible to a newcomer to the field. Secondly, I read through Christopher, Procter and Anderson’s paper on their Nachos operating system, developed for teaching undergraduates at Berkley, which was a valuable overview of the field, and what I can expect from the course. I also did a fair bit of reading around the subject elsewhere, and I think I’m starting to get my head around various concurrency primatives in a way that I hadn’t managed before. 

I’ve also been pressing on with learning some stats - I’ve just started the section on conditional probability in my book, and it treats a subject I’ve previously found quite tricky in a way that’s both rigarous and straightforward - in particular, it contains a derivation of Bayes theorem that follows very naturally from the formulas for joint and conditional probability.  I’m planning to write about this in more detail soon.

Setting up a Python development environment on Ubuntu

I’ve got to do a few assignments in Python for university, and wanted to set up my laptop (running Ubuntu Maveric) to do this. When I’m doing any Ruby development I use the excellent RVM to create a personal (or even project-specific) development environment for my work, and wanted to do the same with Python, not least so that my tentative first experiments in it were sandboxed in some way (I have bad memories of messing up my Cabal install when hacking on some Haskell, causing XMonad to fail to launch next time I rebooted.

Happily, the virtualenv package provides a useful python equivalent to RVM. Simply:

sudo apt-get install python-pip python-virtualenv

As it sounds, python-virtualenv is the apt package that installs virtualenv - python-pip is a python equivalent of soemthing like RubyGems which might come in useful later when I need to install libraries. Now set up a new virtual python environment:

virtualenv --no-site-packages ~/.python-local

This creates a fresh python install in the folder ~/.python-local - the no-site-packages option makes sure your local environment doesn’t import packages from elsewhere on the system- it’s totally stand-alone.

Then, just make sure that the bin directory in your local python directory is on your path, before the system bin directories (so your local one takes precedence) - I use zsh, so that involved adding ~/.python_local/bin to this line in .zshrc, like so:

export PATH=/home/tim/.cabal/bin:/home/tim/bin:/home/tim/.python_local/bin:$PATH

Then close your terminal window, open a new one to allow the PATH changes to take effect, and marvel at your new sandboxed python install! (try ‘which python’, ‘which pip’ or ‘which easy_install’ if you don’t believe me!).

There’s one gotcha however - you’ll notice that if you run virtualenv again (if you can’t get enough stand-alone python installations and just have to have MORE!), you’ll get an error:

Traceback (most recent call last):   File "/usr/bin/virtualenv", line 2, in      import virtualenv ImportError: No module named virtualenv 

It’s pretty easily explained - the virtualenv command you called (in /usr/bin) called out to the python interpreter (now in ~/.python_local/bin, which obviously couldn’t find the virtualenv libraries that are installed in the system python installation. Simply install virtualenv into your new development sandbox and all is fine again:

easy_install virtualenv

Day(s) log, 29th - 31st October 2010: Octopii, Cucumbers, and Probability Distribution Monads.

Aside from making Octopus costumes, and going to ATP’s Release the Bats concert last night (which was incredible - all the bands were great and it was a lot of fun), I’ve had a reasonably productive few days!

Friday was my first Systems Infrastructure lecture,which looks like it’s going to be a really interesting course - we’re currently learning about memory management, but we’re going to be doing loads of stuff about operating systems, compilation and concurrency control.

Aside from that, I learnt some more stats and read the Functional Pearl on Probabilistic Functional Programming in Haskell, which introduces the PFP library. I’d been toying with writing something similar as my dissertation project before finding out it already exists, so now am thinking that I might try implementing something interesting on top of it, a naive Bayesian classifier, perhaps, or something involving Markov chain simulation. The difficulty in coming up with these ideas, I think, comes from having to find something that’s (a) fairly well-defined in terms of scope, (b) sufficiently general in application to be interesting, and (c) within the limits of my abilities. Still, I still have a fair while before I have to choose, so it’s not yet a problem.

PFP really impresses me as a library actually, it’s really well-designed, in that the monadic structure allows you to wrap probability distributions around any other arbitrary domain-specific model types you have, and the actual mechanics of random sampling etc are completely abstracted away into the library’s combinators, meaning you’re only ever dealing with your own types, and the library’s types for representing probabilities and distributions. It’s a really natural, expressive way of dealing with probabilistic computing, and a case study in how to write a really good interface to a library for dealing with really general problems, and really illustrates the power and malleability of Haskell’s type system as a modelling tool.

In Harmonypark news, I’ve been putting the finishing touches to my last chunk of work on Thrive, which has taken a little longer than I expected, due to the fact that the changes caused loads of our cucumber features to fail. I’m starting to develop a few  misgivings about cucumber as a tool, in particular the issue that led to my problems this week - that the setup and teardown of your tests (especially when using a tool like pickle) is intimitely coupled to your implementation, so small changes in your code or model can break totally unrelated tests - something that really isn’t great when a single scenario is meant to test a single feature, atomically.

I suppose there’s a trade-off here: one could do without setup and teardown in features, doing all the setup through browser-steps, at which point you basically lose the atomicity of your test, as basically you have to test the entire liifecycle of your application up until the point you’re interested in. However, this does mean your test is totally general, and is in no way tied to any implementation details that the user or client doesn’t see.

On the other hand, you could use seperate setup steps occasionally, setting up your database with all the state needed to test your feature, making it highly atomic,  but also highly coupled to your implementation. The ideal, I imagine, is probably somewhere between the two. In any case, I think changes of this scale are probably more indicative of a requirements gathering, design, or modelling mistake at some point in the past - making wholesale changes like this is generally bad news, and that’s the real problem, rather than the way in which your test suite reacts to it. Still, it’s all going smoothly now, and we’re very nearly ready for launch, which is exciting.

Finally, I spent a bit of this afternoon organising my massive folder full of pdf’s of papers, ebooks and such using Mendeley Desktop, which seems like a fantastic tool. I’m also using librarything to keep track of physical books that I have read or want to read - I’m finding it hard to keep track of what I want to read, and whether it’s in the UCL library, on my shelves, or is something I might have to go to the British library for, or buy. Using both these tools, I’m hoping I can plan my reading a bit better (I tend to jump between texts and topics fairly freely at the moment, which isn’t necessarily a bad thing, but I’m finding it hard to measure my progress), as well as keeping an annotated record of the things I have read, which’ll make it easier to find books I need to refer back to later, if I need to.

Day log, 28th October 2010

I got a couple of things done today, at least - the work I’m doing for Harmonypark isn’t quite finished, but is coming along nicely, and with a bit of work, it’ll be done tomorrow. I’ve been running our cucumber tests to check for regressions, and have found a few minor ones - I knew our story coverage was a lot more complete than our spec coverage, but I’m slightly surprised by how much it is - I think I’m going to make a habit of immediately writing failing specs then fixing both them and the feature, whenever I come across a bug or regression that’s tested in stories but not specs.

After that, I spent a while having a look at Debasish Ghosh’s posts on Domain modelling with Haskell, and working through his example code myself. It does a really good job of explaining why I like Haskell so much - The expressiveness of the type system lends itself to exactly these sorts of problems, and with a good domain model expressed as types, the actual code can seem like and afterthought, or implementation detail. Haskell is also perfectly suited to writing DSLs it seems - as well as the fact that type declarations can look and act much like Context-free Grammars (I’d be interested to know if there’s an actual isomorphism between the two), Haskell’s also got some nice (if trivial) syntactic features that make DSLs that little bit more pleasant, using whitespace for function application being one good example

I really would like to have a go at implementing something non-trivial in Haskell -and I think I’m almost at the point where I’d be up to the task; I need to take a closer look at the options for Haskell web development frameworks and libraries, as well as options for persistance - both areas that can be pretty bewildering at first. I have got an idea for a non-trivial application that might lend itself well to being implemented in Haskell however, I might have a crack at a similar modelling exercise to the one Debasish demonstrates on my own problem domain, and see how I get on with that, before worrying about how to talk to the web and databases.

Day log, 27th October 2010

I spent the first half of today working on my Mandelbrot set visualisation, the most notable improvement to which is that it now actually displays the Mandelbrot set, rather than the sort-of-looks-like-a-biohazard-sign-and-definitely-isnt-the-Mandelbrot set. I’m going to try adding a UI for zooming in and out and panning around areas of the plane - I’ve started implementing a double-buffered panel class for AWT in Java which I’m going to use for this, but it still needs a bit of work. I spent the rest of the day at Harmonypark, finishing off my Thrive authorization work. I’m going to attempt to read a bit of Godel, Escher, Bach now - it’s pretty much perfect reading for evenings where I’m not quite up to continuing with work or reading textbooks, but still want to do something at least vaguely productive. Tomorrow, I have more work to do on Thrive (our launch date is fast approaching). I’m hoping to get further into my stats book, and do some Architecture and Hardware homework too.

Day(s) log, 23rd-26th Oct

I haven’t got around to writing a day log for each day between friday and now, but here’s the gist of what I’ve been up to:

For Harmonypark, I’ve been working on some last-minute changes to Thrive’s authorization logic before we launch. At this point, we haven’t really got time to do things as neatly as I’d normally like, so I’m really just doing whatever’s quickest to get the features passing, and am planning to refactor it once we have more time. In this situation more than ever, I’m really thankful for automated regression and acceptance tests- even though the solution I’m working on isn’t optimal in my terms, the tests do at least give me confidence that it’s fit for purpose.

Alongside that, I’ve nearly done my final Java programming coursework - to produce a visualisation of the mandelbrot set, which is providing an outlet for my unbounded appetite for abstraction, since I’ve had to surpress it at work for the moment. It consists of a class representing the mandelbrot set itself, another that acts as a generalised viewer for any arbitrary set of points in cartesian space, and an interface that represents a strategy for colouring points in the plane, based on a third dimension of information.

The trouble with it, though, is that it renders a very pretty pattern that doesn’t look much like the Mandelbrot set. I’m not yet sure why this is, but I’m hoping to get it sorted tomorrow.

I’ve also been reading more of my statistics book, which so far I’m very impressed with. It introduces every topic, starting with elementary probability with a lot of mathematical rigour, as well as providing useful real-world examples. Highly recommended. 

Tomorrow I’ll be going to school for an Architecture and Hardware lecture, then on to work for a final push on Thrive before we launch it. I’m hoping to sort out this Mandelbrot set thing once and for all too!