The Growstuff Project

I mentioned in my earlier post about tech stuff that I'm a fan of agile software development and especially of Extreme Programming (aka XP, no relation to the Microsoft Windows version.) I thought it would be good to talk a bit about what that means, and how it affects the Growstuff project.

The underlying values of XP are that software development should be:

* participatory -- special effort is made to include everyone in every stage from planning to release
* adaptive -- able to respond to changes in circumstances/priorities in a fast moving (ahem, Internet) environment
* sustainable -- the code is developed to be maintainable, and people are encouraged to work at a pace that doesn't burn them out

To understand how novel this is, you have to know a bit about how traditional software projects work. The old-fashioned way of developing software (which is sadly still in use in many organisations, especially in large corporates and bureaucracies) goes something like this:

1. People who want the software write a requirements document, often with the assistance of professional business analysts who know how to extract information from the potential users about how they expect the system to work. This document is usually long, detailed, and unwieldy. The stakeholders are expected to read through the final version and sign off on it.

2. A design document is written, based on the requirements spec. The business analysts and/or senior software development people (often with titles like "Architect") take the requirements spec and turn it into a technical specification that defines how the system will be built. There are lots of diagrams of things like database tables and object models. A requirement like "our salespeople need to be able to access the system from their Blackberries" will be translated into technical language about VPN access, mobile interfaces, etc. This will, again, go through a review and signoff process.

3. Everyone argues about how long it will take to implement everything in the design document. The customers want it by June, but the tech people say it will take until at least November. If you're lucky you'll compromise on September; if you're unlucky there's a hard deadline (like a trade show or a legislation change) that means that June is it. At this point, you may try to negotiate to do less work, since you have less time. To remove any features from the system will mean going back and revising the requirements spec and the design document, getting signoff from everyone at each point. This can take a while, meaning you're even further crunched for time.

4. Programming starts! The work is broken up into chunks and assigned to developers, who work on their individual parts. The interactions between the different bits have, hopefully, been clearly defined in the design spec, so you can theoretically work in isolation from the other developers, knowing that they will give you data in X format and that all you have to do is emit results in Y format for the next person to use them. This time is mostly spent heads-down, hacking away in isolation. Every so often a manager comes round to ask how you're doing, or to tell you about changes they were unable to discourage the customers from making. The requirements and design docs no longer match reality. And you don't have time to write any other documentation as you go, so you just hope that someone, somewhere, is keeping track of everything. (Hint: they're not.) What's more, the code you're writing is kind of crap, because you're rushing it, but you tell yourself you'll fix it in the dot-point release after the big one. (Hint: you won't, because that release will be just as rushed.)

5. Finally most of the code is written and you try and integrate everything together. This is where you find out that the other programmers aren't exactly sending you X, and that the Y you're passing on should have been Z instead. Oops! Around this time is where deadlines really start to slip, and you realise you're never going to get it finished in time.

6. You work long hours and rush and rush to try and get it all done. You're tired and constantly either over-or-under caffeinated. Everyone's cranky. The customers are shouting about slipping deadlines and your manager can only insulate you from so much of it (assuming they even try and know how in the first place, which is not a given).

7. At some point stuff gets handed over to testers. It should have been in their hands months ago at this point, but of course they get it at the last possible minute. In the meantime, they've been working up a test plan based on the requirements and design specs. Their job is to make sure your code matches what was asked for in the first place. One problem -- things have changed along the way, and documentation has fallen out of step with reality. The testers work ridiculous hours, weekends, nights, all the time swearing at you. Not that it's your fault... they're just at the pointy end of that original unrealistic deadline.

8. The code is released and it's full of bugs. Worse, the users don't even want it any more, because it's 9 months since they actually needed it, and they -- or the industry you're in, or the public zeitgeist -- have moved on.

9. The next project is exactly the same, only this time you have to build on the mess you made the first time round.

-----

So that's how a typical medium sized software project works in commercial/bureaucratic environments -- most medium to large companies, governments, large non-profits, etc. It's been this way since roughly the 1960s, when computers were far more expensive, printing off reams of paper seemed like a reasonable thing to do, and everything didn't yet run on Internet time.

These days, computers are cheap, really good developers are hard to find, and everything moves at Internet speed. Rather than multi-year project release cycles, we're more likely to work at the scale of months or weeks, and even in that short time everything can change: Twitter could change their API, or some new social network like Pinterest could explode onto the scene, or you might suddenly find yourself with a bunch of users on tablet devices that didn't exist this time last year.

So, what to do? The answer is agile development.

Agile process basically throws out the traditional requirements-design-implementation-testing-release process (known as the Waterfall model, because every stage is meant to flow down to the next) and basically replaces every part of it.

Here are some of the tenets of Extreme Programming, the brand of agile that I like best:

Onsite Customer: the "customer" is part of the developer team. In a business, this may mean that some of the sales people actually come over to sit in amongst the developers so they can answer questions or take part in design discussions as they occur. In the Growstuff project, it means that we won't arbitrarily divide between "developers" and "users" but instead all hang out together on the same mailing lists and forums, and we'll explicitly seek out and value input from people who aren't coding. The customer's main job is to clearly describe what they want, and to help set priorities for tasks.

Short iterations: We'll push new features/bugfixes to the website on a frequent, regular cycle (probably every three weeks, at least at first). What's more, we'll plan each iteration as we come to it, rather than planning too far ahead, because we know that priorities can change quickly. In each iteration, we plan what we'll do using the XP Planning Game, a lightweight and reasonably fun process for specifying tasks (known as Stories in XP), estimating their scope, prioritising them, and assigning them to a developer to work on. At the end of an iteration, we take a few moments to reflect on what just happened, and feed that into the next iteration's planning game.

Sustainable pace: Short iterations mean we're unlikely to have major crunches before a big release, but we'll make it a core part of our value system that people should work at a pace that they find enjoyable and that doesn't burn them out.

Simple design: in XP parlance, when we first approach a problem we'll do "The Simplest Thing That Could Possibly Work", rather than planning too far in advance. No task will be bigger than will fit in a single iteration. As well as avoiding the problems of over-specifying something up front when it might change (XP: "You Ain't Gonna Need It"), this also avoids people biting off more than they can chew.

Collective code ownership: No single person should "own" our code, or any part of it. Not in the legal sense, but in the emotional sense. We want people to collaborate, not just in case of bus accidents, but also because it will get us better quality software. To that end, XP also advocates pair programming where two coders work on the same code at the same time. Although this works best in face-to-face situations, we'll try and do it as much as possible in our online community (there are various technological tricks, such as screen-sharing and skype) and where we can't we will at least have two people assigned to every task, to work together, learn from each other, and review each other's work.

Refactoring: This is a process of continual improvement in code, of removing or improving stuff that has got rusty, and of cleaning up areas of "technical debt". If you're working at a sustainable pace, you'll have time to do this, and it will benefit everyone. It's not an easy discipline to get into, but working in pairs means that one person can remind the other if they forget.

Test driven development: Test driven development means writing the tests before you write the software. It sounds backwards, but it actually helps you think about what outcomes you want from the code before you start, and it means you never run out of testing time at the end. Again, this is a practice that is reinforced by working with a partner.

Continuous integration: Every time one of us makes a change, we make sure it doesn't break anything else in the system. We can do this easily if we have good tests in place, and it means that releases -- which, after all, we do every three weeks -- are likely to run much more smoothly.

Team Coach: In order to keep all of this flowing well, the team needs a Coach (or potentially a pair of coaches), someone whose prime directive is to make sure that the team's values aren't being compromised. The coach leads the Planning Game, encourages good practices, and makes sure everyone is happy and productive. They may or may not work on other tasks, but whatever they do, their coaching duties are paramount.

So that's the idea, in a kind of theoretical sense, of how I'd like this project to work.

Of course, in terms of day-to-day pragmatic stuff (the dates of our iterations, how we write stories, how we pair, who is going to be coach) we'll have to figure that out over the first few iterations. I'd like to cover that in another post, and leave this one just as the underlying principles.

So, is this something that people are happy to sign on to? Any concerns or things you worry might not work? Any thoughts on where this might fall apart and, if so, how to avoid it? Speak now or forever -- well, okay, you can speak any time, but now would be especially good ;)