Frank Carver's Punch Barrel

Agile software development places a lot of emphasis on prioritisation of work and elimination or deferral of anything which is not needed right now. An obvious advantage of this is that important stuff gets done quickly, but a less obvious disadvantage is that deferred work can pile up like snow before a snowplough. For a long-running project, the effort to keep “pushing” these deferred activities can become a significant task in its own right.

The Pragmatic Programmers wrote:

Don’t Live with Broken Windows
Fix bad designs, wrong decisions, and poor code when you see them.

While this is fine as a principle, there will still be times when edge-cases, uncommon modes of operation, laborious work-rounds and generally less valuable aspects of a system should be deferred, even though there is a sense that they will need to be improved later. The question arises of how to best represent and remember these issues so that they are not accidentally lost.

Some development IDEs and other tools incorporate in-code annotations such as TODO or FIXME, and almost every programming language supports comments, where such notes may be paced for more manual processing. These can, however, be tricky to find if you don’t already know where to look, and an apparently obvious note can quickly lose context and usefulness as developers move to focus on other work.

Mark Needham recently wrote of his experiences with a ‘window fixing’ wall.

A project I worked on recently tried a section of a card wall for “issues”, but these languished in a fog of “somebody else’s problem”. We got rid of that and replaced it with a section to collect issues for the next retrospective, but even that has become less used as the project progresses.

Inspiration for a solution might come from another post from mark Needham: “Pain Driven Development.” Although this article emphasises other aspects of agile process, the essence of the idea seems to be that the things that actually get done are the things which cause the most “pain”. Perhaps if we can find a way to convert theoretical “broken windows” and other incomplete work to a sense of “pain”, they will naturally be addressed as part of the normal intuitive prioritisation process.

Has anyone managed this?

It’s happened on most reasonable sized projects I have worked on. The benefits of test coverage an continuous integration are obvious and pay back immediately. But, somehow, as the project grows and diversifies, a point is reached where the complexity and run time of the CI process begins to slow down development rather than assist it.

Jez Humble has put together some interesting thoughts on how to deal with this issue. Read more at Deployment pipeline anti-patterns.

You have to love a good analogy. Here’s one which takes the notion of a “waterfall” development process literally, and contrasts it with a pleasant day out on a serene pond.

The Opposite of Waterfall is Pond – A Metaphor for Agile | Agile Blog: Scaling Software Agility

My favourite snippet:

Eventually, we find a place that everyone agrees looks nice and we pull the boat up on shore. Our project is complete in a way that we couldn’t have predicted exactly because we’ve never been on this particular pond before. We’re ready to set out again just as soon as we’re done with the picnic. In Pond, you always have a picnic at the end of the project.

All of which makes me smile. Back when I was working on an agile project where releases were named after ducks and other wildfowl, I proposed at one point a component named “pond” (standing for “provider of necessary data”) shared by all the deployed ducks.

Ah nostalgia.

It’s a common software development situation. In front of you is a problem which seems to require a large solution. It has several parts which may be deployed separately, or may just need to be swapped out or independently managed. It has infrastructure bits and business-specific bits, servers, services and clients.

Now imagine that you have a large chunk of time (months to years) and a team of several people (or perhaps even several teams). The daunting challenge is how to plan the work so that it all gets done and all works together.

Agile Coach and Mentor Chris Pitts of Thirsty Bear has recently written some thoughts on this issue which broadly align with my experience.

It’s extremely tempting to begin by setting separate people or teams to work on separate parts of the system. Even if the team are working together, it’s tempting to take on one part of the architecture diagram at a time and continue until all the blocks are in place.

This risks several common problems, though.

The most obvious is that this heads straight for the nightmare of “system integration” – that theoretically short, but all too commonly deadline-busting, period after all the components are “done” but before the system can be shown to actually all work together. This is the part of the classic design/build/test lifecycle which is often ignored yet can force significant (and rushed) changes to the components when assumptions are shown to be wrong.

A less obvious problem is that the longer the delay before being able to show full end-to-end operation, the longer the wait for feedback from all the stakeholders who only care about what the system does on the outside, not how it does it inside. Building a system based on guessing what the stakeholders want based on theoretical discussions, documents, and the hot features offered by some big-ticket infrastructure is very risky compared with real feedback on a system which does at least one small thing completely.

Chris suggests

Slice the functionality, not the components

Which is good advice as it goes.

I’d add a warning that it is also all too easy to define the tasks to produce a system in terms of parts and modules. A task card such as “install database”, or a story for “web client” risk distracting the team from the real desired outcomes the stakeholders care about. If you see any such stories (or tasks, issues, tickets or whatever you call them) try and find some end-to-end scenarios or use cases which require such things, and describe them instead. When that story is tackled, then the required infrastructure and components can be implemented as part of that story.

We are currently trying to come to some conclusions about the “shape” of a new software product, and facing a whole lot of problems. Stakeholders are happy to argue for hours about relative priorities of individual features, but so far these exist in a vacuum without an overall vision of a product.

With that in mind we are casting around for any hints on how to map out what we build and in what order. Chris Sims has some suggestions which may be useful, but I’m not sure they address our underlying problem.

InfoQ: Story Mapping Gives Context to User Stories.

Good, thought-provoking stuff. If agile is all about doing what’s necessary when it’s necessary, and not doing what’s not necessary, why should we wait for a retrospective to fix any problems?

Are retrospectives an antipattern? | Steven “Doc” List’s Random Musings.

How minimal can an agile process really go? Brad Cross has described his lightweight process and it does seem pretty lightweight

On the Gnarl Side: Lightweight Lean Agile.

Jason Yip from Thoughtworks has a nice summary of points to remember when organising or managing stand-up meetings.

You’d think with all my video game experience that I’d be more prepared for this: A quick summary of how I like do stand-ups

I’m still searching for decent project hosting. I now have several projects on the go, and several others bumping around in my head, and the fuss and bother of tying together all the various bits of a distributed software project development is making my head hurt.

All the bits I need are available separately, but so far I have not been able to find any single provider (free or paid for) which offers the combination of features I need. Essentially these are:

• Version control. Ideally git, but at a pinch one of the other distributed VCS tools or even subversion would probably do if everything else was in place. GitHub seems good for this.
• A project wiki. Using any other system for project docs just seems so clumsy. There are plenty of these; I use WikiDot for one project.
• Sensible bug/feature tracking. This is a bit more tricky – there is plenty of bug-tracker software, but not much that works equally well for managing unimplemented feature stories and associated tasks. Ideally this should link in with the version control, allowing code and change metadata to be updated in one go. Trac seems a possibility for this.
• Calendar management. For recording and communicating meetings, deadlines etc.. Something which works well with calendar syndication, so that anyone working on the project can see project events in with the rest of their appointments. Plenty of these: Google calendar, 30 boxes, etc. They all have their quirks, though.
• Task (todo) management. I find it amazing that task management is so poor in on-line calendars. There are standalone task tools such as Remember The Milk, but it is integration which is needed.

There are also a few other features which are definitely in the “useful to have” category, but I’m practical enough to use manual or off-line tools if necessary.

• Effort recording, tracking and reporting. For velocity tracking, process improvement, and even billing.
• Collaborative planning and prioritisation. Mingle tries to simulate a task wall, but is somewhat clumsy and irritatingly expensive; I have heard of on-line tools to run “Planning Poker” sessions, but as usual, not integrated with anything else.
• Continuous Integration. I’m not aware of any really smart tools to make use of distributed version control for this, yet. Our Cruise Control installation just stops and complains when something breaks, for example, but it should be possible to just “park” the failing patch and continue building with others in a real dvcs-based approach.

If anyone has any suggestions – or wants to build a product which does all this stuff – please let me know!

For interest, here are a few associated links.

Cuberick: Distribute Your Software Just Like Ubuntu With Launchpad

Comparison of open source software hosting facilities: Wikipedia

I use Test-driven Development (TDD) all the time. Although my immediate work colleagues have “bought in” to the idea, I sometimes find my self explaining about it to other people in other situations. A common question is whether the extra effort of writing all those tests slows down the overall development process. Mark Needham has an article on this very question.

TDD: Does it make you slower? at Mark Needham

I have noticed this phenomenon during our “planning poker” estimation meetings, and to some extent in retrospectives and other meetings about iterations, planning and progress. Each time a meeting is held, the number of attendees increases slightly, and the meeting (sometimes imperceptibly, but the cumulative effect is there) slows down. This slowdown leads to a perceived idea that if only we had more stakeholders in the room, we could get more answers more quickly, so more people are invited. And so it gets worse.

Chris Johnston » Of kickoffs and walk throughs

Is this a general curse of agile processes, or is it a particular anti-pattern or failure mode which can be avoided?

There are a lot of things to like about the Lean movement and the Toyota production system, but I worry that blanket application of their principles to software development may be misguided. Just as with other metaphors used to build a mental model, the metaphor of software development as a factory has big holes in it.

This becomes particularly apparent when considering the concept of defects. In a factory the job is to produce output in large quantities, all of which is consistent with an initial design. From this premise it is easy to understand that any difference from the initial archetype is a defect, which means that the factory process is not working correctly and needs to be fixed.

In software development, the job is not to produce multiple identical things, but to produce new things which did not exist before. From this premise it follows that there is no definitive archetype against which to measure differences. Sure, producing identical copies of software for sale or distribution is like a factory, but typically that process is almost trivial, requiring not much more than a “copy” command.

When developing software we sometimes find ourselves in the uncomfortable situation of not knowing whether something is a defect or not. All we know is that it is different from some expectation, but any particular difference may be a defect or it may be an improvement.

Testing: What is a defect? at Mark Needham

In an agile process, what is a defect?

Can anyone point me at any resources on Toyota’s approach to product development, rather than the familiar material on their approach to factory production?

It’s a common conundrum once a team embraces test-driven development (TDD). What becomes of the role of architect? Some suggestions include moving the role of the architect even higher than in a typical waterfall or “over the wall” process.

Sarah Taraporewalla’s Technical Ramblings » How does Architecture fit with TDD

My take on this is a bit different. One of the toughest things when developing software using TDD is choosing which tests to write next. This drives in a very real way the capabilities and design of the resulting system. At a low level the choice of test is often relatively easy, but at a higher level it can be easy to get lost in the detail.

So here’s my suggestion for the role of the architect in TDD-based development: driving the choice of high-level test cases to implement, and moderating the discussions between developers about how such test cases are split into finer tests and eventually implemented.

The benefits of TDD (evolutionary design, scaffolding, etc.) are still there, as is the input from an individual with collected understanding of which possibilities are implemented and which are not.

Thoughts?

A pleasantly witty yet insightful post about how different processes and methodologies can suffer similar problems, and how this relates to the individual learning journeys of practitioners.

lizkeogh.com » The problem with Scrum

Every now and then we have a discussion about bugs, whether and how they should be fixed, and the relative importance of each bug compared with key missing features. Solutions range from the naive to the overcomplicated, but at least nobody yet has suggested something like the following…

The Defect Black Market – The Daily WTF

Our agile team is finding some things challenging. In particular deciding how to prioritise and work on “bugs” in the midst of a pool of prioritised and scheduled feature stories.

“Agile In Action” has a nice summary of an approach to software development. Most agile practitioners won’t find anything to object to.

AGILE IN ACTION: The Energized Way

Our first problem is with the second bullet “Work with clients every day“. As a team we would love to work with clients every day, but there seems to be a thick layer of representatives and proxies in between us and real customers. This is made especially difficult as we are currently serving the needs of several customers with a single product, and resolving customer differences is proving tricky.

Our second problem is with “Fix defects as soon as they’re discovered“. In principle this seems obvious, but the trouble we are having rests on the definition of a defect. As an agile team we keep up-front specification to a minimum, and in-effect treat every delivery as a prototype ready for customer feedback. Plenty of people in the company have opinions on such prototypes – things they think it should do, things they think it should not do, and things they think it does wrong. Any of these could be considered as defects (and indeed many of them are raised in our bug tracking system.) If we stopped new work to make all these changes we would (a) greatly reduce our feature velocity, (b) bypass the prioritisation process used to “Deliver the client’s highest-value stuff first“, and (c) leave us stuck in a mire of conflicting opinions.

We certainly do not ever want to deliver “broken” software, but its a fact of life that some “bugs” are lower in priority than others. Some “bugs” are also lower in priority than new features, but this is more of a business decision than a development decision. Working out how deliver prompt, appropriate and minimal software in the face of such a slew of opinions is proving contentious.

I’d be interested in reading any suggestions or answers to these problems.

My browser is full of tabs, each representing something I intend to blog about. I need to clear some space, so here’s a few interesting links without comments.

• Reaching Hyper-Productivity with Outsourced Development Teams
• Pressure and Performance – The CTO’s Dilemma
• Agile Usability
• Web 2.0 Storytelling: Emergence of a New Genre

Nice little post highlighting the need for value information along with cost estimates.

Don’t worry about how valuable it will be, just tell me how much it costs?

I’d not seen this before, but it certainly has echoes of real life.

The Parable of the Two Programmers

We are currently struggling with how to integrate work on refactoring/simplifying/cleaning our product codebase with existing streams of stories and bugs. One of the tricky aspects of this is how to estimate and prioritise the cleanup work: how much is it worth, and how much time should we spend on it this iteration?

Martin Fowler has written about extending the idea of “technical debt” by including the concept of “interest”. Extra time spent on completing any given task compared with the time which would/might have been spent implementing that outcome on a cleaner system is effectively an “interest payment” on the “technical debt”. Noting an estimate of such an amount along with the elapsed time on each piece of work allows a relatively simple calculation of the total “interest payments”, and helps inform any decision to “pay off” the debt by refactoring, simplification, etc.

MF Bliki: EstimatedInterest