Frank Carver's musings about software and life
I’ll say up front that I have some major issues with the idea of estimation in agile processes. There are plenty of alternatives to traditional time/manpower estimation which are at least as useful in delivering real, useful results.
However, if you are working in a context which values estimation, Adrian Wible has some interesting points about whether it is wise to re-estimate incomplete stories each iteration
Adrian’s Tech Blog: The case against iteration based re-estimation.
Monday was a bank holiday, so I found a little time to catch up with some of the web videos in my queue.
First I watched an inspiring session from TechCrunch Nordic which likens achieving an “exit” for a startup company to dating. Fun, and with a strand of truth.
TechCrunch Nordic – Tommy Ahlers from Mike Butcher on Vimeo.
Anyone running a startup, or thinking about it, should watch this one.
Second I watched a presentation ostensibly about Kanban and “single piece flow”, but really about much wider issues in planning and managing software development. I found the approach presented particularly interesting as it correlates very well with where my thoughts are at right now.
This video is best watched at infoq, to see both the presenters and the slides
Anyone working in software development, or managing a software project really needs to watch this one.
I have spent many years developing web applications in Java for corporate clients. During that time I have used a wide range of frameworks, APIs and other useful stuff. I have written my own versions (sometimes several) of many of these components, learned the tools well, and become very productive and effective.
However, it has been almost impossible to take that expertise, hard work, and custom code and easily/cheaply turn it into usable web applications for general public use. Low-cost hosting providers have generally shunned Java support. The most reasonable one I have found is LunarPages who actually support both ad-hoc JSP and deployment of custom web applications as war files, even though you have to search pretty deep in their web site to find out, and pay an extra dollar or two a month for the privilege.
So for my own projects I have been looking for an alternative for a long time. A way of developing and testing web apps on my various development boxes (currently running Ubuntu, MacOS X and Windows XP) and easily deploying to a low-cost hosting provider.
By far the most popular web development language is PHP. It’s available pretty much everywhere. It’s so focussed on web app development that the primary unit of coding is the web page. Believe me, I have tried to like PHP, but it’s just so clumsy. After a few paces the application begins to get tangled, development speed drops, and bugs creep in. Its web focus makes it tricky to unit test, and end-to-end testing seems to require a full-fat HTTP server. Not for me.
Beyond the traditional stomping ground of PHP, perl, and other CGI fodder is the new range of “trendy” languages. Ruby, Python, Erlang and even the venerable smalltalk are trying to position themselves as the thinking-person’s web development tool.
Erlang syntax is a bit too odd for me right now, although I may come back to it later. Smalltalk is interesting, but carries with it so much history and is hardly a popular choice for low-cost hosting providers. That leaves a short-list of Python and Ruby.
As languages I like both Python and Ruby. They have broadly similar design goals, both have keen developer communities with plenty of open source resources, and both are commonly found on linux-based web hosts. In order to decide between them I took a look around their frameworks and APIs.
Python has Google on its side. It’s the “native language” of the hugely-scalable Google AppEngine. However, a scan around the web looking for ways of writing web applications led largely to two: Zope and Django. (for balance, there are plenty of others, but these are the standout examples). Zope is old and sprawling. It has some great ideas but is hardly an obvious choice for small, tactical web apps. Django is a bit lighter, but still seems to assume a lot and require a lot of relatively fiddly config.
In the Ruby world the blindingly obvious choice for a web application framework is Rails. Sometimes it seems as if “Ruby” is just another way of saying “Rails”. However Rails, like Django, assumes a lot about the eventual application. Rails also makes a lot of use of code generation, which I simply do not like. I would always prefer that a framework eliminate boilerplate rather than just generate it for me. Looking a little further beyond Rails I came across some really interesting alternatives, and the one which really sparked my interest is Sinatra. For me this one framework made the difference. It’s so streamlined that a basic web app is as simple as:
require ‘sinatra’
get ‘/’ do
‘Hello world!’
end
Best of all, simply running the above file using ruby hello.rb starts up a web server and begins serving pages on port 4567. No extra config or faffing.
Looking around for low cost hosting I found that the provider I already use (Dreamhost) supports Ruby web hosting using Passenger. An ssh to the Dreamhost server for “gem install sinatra” followed by a little bit of FTP and my first Ruby/Sinatra application was live!
A key part of any web application is the pages, and a key part of generating pages is a good template language. As a general-purpose templating language, I still prefer my own Stringtree Templater, part of the Mojasef Java framework. So far I have not found a similar templating solution for any other language. However, for the limited and specific requirement of generating web pages, I am becoming quite fond of Haml. It’s not at all a general templating solution, but it does massively simplify the generation of web pages.
Finally, after years of CVS and Subversion, I have made the jump to distributed version control. Not only is it more trendy, but the ability to work on multiple code branches, on multiple machines, with or without an internet connection, and easily share, identify and merge when necessary has been a key benefit. Occasionally I wonder if one of the other distributed VCS (Mercurial, Bazaar, darcs, etc.) would have been a better choice, but I needed to settle on one and get to grips with it. So I chose git.
There have been a few speed-bumps. Attempting to learn git from manual entries and simplistic tutorials gives very little help on deciding what is worth doing. After about a week of following the techniques in A Git Workflow for Agile Teams, though, I began to get comfortable. Add to that some lovely web deployment tricks from A web-focused Git workflow and a selection of GUI tools ( Git Cola, GitX, and Git GUI ) to get round all that tedious “git add” stuff and simplify infrequent operations, and git becomes a very useful and productive tool.
The upshot is that I currently feel very productive.
My toolbox has a few other things in it, but they are still candidates for change if I find anything better. I am managing my projects and git repositories using Unfuddle but it has a few shortcomings including a ticket system which is a pale and clumsy cousin of bugzilla. For editing I use generic syntax-highlighting text editors (gedit on Linux, TextEdit on Windows and Smultron on Mac) but none of them have the nice autocompletion and refactoring tools I grew addicted to in Eclipse. Suggestions welcome!
Finally, a few spare links relating to the tools mentioned above.
Sinatra: 29 Links and Resources For A Quicker, Easier Way to Build Webapps
We are currently trying to come to some conclusions about the “shape” of a new software product, and facing a whole lot of problems. Stakeholders are happy to argue for hours about relative priorities of individual features, but so far these exist in a vacuum without an overall vision of a product.
With that in mind we are casting around for any hints on how to map out what we build and in what order. Chris Sims has some suggestions which may be useful, but I’m not sure they address our underlying problem.
We currenty use a wall, covered with brown envelopes, for story and task tracking. It has its advantages but prople, particularly people not based in this office, often ask for something else. Chris Sims at InfoQ has a useful summary of the pros and cons of high-tech and low-tech “information radiators”
YAGNI – it’s a neat term for a valuable technique. Ignoring an unknown future to concentrate on a known present. That does not mean that it’s application is obvious, though. I often find myself in “discussions” with architects and designers who recoil at the idea of building something specific to one customer or situation, when it is “obvious” that a general solution will be better.
Mark Needham has written some of his thoughts on this topic.
As you may recall from a previous post, I am looking for some decent project hosting both for my open source software and for some business ideas. I want a single point of contact which offers as a minimum: version control, wiki, bug/ticket tracking, calendar and tasks/todo. Nice to haves include time-tracking, collaborative planning and continuous integration.
Eventually I have found two likely candidates: unfuddle and assembla. Both offer most of what I want, but with a few differences.
Unfuddle is probably the simpler of the two. It offers subversion and git hosting, a kind of wiki called “Notebook”, a kind of low-rent bulletin board called “Messages” for discussions, deadline management using simple text milestones, and bugs/tickets using Trac. It also provides RSS and iCal feeds of workspace events and upcoming milestones. With increased monthly payment you can also get time tracking and file attachment storage for messages.
Assembla offers subversion, mercurial and git hosting (and can integrate with remote subversion and github repositories), a wiki with extensions to directly reference tickets, bugs, software versions etc., similar “messages” to unfuddle but with file attachment storage included, bugs/tickets using trac or a proprietary alternative, similar milestones to unfuddle (but I can’t find any iCal feeds which is a shame), time recording, scrum-style progress reporting, and a few other options such as a “chat” facility and a specialist repository for storing and annotating images. It can provide updates of events using twitter or by HTTP call-outs, which seems pretty flexible.
Neither one seems to have a very sophisticated calendar, so no arranging meetings etc. Neither one has any significant collaborative planning in the vein of Mingle or Silver Catalyst. Neither supports recording of anything other than time (money spent on each task would be very useful, for example.) And neither supports continuous integration as such (although it could probably be hacked together using Assembla’s HTTP call-outs.)
Pricing seems roughly similar, although calculated differently.
Unfuddle offers five price bands (free for open source, $9, $24, $49, and $99 per month). Each band offers larger quantities of storage, numbers of participants and projects etc. The $9/month plan offers 512MB of storage for 10 people on 4 projects, so for an example small private team of three developers the cost would be $9/month
Assembla determines pricing per user per “space” (a space seems roughly to equate to a single project). Free for public spaces. For private ones, each user/space is $2/month, and $3 per gigabyte of storage per month. So for the same small private team of three developers the cost would also be $9/month but for more features and more storage. The down side is that adding any new team member, even one who only needs occasional access, costs extra.
I have registered for free accounts on both systems and have started to try out everything I can. I’ll report back soon on my findings.
In the meanwhile I’d love to hear from anyone with any other suggestions for project hosting services along the lines of these two.
There are lots of approaches to estimation for agile projects, but not all of them include estimation of risk. Some approaches deliberately ignore risk, preferring to work from hindsight and averages.
For the others, a consideration of levels of risk for each estimated story or task seems a good idea.
While setting up for some new work on my masters dissertation today I checked to make sure I had the latest version of TiddlyWiki. While looking I found a whole “ecosystem” of TiddlyWiki variants.
One which particularly caught my eye was “teamtasks”, a neat way of using the TiddlyWiki technique to simplify project planning and tracking
Anyone who has worked with me in the past will probably recognize my standard response to vague or unclear requirements – “how will I know when I’m done?”. I use it so much becuase the simple trick of changing viewpoint to view work in terms of acceptance criteria is key to enabling sensible discussion, estimation, planning and development to begin.
Mark Needham has been considering similar issues, but rather than using the technique to clarify requirements so a pice of work can enter the development process, he looks at how to decide when to remove it.
In theory, an agile story is a simple and obvious thing with many purposes. A description of some desired usage; a token for discussion; a prompt for acceptance tests; a grain around which to gather more detail. In practice, a story can sometimes be more like a traditional feature requirement, or more like a delivery contract. In these cases, estimating, developing, and delivering the story can provoke problems, risk, and conflict.
One such case is when a story is too large for comfortable estimation. Commonly this arises when a story is driven by a desire for a feature or sweeping change – something easy to give a name to, but complex in its detail. We have faced at least one of these most iterations.
Bill de hÓra discusses some approaches to tackling such large stories in a recent blog post:
Bill de hÓra: Managing large stories on agile projects
In our team we typically tackle the problem slightly differently. The items on our progress wall are represented by brown A5 envelopes, each with summary information (title, id, estimate) written in large, friendly letters on the outside, and an accumulation of detail (notes, time spent, diagrams, etc.) inside. To stop the contents falling out, the envelopes are all placed on the wall with the opening on the side.
Why do I mention this? We use a quick of this to deal with large stories. A story too large to estimate and develop on its own is represented by an envelope with the opening at the bottom. This implies that such stories can not accumulate detail of their own, but only reference other stories. We split the large customer story into several internal stories, estimate, annotate an develop them independently, but do not consider the main story complete until all the component stories are also complete. To show the relationship we write the ids of the component stories on the outside of the “vertical” envelope.
This approach seems to work. Stakeholders still have visibility of the status of the large story, and the development team is free to implement the component stories in parallel or in sequence, in whatever order is most effective and efficient.
I have noticed this phenomenon during our “planning poker” estimation meetings, and to some extent in retrospectives and other meetings about iterations, planning and progress. Each time a meeting is held, the number of attendees increases slightly, and the meeting (sometimes imperceptibly, but the cumulative effect is there) slows down. This slowdown leads to a perceived idea that if only we had more stakeholders in the room, we could get more answers more quickly, so more people are invited. And so it gets worse.
Chris Johnston » Of kickoffs and walk throughs
Is this a general curse of agile processes, or is it a particular anti-pattern or failure mode which can be avoided?
In our team we have found a tendency for the time spent estimating stories to creep upward. More and more design seems to be considered as we work through the stories. While this is a very valuable way of sharing information among the team, it is not a very effective way of estimating.
Akshay Dhavle has written a brief blog post reminding that sometimes, all that is needed is a simple “small, mediun or large” estimate. Business Analysis: Can’t negotiate size… and a good thing too
Nice little post highlighting the need for value information along with cost estimates.
Don’t worry about how valuable it will be, just tell me how much it costs?
We are currently struggling with how to integrate work on refactoring/simplifying/cleaning our product codebase with existing streams of stories and bugs. One of the tricky aspects of this is how to estimate and prioritise the cleanup work: how much is it worth, and how much time should we spend on it this iteration?
Martin Fowler has written about extending the idea of “technical debt” by including the concept of “interest”. Extra time spent on completing any given task compared with the time which would/might have been spent implementing that outcome on a cleaner system is effectively an “interest payment” on the “technical debt”. Noting an estimate of such an amount along with the elapsed time on each piece of work allows a relatively simple calculation of the total “interest payments”, and helps inform any decision to “pay off” the debt by refactoring, simplification, etc.
Interesting thoughts from Mana at “Geek Diva”
Planning what order to do stuff in is a vital, yet very difficult, part of software development. Agile wisdom usually stresses the need to do things in order of “business value”, but this can sometimes be extremely tricky to evaluate.
Another approach is to do things in an order intended to decrease risk, such as starting with the most worrying, or least understood parts of a system.
InfoQ: Iterating To Acquire Knowledge, Not Just ‘Business Value’
While this sounds plausible, I am not sure I entirely agree the reasoning. Behind the idea of ordering things by risk seems to be the idea that there is some bounded set of requirements, known early enough in the process to be able to determine their “worry”. To me the whole point of agile development is to step outside that assumption into the real world where work needs to start before full scope is defined, where priorities change on a whim, and development on a product may be cancelled at any time.
Starting work by spending a few iterations investigating complex/risky parts of some imagined future product instead of delivering usable but minimal versions from the first iteration seems somewhat irresponsible.
I started to idly watch this video presentation, but quickly found that it describes a situation eerily close to our own, so I have decided to set some time aside to watch it in proper detail. I hope it pays off!
A lot of writings about agile processes seem to assume that everyone comes with all the skills they need, but in the real world people sometimes need to gain new skills which cannot easily be learned “on the job”. Planning ahead enough to ensure team skills are available when needed, and dealing with the impact on development speed of people spending time on training or independent study rather than productive work seem like tricky problems.
Jooli Atkins has written a bit on this topic for the British Computer Society (BCS) Agile Training : Blogs : BCS
I’d be really interested to hear from anyone how they deal with this!
We don’t use scrum as such, but we still face the issue of team-member absence during fixed iterations. It’s a tricky one to deal with, especially as absence comes in both predictable and unpredictable forms.
InfoQ has a nice summary of some of these issues and some solutions. There is till no “silver bullet” for this, though. Losing staff will almost always mean losing productivity, which in turn means that iteration (“sprint”) commitments will need to be renegotiated, which works best if stakeholders are informed as soon as possible.