Frank Carver's musings about software and life
A fascinating presentation from Barcelona Ruby Conference. Fred George talks through the history and examples of his thinking about system architectures composed of micro services.
I found this particularly interesting as it has so many resonances with systems I have designed and worked on, even addressing some of the tricky temporal issues which Fred has avoided.
Thanks to Steve Cresswell for pointing this one out to me.
Agile software development places a lot of emphasis on prioritisation of work and elimination or deferral of anything which is not needed right now. An obvious advantage of this is that important stuff gets done quickly, but a less obvious disadvantage is that deferred work can pile up like snow before a snowplough. For a long-running project, the effort to keep “pushing” these deferred activities can become a significant task in its own right.
The Pragmatic Programmers wrote:
Don’t Live with Broken Windows
Fix bad designs, wrong decisions, and poor code when you see them.
While this is fine as a principle, there will still be times when edge-cases, uncommon modes of operation, laborious work-rounds and generally less valuable aspects of a system should be deferred, even though there is a sense that they will need to be improved later. The question arises of how to best represent and remember these issues so that they are not accidentally lost.
Some development IDEs and other tools incorporate in-code annotations such as TODO or FIXME, and almost every programming language supports comments, where such notes may be paced for more manual processing. These can, however, be tricky to find if you don’t already know where to look, and an apparently obvious note can quickly lose context and usefulness as developers move to focus on other work.
Mark Needham recently wrote of his experiences with a ‘window fixing’ wall.
A project I worked on recently tried a section of a card wall for “issues”, but these languished in a fog of “somebody else’s problem”. We got rid of that and replaced it with a section to collect issues for the next retrospective, but even that has become less used as the project progresses.
Inspiration for a solution might come from another post from mark Needham: “Pain Driven Development.” Although this article emphasises other aspects of agile process, the essence of the idea seems to be that the things that actually get done are the things which cause the most “pain”. Perhaps if we can find a way to convert theoretical “broken windows” and other incomplete work to a sense of “pain”, they will naturally be addressed as part of the normal intuitive prioritisation process.
Has anyone managed this?
I’ll say up front that I have some major issues with the idea of estimation in agile processes. There are plenty of alternatives to traditional time/manpower estimation which are at least as useful in delivering real, useful results.
However, if you are working in a context which values estimation, Adrian Wible has some interesting points about whether it is wise to re-estimate incomplete stories each iteration
Adrian’s Tech Blog: The case against iteration based re-estimation.
Thinking of giving up trying because nothing seems to make a difference? Jason Yip has an interesting article about how a feeling of helplessness is easily learned, but can be overcome, even in software development.
A nice concise article with some good suggestions on how to build trust among a team: Pragmatic Agilist: On Trust.
Monday was a bank holiday, so I found a little time to catch up with some of the web videos in my queue.
First I watched an inspiring session from TechCrunch Nordic which likens achieving an “exit” for a startup company to dating. Fun, and with a strand of truth.
TechCrunch Nordic – Tommy Ahlers from Mike Butcher on Vimeo.
Anyone running a startup, or thinking about it, should watch this one.
Second I watched a presentation ostensibly about Kanban and “single piece flow”, but really about much wider issues in planning and managing software development. I found the approach presented particularly interesting as it correlates very well with where my thoughts are at right now.
This video is best watched at infoq, to see both the presenters and the slides
Anyone working in software development, or managing a software project really needs to watch this one.
Kent Beck, rightly well-known for Extreme programming, Test-Driven Development and jUnit gave a really thought-provoking talk at the Startup Lessons Learned Conference in San Francisco on April 23, 2010.
Watch live video from Startup Lessons Learned on Justin.tv
If you are at all interested in software and/or startups, this is well worth a viewing.
Via Energized Work.
You have to love a good analogy. Here’s one which takes the notion of a “waterfall” development process literally, and contrasts it with a pleasant day out on a serene pond.
The Opposite of Waterfall is Pond – A Metaphor for Agile | Agile Blog: Scaling Software Agility
My favourite snippet:
Eventually, we find a place that everyone agrees looks nice and we pull the boat up on shore. Our project is complete in a way that we couldn’t have predicted exactly because we’ve never been on this particular pond before. We’re ready to set out again just as soon as we’re done with the picnic. In Pond, you always have a picnic at the end of the project.
All of which makes me smile. Back when I was working on an agile project where releases were named after ducks and other wildfowl, I proposed at one point a component named “pond” (standing for “provider of necessary data”) shared by all the deployed ducks.
Ah nostalgia.
Stephan Schmidt writes
Agile is mostly driven driven by managment and consultants, seldom bottom up by developers.
I found this very odd, and completely contrary to my own experience. From the perspective of someone who first encountered agile approaches in the form of Extreme Programming the problem is almost always how to explain the obvious developer benefits in ways which management will understand.
This phrasing of agile for managers seems to be the purpose of Scrum. I tend to recommend that puzzled managers read up on Scrum, but puzzled developers start with XP and Test Driven Development, which provide some much more tangible developer advantages.
via Code Monkeyism: What Developers Need to Know About Agile.
It’s a common software development situation. In front of you is a problem which seems to require a large solution. It has several parts which may be deployed separately, or may just need to be swapped out or independently managed. It has infrastructure bits and business-specific bits, servers, services and clients.
Now imagine that you have a large chunk of time (months to years) and a team of several people (or perhaps even several teams). The daunting challenge is how to plan the work so that it all gets done and all works together.
Agile Coach and Mentor Chris Pitts of Thirsty Bear has recently written some thoughts on this issue which broadly align with my experience.
It’s extremely tempting to begin by setting separate people or teams to work on separate parts of the system. Even if the team are working together, it’s tempting to take on one part of the architecture diagram at a time and continue until all the blocks are in place.
This risks several common problems, though.
The most obvious is that this heads straight for the nightmare of “system integration” – that theoretically short, but all too commonly deadline-busting, period after all the components are “done” but before the system can be shown to actually all work together. This is the part of the classic design/build/test lifecycle which is often ignored yet can force significant (and rushed) changes to the components when assumptions are shown to be wrong.
A less obvious problem is that the longer the delay before being able to show full end-to-end operation, the longer the wait for feedback from all the stakeholders who only care about what the system does on the outside, not how it does it inside. Building a system based on guessing what the stakeholders want based on theoretical discussions, documents, and the hot features offered by some big-ticket infrastructure is very risky compared with real feedback on a system which does at least one small thing completely.
Chris suggests
Slice the functionality, not the components
Which is good advice as it goes.
I’d add a warning that it is also all too easy to define the tasks to produce a system in terms of parts and modules. A task card such as “install database”, or a story for “web client” risk distracting the team from the real desired outcomes the stakeholders care about. If you see any such stories (or tasks, issues, tickets or whatever you call them) try and find some end-to-end scenarios or use cases which require such things, and describe them instead. When that story is tackled, then the required infrastructure and components can be implemented as part of that story.
I have spent many years developing web applications in Java for corporate clients. During that time I have used a wide range of frameworks, APIs and other useful stuff. I have written my own versions (sometimes several) of many of these components, learned the tools well, and become very productive and effective.
However, it has been almost impossible to take that expertise, hard work, and custom code and easily/cheaply turn it into usable web applications for general public use. Low-cost hosting providers have generally shunned Java support. The most reasonable one I have found is LunarPages who actually support both ad-hoc JSP and deployment of custom web applications as war files, even though you have to search pretty deep in their web site to find out, and pay an extra dollar or two a month for the privilege.
So for my own projects I have been looking for an alternative for a long time. A way of developing and testing web apps on my various development boxes (currently running Ubuntu, MacOS X and Windows XP) and easily deploying to a low-cost hosting provider.
By far the most popular web development language is PHP. It’s available pretty much everywhere. It’s so focussed on web app development that the primary unit of coding is the web page. Believe me, I have tried to like PHP, but it’s just so clumsy. After a few paces the application begins to get tangled, development speed drops, and bugs creep in. Its web focus makes it tricky to unit test, and end-to-end testing seems to require a full-fat HTTP server. Not for me.
Beyond the traditional stomping ground of PHP, perl, and other CGI fodder is the new range of “trendy” languages. Ruby, Python, Erlang and even the venerable smalltalk are trying to position themselves as the thinking-person’s web development tool.
Erlang syntax is a bit too odd for me right now, although I may come back to it later. Smalltalk is interesting, but carries with it so much history and is hardly a popular choice for low-cost hosting providers. That leaves a short-list of Python and Ruby.
As languages I like both Python and Ruby. They have broadly similar design goals, both have keen developer communities with plenty of open source resources, and both are commonly found on linux-based web hosts. In order to decide between them I took a look around their frameworks and APIs.
Python has Google on its side. It’s the “native language” of the hugely-scalable Google AppEngine. However, a scan around the web looking for ways of writing web applications led largely to two: Zope and Django. (for balance, there are plenty of others, but these are the standout examples). Zope is old and sprawling. It has some great ideas but is hardly an obvious choice for small, tactical web apps. Django is a bit lighter, but still seems to assume a lot and require a lot of relatively fiddly config.
In the Ruby world the blindingly obvious choice for a web application framework is Rails. Sometimes it seems as if “Ruby” is just another way of saying “Rails”. However Rails, like Django, assumes a lot about the eventual application. Rails also makes a lot of use of code generation, which I simply do not like. I would always prefer that a framework eliminate boilerplate rather than just generate it for me. Looking a little further beyond Rails I came across some really interesting alternatives, and the one which really sparked my interest is Sinatra. For me this one framework made the difference. It’s so streamlined that a basic web app is as simple as:
require ‘sinatra’
get ‘/’ do
‘Hello world!’
end
Best of all, simply running the above file using ruby hello.rb starts up a web server and begins serving pages on port 4567. No extra config or faffing.
Looking around for low cost hosting I found that the provider I already use (Dreamhost) supports Ruby web hosting using Passenger. An ssh to the Dreamhost server for “gem install sinatra” followed by a little bit of FTP and my first Ruby/Sinatra application was live!
A key part of any web application is the pages, and a key part of generating pages is a good template language. As a general-purpose templating language, I still prefer my own Stringtree Templater, part of the Mojasef Java framework. So far I have not found a similar templating solution for any other language. However, for the limited and specific requirement of generating web pages, I am becoming quite fond of Haml. It’s not at all a general templating solution, but it does massively simplify the generation of web pages.
Finally, after years of CVS and Subversion, I have made the jump to distributed version control. Not only is it more trendy, but the ability to work on multiple code branches, on multiple machines, with or without an internet connection, and easily share, identify and merge when necessary has been a key benefit. Occasionally I wonder if one of the other distributed VCS (Mercurial, Bazaar, darcs, etc.) would have been a better choice, but I needed to settle on one and get to grips with it. So I chose git.
There have been a few speed-bumps. Attempting to learn git from manual entries and simplistic tutorials gives very little help on deciding what is worth doing. After about a week of following the techniques in A Git Workflow for Agile Teams, though, I began to get comfortable. Add to that some lovely web deployment tricks from A web-focused Git workflow and a selection of GUI tools ( Git Cola, GitX, and Git GUI ) to get round all that tedious “git add” stuff and simplify infrequent operations, and git becomes a very useful and productive tool.
The upshot is that I currently feel very productive.
My toolbox has a few other things in it, but they are still candidates for change if I find anything better. I am managing my projects and git repositories using Unfuddle but it has a few shortcomings including a ticket system which is a pale and clumsy cousin of bugzilla. For editing I use generic syntax-highlighting text editors (gedit on Linux, TextEdit on Windows and Smultron on Mac) but none of them have the nice autocompletion and refactoring tools I grew addicted to in Eclipse. Suggestions welcome!
Finally, a few spare links relating to the tools mentioned above.
Sinatra: 29 Links and Resources For A Quicker, Easier Way to Build Webapps
I have seen several attempts to implement an agile approach to software development within large organizations, and in many (if not all) of those cases the end result has not been very compelling. This is in direct contrast to agile adoption in smaller, typically single-team, companies.
It’s natural enough to assume that problems with implementing agile processes in large organizations stem from some characteristic of the organizations themselves. Amr Elssamadisy, however, has a different suggestion. Perhaps some of the problems stem from the use of external agile “coaches”, and the way that they affect the way both the agile team and the organization operates simply by being there.
Read more at: InfoQ: Opinion: Agile Coaches Frequently a Source of Adoption Problems.
In tough economic times with a lot of people out of work and employers cutting back on training budgets few can afford the kind of big-budget, fancy hotel courses which were the staple of corporate training even just a short while ago. So Tobias Mayer has started a no-frills way to get “certified scrum master” status and spread the agile/scrum way of working. He calls it WelfareCSM.
The cost is essentially $50 (to the Scrum Alliance for the certificate) plus a voluntary contribution to cover any room costs etc. Sort out your own food and transport. Even for those of us who regard agile certification as a dubious concept this is pretty tempting.
Reading between the lines, though. It appears that the different agile “camps” may be slipping into a battle for mind-share. If Scrum certification becomes more widely available and understood, then alternative approaches (particularly XP, but also all the other adapted, customized and home-brewed agile working practices) may seem to have less value to employers and clients.
Hmm…
We are currently trying to come to some conclusions about the “shape” of a new software product, and facing a whole lot of problems. Stakeholders are happy to argue for hours about relative priorities of individual features, but so far these exist in a vacuum without an overall vision of a product.
With that in mind we are casting around for any hints on how to map out what we build and in what order. Chris Sims has some suggestions which may be useful, but I’m not sure they address our underlying problem.
Good, thought-provoking stuff. If agile is all about doing what’s necessary when it’s necessary, and not doing what’s not necessary, why should we wait for a retrospective to fix any problems?
Are retrospectives an antipattern? | Steven “Doc” List’s Random Musings.
An excellent point from Andy Palmer. Code which starts out too neat and ordered can make refactoring too difficult. Throw it all in and let the organization emerge from the functionality.
Do you want to get certified in agile software development?
The team at Energized Work are very upbeat about their process, and in particular how it helps generate real company income every week.
Much of their recipe will be familiar to most agile practitioners, but there are some good extra suggestions, too.
Wow, is Test-Driven Developement (TDD) really 10 tears old? Gojko Adzic is blogging the QCon London conference, and has a summary of the TDD lessons-learned session.
Gojko Adzic » QCon London 2009: What have we learned from 10 Years of TDD?.
Agile development relies for its speed and productivity on making only that which is really needed. Sometimes it can be easy to forget that performance and scalability can be needed too.
Alex Verkhovsky has an incisive post about how to deal with the notion of “premature optimization”.
Thread.current.to_s: The keyword in “premature optimization is evil” is “premature”