Frank Carver's Punch Barrel

It’s a common software development situation. In front of you is a problem which seems to require a large solution. It has several parts which may be deployed separately, or may just need to be swapped out or independently managed. It has infrastructure bits and business-specific bits, servers, services and clients.

Now imagine that you have a large chunk of time (months to years) and a team of several people (or perhaps even several teams). The daunting challenge is how to plan the work so that it all gets done and all works together.

Agile Coach and Mentor Chris Pitts of Thirsty Bear has recently written some thoughts on this issue which broadly align with my experience.

It’s extremely tempting to begin by setting separate people or teams to work on separate parts of the system. Even if the team are working together, it’s tempting to take on one part of the architecture diagram at a time and continue until all the blocks are in place.

This risks several common problems, though.

The most obvious is that this heads straight for the nightmare of “system integration” – that theoretically short, but all too commonly deadline-busting, period after all the components are “done” but before the system can be shown to actually all work together. This is the part of the classic design/build/test lifecycle which is often ignored yet can force significant (and rushed) changes to the components when assumptions are shown to be wrong.

A less obvious problem is that the longer the delay before being able to show full end-to-end operation, the longer the wait for feedback from all the stakeholders who only care about what the system does on the outside, not how it does it inside. Building a system based on guessing what the stakeholders want based on theoretical discussions, documents, and the hot features offered by some big-ticket infrastructure is very risky compared with real feedback on a system which does at least one small thing completely.

Chris suggests

Slice the functionality, not the components

Which is good advice as it goes.

I’d add a warning that it is also all too easy to define the tasks to produce a system in terms of parts and modules. A task card such as “install database”, or a story for “web client” risk distracting the team from the real desired outcomes the stakeholders care about. If you see any such stories (or tasks, issues, tickets or whatever you call them) try and find some end-to-end scenarios or use cases which require such things, and describe them instead. When that story is tackled, then the required infrastructure and components can be implemented as part of that story.

I have spent many years developing web applications in Java for corporate clients. During that time I have used a wide range of frameworks, APIs and other useful stuff. I have written my own versions (sometimes several) of many of these components, learned the tools well, and become very productive and effective.

However, it has been almost impossible to take that expertise, hard work, and custom code and easily/cheaply turn it into usable web applications for general public use. Low-cost hosting providers have generally shunned Java support. The most reasonable one I have found is LunarPages who actually support both ad-hoc JSP and deployment of custom web applications as war files, even though you have to search pretty deep in their web site to find out, and pay an extra dollar or two a month for the privilege.

So for my own projects I have been looking for an alternative for a long time. A way of developing and testing web apps on my various development boxes (currently running Ubuntu, MacOS X and Windows XP) and easily deploying to a low-cost hosting provider.

By far the most popular web development language is PHP. It’s available pretty much everywhere. It’s so focussed on web app development that the primary unit of coding is the web page. Believe me, I have tried to like PHP, but it’s just so clumsy. After a few paces the application begins to get tangled, development speed drops, and bugs creep in. Its web focus makes it tricky to unit test, and end-to-end testing seems to require a full-fat HTTP server. Not for me.

Beyond the traditional stomping ground of PHP, perl, and other CGI fodder is the new range of “trendy” languages. Ruby, Python, Erlang and even the venerable smalltalk are trying to position themselves as the thinking-person’s web development tool.

Erlang syntax is a bit too odd for me right now, although I may come back to it later. Smalltalk is interesting, but carries with it so much history and is hardly a popular choice for low-cost hosting providers. That leaves a short-list of Python and Ruby.

As languages I like both Python and Ruby. They have broadly similar design goals, both have keen developer communities with plenty of open source resources, and both are commonly found on linux-based web hosts. In order to decide between them I took a look around their frameworks and APIs.

Python has Google on its side. It’s the “native language” of the hugely-scalable Google AppEngine. However, a scan around the web looking for ways of writing web applications led largely to two: Zope and Django. (for balance, there are plenty of others, but these are the standout examples). Zope is old and sprawling. It has some great ideas but is hardly an obvious choice for small, tactical web apps. Django is a bit lighter, but still seems to assume a lot and require a lot of relatively fiddly config.

In the Ruby world the blindingly obvious choice for a web application framework is Rails. Sometimes it seems as if “Ruby” is just another way of saying “Rails”. However Rails, like Django, assumes a lot about the eventual application. Rails also makes a lot of use of code generation, which I simply do not like. I would always prefer that a framework eliminate boilerplate rather than just generate it for me. Looking a little further beyond Rails I came across some really interesting alternatives, and the one which really sparked my interest is Sinatra. For me this one framework made the difference. It’s so streamlined that a basic web app is as simple as:

require ’sinatra’
get ‘/’ do
‘Hello world!’
end

Best of all, simply running the above file using ruby hello.rb starts up a web server and begins serving pages on port 4567. No extra config or faffing.

Looking around for low cost hosting I found that the provider I already use (Dreamhost) supports Ruby web hosting using Passenger. An ssh to the Dreamhost server for “gem install sinatra” followed by a little bit of FTP and my first Ruby/Sinatra application was live!

A key part of any web application is the pages, and a key part of generating pages is a good template language. As a general-purpose templating language, I still prefer my own Stringtree Templater, part of the Mojasef Java framework. So far I have not found a similar templating solution for any other language. However, for the limited and specific requirement of generating web pages, I am becoming quite fond of Haml. It’s not at all a general templating solution, but it does massively simplify the generation of web pages.

Finally, after years of CVS and Subversion, I have made the jump to distributed version control. Not only is it more trendy, but the ability to work on multiple code branches, on multiple machines, with or without an internet connection, and easily share, identify and merge when necessary has been a key benefit. Occasionally I wonder if one of the other distributed VCS (Mercurial, Bazaar, darcs, etc.) would have been a better choice, but I needed to settle on one and get to grips with it. So I chose git.

There have been a few speed-bumps. Attempting to learn git from manual entries and simplistic tutorials gives very little help on deciding what is worth doing. After about a week of following the techniques in A Git Workflow for Agile Teams, though, I began to get comfortable. Add to that some lovely web deployment tricks from A web-focused Git workflow and a selection of GUI tools ( Git Cola, GitX, and Git GUI ) to get round all that tedious “git add” stuff and simplify infrequent operations, and git becomes a very useful and productive tool.

The upshot is that I currently feel very productive.

My toolbox has a few other things in it, but they are still candidates for change if I find anything better. I am managing my projects and git repositories using Unfuddle but it has a few shortcomings including a ticket system which is a pale and clumsy cousin of bugzilla. For editing I use generic syntax-highlighting text editors (gedit on Linux, TextEdit on Windows and Smultron on Mac) but none of them have the nice autocompletion and refactoring tools I grew addicted to in Eclipse. Suggestions welcome!

Finally, a few spare links relating to the tools mentioned above.

Sinatra: 29 Links and Resources For A Quicker, Easier Way to Build Webapps

HAML reference

Rack

Mini reviews of 19 Ruby template engines

The Forgotten Ruby Web Frameworks

I don’t really consider myself a Python developer, although I have dabbled. As with many of the less mainstream languages there are keen developers and thinkers trying to clear away the confusion and push the limits of what is possible.

Snakes on the Web

The article is an enthusiastic call to action for “Pythonistas”, but also a really useful summary of big problems and issues applicable to any language, framework or development approach. Read it, even if just for the excellent (and scary) summary of all the things a modern web application developer needs to be aware of.

As the author admits “We’re doing nothing mind-boggling here”, but the recommendation to provide a tooltip explaining why an inactive button or menu option is not available is certainly a good one.

Why is that button gray? « EclipseSource Blog.

Although I like the idea of git, I find that I don’t actually use it often enough to have internalised the command-line options. Perhaps the easy integration between Eclipse and subversion has made me lazy

I recently noticed some git-specific tools, which might go some way to making such occasional use more palateable. For example, GitX is a Mac OS X GUI tool which monitors a git workspace and supports easy selection, evaluation, and committing of changes. More details in the GitX User Manual.

After playing with it a little, I can see that it is helpful for managing a local git workspace and repository. However, what is missing is any support for the relatively common scenario of grabbing the contents of a remote repository, making a sequence of changes and local commits, then sending back some or all of those commits to the remote repository.

This omission seems common. I have not found any Eclipse tools to help with this, either, which leaves arguably the fiddliest and least memorable part of the process in the realm of poorly-documented command line tools.

Anyone found anything to help manage this?

Fred Brooks’ “Mythical Man Month” is a classic book on software development, and its core message has been a vital part of project management for decades. With that in mind it’s somewhat surprising to find someone willing to challenge this prevailing view.

Andy Singleton of Assembla has written some interesting blog posts suggesting an alterative interpretation of Brooks’ data.

Time to Vanquish the Mythical Man Month

Mythical Man Month revisited, again.

Interesting.

This has been irritating me ever since I got my MacBook. I use lots of different computers with lots of different operating systems and the simplest way to transfer information is often to use a FAT32-formatted removable USB flash drive.

On my Windows and Linux machines this works great. I plug in the drive and the files are available. No problem. On my Mac I plug in the drive and it messes with it. It splats a bunch of “hidden” folders on the drive then locks the drive from being dismounted while it spends a lot of CPU cycles attempting to index it. All of which is completely pointless. And there’s no config or preference to switch it off.

Worst of all, it even does it to the memory cards from my camera. This is inexcusable.

Searching around I found a lot of people complaining about this, but no real solutions, until I stumbled on macosxhints.com – 10.4: Disable Spotlight on a FAT32 external drive.

Simply create an empty file at the top level of the drive named .metadata_never_index and the Mac OS X “Spotlight” indexer should leave it alone forever.

Excellent.

A “workflow engine” is becoming the new must-have for enterprise system development. In days gone by it might have been an automatic choice to go for an expert system, Enterprise Service Bus, messaging infrastructure or big-ticket database, but those now seem a little bit passé.

There are several commercial workflow engines available, and a whole bunch of open source ones. Here’s a list of workflow engines written in Java, for example.

Boris Lublinsky has written an article at InfoQ expressing the strong opinion that writing your own workflow engine should not be an option.

I’m not sure I completely agree. For me the most telling part of the article appears near the end:

People today rarely implement their own database or O/R mapper or application server. Why is it often that people think that they should write their own workflow engine?

As it happens, I have implemented my own database, O/R mapper and application server, and found the experience invaluable in understanding the challenges and important features of such software. Writing a workflow engine would presumably be a similarly valuable lesson.

I have seen several attempts to implement an agile approach to software development within large organizations, and in many  (if not all) of those cases the end result has not been very compelling. This is in direct contrast to agile adoption in smaller, typically single-team, companies.

It’s natural enough to assume that problems with implementing agile processes in large organizations stem from some characteristic of the organizations themselves. Amr Elssamadisy, however, has a different suggestion. Perhaps some of the problems stem from the use of external agile “coaches”, and the way that they affect the way both the agile team and the organization operates simply by being there.

Read more at: InfoQ: Opinion: Agile Coaches Frequently a Source of Adoption Problems.

A thoughtful article on how attempts to provide a universal user interface typically create more problems than they solve.

Unqualified Reservations: Wolfram Alpha and hubristic user interfaces.

Not a deep blog post, this one. Just a few links related to writing.

• How to Write Articles and Essays Quickly and Expertly ~ Stephen’s Web ~ by Stephen Downes
• How to write a Novel
• The “snowflake method”
• Writing the perfect scene

A few months ago I made the decision to move the Stringtree and Mojasef code-bases on from their requirement to support older Java versions. I tagged a particular version of the code as “1.4 final” and proceeded to work through the trunk code in the repository to bring it in line with key Java 5 features such as generics, varargs and the enhanced for loop.

Since then I have progressively realized that I may have been a bit optimistic. A few odd bugs have been found in the 1.4 code, and a few key improvements have been made to the Java 5 code which by rights should also be implemented for 1.4. So I have decided to try and fix the situation.

As of today, the tag “1.4-final” in both Stringtree and Mojasef should be considered as deprecated, and any code which used to use that tagged version should instead use the branches (“Stringtree for Java 1.4″ and “Mojasef for Java 1.4″) I have just created. Initially, the code is identical, but the plan is to go through and apply known bug fixes and fully-backward-compatible internal improvements to the branched versions.

In tough economic times with a lot of people out of work and employers cutting back on training budgets few can afford the kind of big-budget, fancy hotel courses which were the staple of corporate training even just a short while ago. So Tobias Mayer has started a no-frills way to get “certified scrum master” status and spread the agile/scrum way of working. He calls it WelfareCSM.

The cost is essentially $50 (to the Scrum Alliance for the certificate) plus a voluntary contribution to cover any room costs etc. Sort out your own food and transport. Even for those of us who regard agile certification as a dubious concept this is pretty tempting.

Reading between the lines, though. It appears that the different agile “camps” may be slipping into a battle for mind-share. If Scrum certification becomes more widely available and understood, then alternative approaches (particularly XP, but also all the other adapted, customized and home-brewed agile working practices) may seem to have less value to employers and clients.

Hmm…

The news was pointed out to me by a friend before I could blog about it, but Google App Engine supporting Java and servlets is a massive move forward for server side Java.

I’m currently experimenting with my own evaluation of this, but in the meanwhile, here’s Michael Yuan’s take on it.

Michael Yuan » Blog Archive » Google App Engine for Java — First Impressions

… if they involve pancakes and beer!

AGILE IN ACTION: A whole lotta fun to be had.

We are currently trying to come to some conclusions about the “shape” of a new software product, and facing a whole lot of problems. Stakeholders are happy to argue for hours about relative priorities of individual features, but so far these exist in a vacuum without an overall vision of a product.

With that in mind we are casting around for any hints on how to map out what we build and in what order. Chris Sims has some suggestions which may be useful, but I’m not sure they address our underlying problem.

InfoQ: Story Mapping Gives Context to User Stories.

Good, thought-provoking stuff. If agile is all about doing what’s necessary when it’s necessary, and not doing what’s not necessary, why should we wait for a retrospective to fix any problems?

Are retrospectives an antipattern? | Steven “Doc” List’s Random Musings.

An excellent point from Andy Palmer. Code which starts out too neat and ordered can make refactoring too difficult. Throw it all in and let the organization emerge from the functionality.

Andy Palmer » Code should start out messy.

Do you want to get certified in agile software development?

Fastest Agile Certification On The Web!.

I like to read books, and have learned a lot from reading technical books in the past. These days I somehow always feel too busy to go back and re-read anything I have already read.

Mark Needham and his commenters offer come compelling arguments that it might actually be re-reading which brings the greatest value.

Re-reading books at Mark Needham.