Frank Carver's Punch Barrel

An excellent article from Michael Chletsos and Titas Norkunas at Assembla, which reminded me how important it is to keep anything which might fail or need rework off the master branch.

It’s a truism about software development that you never know where the bugs will be until you find them. This can be a real problem if you find bugs in an integrated delivery, as it prevents the whole bunch from shipping. Assembla have some interesting stats about how they have been able to release code much more frequently by doing as much testing and development as possible on side-branches.

Read more at: Avoiding Premature Integration or: How we learned to stop worrying and ship software every day.

I just read an article (Do You Really Want to be Doing this When Youre 50?) from James Hague, who describes himself as a “Recovering Programmer“.

I understand his experience, and his reasons for deciding that it’s not the job for him. I even like that he has blogged about it.

What I really don’t like, though, is the sweeping generalisation that this is a universally-applicable, age-related issue. Software development is already a field riddled with ageism from employers and management (see Silicon Valley’s Dark Secret: It’s All About Age from TechCrunch), and the last thing we need is an assumption that one person’s dissatisfaction with a particular job means that, in James Hague’s closing words, “large scale, high stress coding? I may have to admit that’s a young man’s game”

A programmer in his or her 40s or 50s who has kept up with the evolution of the field can be one of the most awesome and effective developers you’ll ever meet.

It’s a common pattern, an enterprise software solution has a body of code, managed by version control, verified by continuous integration and eventually delivered to some deployment system. If that were the whole picture things would be relatively straightforward. Tests pass? deploy it!

Unfortunately there is often also a database in the mix. This complicates things. Decisions about how many databases to use in the development process, how to ensure that development workspaces, test systems, and live deployments always use an appropriate database, and how to ensure that database changes and code changes don’t break each other occupy an increasing amount of time and energy.

Tom Czarniecki discusses one path through some of these issues. His suggestions do introduce some potential extra complexity in the code, though.
Zero-downtime deployments « No One Is Perfect.

Sometimes I can’t resist the urge to grumble to colleagues when confronted with systems developed in a way I would not have chosen. Maybe I just need A change in attitude.

On the other hand, often the frustration comes from knowing that there is a better way, and I’m not sure that learning to love legacy code is that easy.

Thinking of giving up trying because nothing seems to make a difference? Jason Yip has an interesting article about how a feeling of helplessness is easily learned, but can be overcome, even in software development.

Learned helplessness in the workplace.

test-Driven development (TDD) is a very powerful and beneficial approach to developing software. I routinely recommend it as a solution to untested and hard-to maintain software. However, every rule has its exceptions and there are occasions when a pure TDD approach is not very helpful.

Mark Needham wrote a recent blog post on the subject where he describes a kind of “Spike Driven Development”: Coding: Spike Driven Development at Mark Needham.

I tend to think of this as some sort of experiment-driven development. I start by choosing a goal. Typically this is either finding out about some murky or unknown aspect of a system such as interfacing with a third party, or improving a variable quantity such as performance. This in turn leads to thinking about ways to assess whether we have done enough to meet the goal and how much time to allocate to the exercise. Finally, a period of playing, following where each trial leads. The important bit is to repeatedly check progress against the goal – that’s what links it to the aspects of TDD which work well.

It’s a tough call to make, but sometimes the best thing for a business to do is to throw away working code if it’s not what the business needs to be successful right now.

Eric Ries wrote an insightful post back in February 2009.

Lessons Learned: Throwing away working code.

A business which is not making money is dying, however good the code, but changing the business into one which does make money may mean ditching lots of things which have cost a lot, and even seem to have intrinsic value.

I’m doing quite a lot of agile and TDD mentoring at the moment, so such things are always floating near the top of my mind.

A recent article from Mark Needham presents an interesting view on how to construct code within tests, and the impact that might have on how tests drive design of code.

There’s an interesting discussion going on at The Wisdom of Ganesh in which Ganesh Prasad and “Integral ):( Reporting” (presumably the “JJ Dubray” mentioned in the article) are trying to work out some issues around versioning, REST and SOAP. This post is also referenced and commented on at infoQ.

In the 14th comment to the original article, JJ calls for a real-world example – this is a good idea which should help ground the discussion and diffuse the temptation to communicate in abstract principles. In the spirit of this request, I will describe an example based on problems I faced in a previous job.

The first version of the system we were working on consisted of a server component, coded in Java, which provided remote services to several client systems. Each of the client systems was a Java web application, responsible for providing customer-specific web pages and navigation using the services and content provided by the central server. For some customers there would be just one instance of the web client, for others with greater load there would be several.

The server system was responsible for managing products for sale, including descriptions in multiple languages, format and compatibility meta-data, thumbnail images in several sizes, prices in multiple currencies. As well as access to individual items and collections, a search facility provided a way to discover or look up products. The server also managed customer details and purchase histories, and handled both delivery of content and general messaging (by email and SMS) during the purchase process. The client systems were responsible for display and navigation of a product catalogue, customer branding, “shopping cart”, login/logout, and everything else a user experiences.

The API between client and server included ways to list products by category, fetch details of specific products, search using a variety of criteria, make and cancel purchases, fetch and update user data and so on.

I suggest this should be a reasonable system on which to base a discussion – it embodies similar concepts to many other systems, and is neither trivial nor impossible to build. Interestingly, both SOAP and REST versions of the interface between the clients and the server were implemented at different points in the life of the product.

So, on to versioning. It seems to me that there are several kinds of changes which might require versioning. I will list a few here, but I welcome suggestions for others I might have missed.

• The first deployment of the system worked with only a single currency. The move to multiple currencies was a potentially major change in the interaction between client and server, even though each client works only in a single currency.
• The customer for one client implementation required the provision of extra services, specific to their market, which were not required by (and should not be available to) other clients.
• The initial version of the search facility returned data for all matches, with paging through the results implemented entirely on the client. When there was a lot of data in the catalogue this became an unacceptable performance bottleneck, especially as results beyond the first few pages were hardly ever viewed. An alternative approach was suggested where a client would request a specified page (of a specified page size) from the search results. This required changes in both the search request and search results.
• A new source of products introduced the idea of collections. Items in a collection could be purchased individually or as a bundle. Viewing a collection while browsing, searching, or in a user’s purchase history, should allow a user to “drill in” to the collection to view the items within it. Items located while searching should indicate how to purchase the collection, as well as how to purchase the individual item.

Is this a useful amount of detail for a scenario? I have my own experiences of working with this system and the choices we made, but I’d love to read any suggestions for how you might handle any of the transitions above in a live system with multiple client systems deployed, many of which can not be changed easily or often, and a server full of historical purchase data. Procedural remoting approaches (SOAP, XML-RPC and Java RMI etc.) and resource-based approaches (REST) are all welcome.

Monday was a bank holiday, so I found a little time to catch up with some of the web videos in my queue.

First I watched an inspiring session from TechCrunch Nordic which likens achieving an “exit” for a startup company to dating. Fun, and with a strand of truth.

TechCrunch Nordic – Tommy Ahlers from Mike Butcher on Vimeo.

Anyone running a startup, or thinking about it, should watch this one.

Second I watched a presentation ostensibly about Kanban and “single piece flow”, but really about much wider issues in planning and managing software development. I found the approach presented particularly interesting as it correlates very well with where my thoughts are at right now.

This video is best watched at infoq, to see both the presenters and the slides

Anyone working in software development, or managing a software project really needs to watch this one.

I guess its is a common problem for software developers with a day job as well as out-of-hours projects, but it can be hard to get around to spending time on the out-of-hours stuff.

I read with interest that James Carr sought out someone to pair with and found it a useful and enjoyable experience. ( James Carr » Blog Archive » Feels GREAT to Pair Again. ) It occurred to me that perhaps this is what I need, and by implication, perhaps this is what you might benefit from, too.

So I have an open offer. If you like the idea of spending some evening or weekend time pairing with me, please get in touch. Obviously I’d like to make some progress on some of my own projects, but I’m also very happy to spend time on your projects, too.

There’s a fighting chance that the commitment of an appointment with someone else might get both of us motivated and productive, and that we both might learn some good stuff along the way.

What can we work out together?

You have to love a good analogy. Here’s one which takes the notion of a “waterfall” development process literally, and contrasts it with a pleasant day out on a serene pond.

The Opposite of Waterfall is Pond – A Metaphor for Agile | Agile Blog: Scaling Software Agility

My favourite snippet:

Eventually, we find a place that everyone agrees looks nice and we pull the boat up on shore. Our project is complete in a way that we couldn’t have predicted exactly because we’ve never been on this particular pond before. We’re ready to set out again just as soon as we’re done with the picnic. In Pond, you always have a picnic at the end of the project.

All of which makes me smile. Back when I was working on an agile project where releases were named after ducks and other wildfowl, I proposed at one point a component named “pond” (standing for “provider of necessary data”) shared by all the deployed ducks.

Ah nostalgia.

This is a neat idea, and certainly more friendly than “keyhole surgery” with ssh and vi if you need to edit files on a remote server over the web. Does not seem to have had a lot of development recently, but seems pretty usable as it is.

ecoder | home

I have spent many years developing web applications in Java for corporate clients. During that time I have used a wide range of frameworks, APIs and other useful stuff. I have written my own versions (sometimes several) of many of these components, learned the tools well, and become very productive and effective.

However, it has been almost impossible to take that expertise, hard work, and custom code and easily/cheaply turn it into usable web applications for general public use. Low-cost hosting providers have generally shunned Java support. The most reasonable one I have found is LunarPages who actually support both ad-hoc JSP and deployment of custom web applications as war files, even though you have to search pretty deep in their web site to find out, and pay an extra dollar or two a month for the privilege.

So for my own projects I have been looking for an alternative for a long time. A way of developing and testing web apps on my various development boxes (currently running Ubuntu, MacOS X and Windows XP) and easily deploying to a low-cost hosting provider.

By far the most popular web development language is PHP. It’s available pretty much everywhere. It’s so focussed on web app development that the primary unit of coding is the web page. Believe me, I have tried to like PHP, but it’s just so clumsy. After a few paces the application begins to get tangled, development speed drops, and bugs creep in. Its web focus makes it tricky to unit test, and end-to-end testing seems to require a full-fat HTTP server. Not for me.

Beyond the traditional stomping ground of PHP, perl, and other CGI fodder is the new range of “trendy” languages. Ruby, Python, Erlang and even the venerable smalltalk are trying to position themselves as the thinking-person’s web development tool.

Erlang syntax is a bit too odd for me right now, although I may come back to it later. Smalltalk is interesting, but carries with it so much history and is hardly a popular choice for low-cost hosting providers. That leaves a short-list of Python and Ruby.

As languages I like both Python and Ruby. They have broadly similar design goals, both have keen developer communities with plenty of open source resources, and both are commonly found on linux-based web hosts. In order to decide between them I took a look around their frameworks and APIs.

Python has Google on its side. It’s the “native language” of the hugely-scalable Google AppEngine. However, a scan around the web looking for ways of writing web applications led largely to two: Zope and Django. (for balance, there are plenty of others, but these are the standout examples). Zope is old and sprawling. It has some great ideas but is hardly an obvious choice for small, tactical web apps. Django is a bit lighter, but still seems to assume a lot and require a lot of relatively fiddly config.

In the Ruby world the blindingly obvious choice for a web application framework is Rails. Sometimes it seems as if “Ruby” is just another way of saying “Rails”. However Rails, like Django, assumes a lot about the eventual application. Rails also makes a lot of use of code generation, which I simply do not like. I would always prefer that a framework eliminate boilerplate rather than just generate it for me. Looking a little further beyond Rails I came across some really interesting alternatives, and the one which really sparked my interest is Sinatra. For me this one framework made the difference. It’s so streamlined that a basic web app is as simple as:

require ‘sinatra’
get ‘/’ do
‘Hello world!’
end

Best of all, simply running the above file using ruby hello.rb starts up a web server and begins serving pages on port 4567. No extra config or faffing.

Looking around for low cost hosting I found that the provider I already use (Dreamhost) supports Ruby web hosting using Passenger. An ssh to the Dreamhost server for “gem install sinatra” followed by a little bit of FTP and my first Ruby/Sinatra application was live!

A key part of any web application is the pages, and a key part of generating pages is a good template language. As a general-purpose templating language, I still prefer my own Stringtree Templater, part of the Mojasef Java framework. So far I have not found a similar templating solution for any other language. However, for the limited and specific requirement of generating web pages, I am becoming quite fond of Haml. It’s not at all a general templating solution, but it does massively simplify the generation of web pages.

Finally, after years of CVS and Subversion, I have made the jump to distributed version control. Not only is it more trendy, but the ability to work on multiple code branches, on multiple machines, with or without an internet connection, and easily share, identify and merge when necessary has been a key benefit. Occasionally I wonder if one of the other distributed VCS (Mercurial, Bazaar, darcs, etc.) would have been a better choice, but I needed to settle on one and get to grips with it. So I chose git.

There have been a few speed-bumps. Attempting to learn git from manual entries and simplistic tutorials gives very little help on deciding what is worth doing. After about a week of following the techniques in A Git Workflow for Agile Teams, though, I began to get comfortable. Add to that some lovely web deployment tricks from A web-focused Git workflow and a selection of GUI tools ( Git Cola, GitX, and Git GUI ) to get round all that tedious “git add” stuff and simplify infrequent operations, and git becomes a very useful and productive tool.

The upshot is that I currently feel very productive.

My toolbox has a few other things in it, but they are still candidates for change if I find anything better. I am managing my projects and git repositories using Unfuddle but it has a few shortcomings including a ticket system which is a pale and clumsy cousin of bugzilla. For editing I use generic syntax-highlighting text editors (gedit on Linux, TextEdit on Windows and Smultron on Mac) but none of them have the nice autocompletion and refactoring tools I grew addicted to in Eclipse. Suggestions welcome!

Finally, a few spare links relating to the tools mentioned above.

Sinatra: 29 Links and Resources For A Quicker, Easier Way to Build Webapps

HAML reference

Rack

Mini reviews of 19 Ruby template engines

The Forgotten Ruby Web Frameworks

I don’t really consider myself a Python developer, although I have dabbled. As with many of the less mainstream languages there are keen developers and thinkers trying to clear away the confusion and push the limits of what is possible.

Snakes on the Web

The article is an enthusiastic call to action for “Pythonistas”, but also a really useful summary of big problems and issues applicable to any language, framework or development approach. Read it, even if just for the excellent (and scary) summary of all the things a modern web application developer needs to be aware of.

I have seen several attempts to implement an agile approach to software development within large organizations, and in many  (if not all) of those cases the end result has not been very compelling. This is in direct contrast to agile adoption in smaller, typically single-team, companies.

It’s natural enough to assume that problems with implementing agile processes in large organizations stem from some characteristic of the organizations themselves. Amr Elssamadisy, however, has a different suggestion. Perhaps some of the problems stem from the use of external agile “coaches”, and the way that they affect the way both the agile team and the organization operates simply by being there.

Read more at: InfoQ: Opinion: Agile Coaches Frequently a Source of Adoption Problems.

An excellent point from Andy Palmer. Code which starts out too neat and ordered can make refactoring too difficult. Throw it all in and let the organization emerge from the functionality.

Andy Palmer » Code should start out messy.

I love this kind of analogy. Thinking and learning about one area by considering another is a very powerful technique for boosting overall understanding and wisdom. In this case, Steve Freeman has scraped off the climbing terminology from a snippet of an article and replaced it with jargon from software development.

Thought-provoking.

Tips on Test-Driven Development | Steve Freeman

I use Test-Driven Development (TDD) every day, and find it very helpful. It can be hard to get to grips with, though. I was pleased to read that acceptance-testing pundit Gojko Adzic had fun with some very strict TDD rules.

Gojko Adzic » Thought-provoking TDD exercise at the Software Craftsmanship conference.

My approach to TDD is really quite similar to these strict rules. I’ll admit that I do (sometmes) cut a few corners. The biggest aid to getting the most from TDD is definately good refactoring tools. Without that the whole thing becomes so much more clumsy.

TDD is very comfortable with Java and Eclipse. Recently I have moved out of that comfort zone into some fairly deep JavaScript work, and really miss the ability to automatically extract or move methods etc.

Now this is an interesting result. It appears that even when running in a virtual machine, Linux can be a faster development system than the Windows the VM is running on!

Linux VirtualBox vs Windows for Rails Dev – James Crisp.