Frank Carver's Punch Barrel

I love Test Driven Development (TDD). If you look back through the posts on this blog that soon becomes apparent. I’m pretty comfortable with using TDD techniques at all levels of a solution, from the tiniest code snippet to multiply-redundant collaborating systems. Of course, the difficulty of actually coding the tests in a test-driven design can vary widely based on the context and complexity of the interaction being tested, and everyone has a threshold at which they decide that the effort is not worth it.

The big difference between testing small code snippets and large systems is the combinatorial growth of test cases. To fully test at a higher (or perhaps more appropriately, outer) layer would require an effectively infinite number of tests. By a happy co-incidence, though, if you have good test coverage at lower/inner layers of the code, you can rely on them to do their job, so it’s only needed to test the additional behaviour at each layer. Even with this simplifying assumption the problem does not fully go away. Very often the nature of outer layers implies a broader range of inputs and outputs, and a greater dependence on internal state from previous actions. It can still seem to need a ridiculous amount of tests to properly cover a complex application. And worst of all, these tests are often boring. Slogging through acres of largely similar sequences, differing only in small details, can be a real turn off for developers, so once again, it can feel that the effort is not worth it.

If fully covering the outer layers of a system, with tests for every combination of data, every sequence of actions, and every broken configuration is prohibitively expensive, time-consuming and downright boring, then it makes sense to be smart about it, and prioritise those tests which have the highest value. Value in this sense is a loose term encompassing cost of failures, importance of features, and scale of use. A scenario with a high business value if successful, a high cost of failure, which is used often by large numbers of people would be an obvious choice for an outer layer test. Something of little value which nobody cares about and which is hardly ever used would be much further down the list.

And this leads neatly to the concept of automated “functional testing”. Functional tests being outer layer tests which exercise these important, valuable interactions with the system as a whole, Arguably there is a qualitative difference between unit tests of internal components and functional tests of outer layers. Internal components have an internal function, and relate mostly with other internal components and their interactions are designed by developers for code purposes. This makes it relatively easy for developers to decide what to test, so specifying tests in similar language to the implementation code is a straightforward and effective way to get the tests written. Outer layers and whole applications have an external function, and business drivers for what they do and how they do it. This can be less amenable to describing tests in the language used by programmers. Add to this the potentially tedious nature of these external, functional, tests and its easy to see why they sometimes get overlooked, despite their potentially high business value.

Many attempts have been made over the years to come up with ways to get users and business experts involved in writing and maintaining external functional test cases. My friend, some-time colleague and agile expert Steve Cresswell has recently blogged about a comparison of Behaviour-Driven Design (BDD) tools. It’s an interesting article, but I can’t help thinking that there is another important dimension to these tools which also needs to be unpicked.

Along this dimension, testing tools range from “raw program code” (with no assistance from a framework); through “library-style” frameworks which just add some extra features to a programming language using its existing extension mechanisms; “internal DSL” (Domain Specific Language) frameworks which use language meta-programming features to re-work the language into something more suitable for expressing business test cases; “external languages” which are parsed (and typically interpreted, but compilation is also an option) by some specialist software; through to “non-textual” tools where the test specification is stored and managed in another way, for example by recording activity, or entering details in a spreadsheet.

In my experience, most “TDD” tools (JUnit and the like) sit comfortably in the “library-style” group. test case specification is done in a general-purpose programming language, with additions to help with common test activities such as assertions and collecting test statistics. Likewise, most “BDD” tools (Cucumber and the like) are largely in the “external language” group. This is slightly complicated by the need to drop “down” to a general-purpose language for the details of particular actions and assertions.

Aside from the “raw program code” option which, by definition, has no frameworks, the great bulk of test tools occupy the “library style” and “external language” groups, with a small but significant number of tools in the “non-textual” group. I find it somewhat surprising how few there seem to be in the “internal DSL” category, especially given how popular internal DSL approaches are for other domains such as web applications. There are some, of course, (coulda, for example) which claim to be internal DSLs for BDD-style testing, but there is still a more subtle issue with these and with many external languages.

The biggest problem I have with most BDD test frameworks is related to the concept of “comments” in programming languages. Once upon a time, when I was new to programming, it was widely assumed that adding comments to code was very important. I recall several courses which stressed this so much that a significant point of the mark was dependent on comments. Student (and by implication, junior developer) code would sometimes contain more comments than executable code, just to be on the safe side. Over the years it seems that the prevailing wisdom has changed. Sure, there are still books and courses which emphasise comments, but many experienced developers try to avoid the need for comments wherever possible, using techniques such as extracting blocks of code to named methods and grouping free-floating variables into semantically meaningful data structures, as well as brute-force approaches such as just deleting comments.

The move away from comments has developed gradually, as more and more programmers have found themselves working on code produced by the above mentioned comment-happy processes. The more you do this, the more you realise that the natural churn and change of code has an unpleasant effect on comments. By their nature comments are not part of the executable code (I am specifically excluding “parsed comments” which function like hints to a compiler or interpreter here). This in turn means that comments can not be tested, and thus cannot automatically be protected against errors and regressions. Add to this the common pressure to get the code working as quickly and efficiently as possible, you can see that comments often go unchanged even when the code being commented is radically different. This effect then snowballs – the more that comments get out of step with the code, the less value they provide, and so the less effort is made to update them. Soon enough most (or all!) comments are at best a useless waste of space, and at worst dangerously misleading.

What does this have to do with BDD languages? If you look in detail at examples of statements in many BDD languages, they have a general form something like keyword "some literal string or regular expression" Typical keywords are things like “Given”, “When”, “Then”, “And”, and so on. For example: Given a payment server with support for Visa and Mastercard. This seems lovely and friendly, phased in business terms. But let’s dig into how this is commonly implemented. Very likely, somewhere in the code will be a “step” definition.

An example from spinach, which is written in Ruby, might be:

  step 'a payment server with support for Visa and Mastercard' do
    @server = deploy_server(:payment, 8099)
    @server.add_module(@@modules[:visa])
    @server.add_module(@@modules[:mc])
    @server.start
  end

This also looks lovely, linking the business terminology to concrete program actions in a nice simple manner. However, this apparent simplicity hides the fact that the textual step name has no actual relationship to the concrete code. Sure, the text is used in at least two places, so it seems as if the system is on the case to prevent typos and accidental mis-edits, but it still says nothing about what the step code actually does. As an extreme example, suppose we changed the step code to be:

  step 'a payment server with support for Visa and Mastercard' do
    Dir.foreach('/') {|f| File.delete(f) if f != '.' && f != '..'}
  end

The test specifications in business language would still be sensible, but running the tests would potentially delete a whole heap of files!

For a less extreme example, imagine that the system has grown a broad BDD test suite, with many cases which use “Given a payment server with support for Visa and Mastercard”. Now we need to add support for PayPal. There are several options including:

• copy the existing step to a new one with a different name and an extra line to install a PayPal module (including going through all the test code to decide which tests should use the old step and which should use the new one
• add the extra line to the existing step and modify the name to include PayPal then change all the references to the new name
• or just add the extra module and leave the step name the same.

The last option happens more often than you might think. Just as with the comments, the BDD step name is not part of the test code, and is not itself tested, so there is nothing in the system to keep it in step with the implementation. And the further this goes, the less the business language of the test specifications can be trusted.

I have worked on a few projects which used these BDD approaches, and all of them fell foul of this problem to some degree. Once this “rot” takes hold, it seems almost inevitable that the BDD specifications either become just another set of “unit” tests, only understood by developers who can dig into the steps to work out what is really happening, or they are progressively abandoned. It seems unlikely that many projects would be willing to take on the costs of going through a large test suite and checking a bunch arbitrary text that’s not part of the deliverable, just to see if it makes sense.

I wonder if the main reason that this is not seen to be more of a problem is that so few projects have progressed with BDD through several iterations of the system, and are still using the approach.

So, is there any way out of this trap? For some cases, using regular expressions for step name matching can help. This can provide a form of parameter-passing to steps, increasing reuse for multiple purposes and reducing churn in the test specifications. This does not solve the overall problem, though, as it still has chunks of un-parsed literal text. For that we would need a more sweeping change.

Which brings me back to internal and external DSLs. To my mind, the only way to address this issue in the longer term is to define the test specifications in a much more flexible language, one which suits both the domain being tested and the domain of testing itself. My aim would be to avoid the need for those comment-like chunks of un-parsed literal text, and align the programming language with the business language well enough that expressions in the business language make sense as programming constructs. If done well, this should allow test specifications to be changed at a business level and still be a valid and correct expression of the desired actions and assertions.

Although some modern programming languages are relatively adept in metaprogramming and internal DSLs (Ruby is well-known for this), mostly they don’t go far enough. Arguably a better choice would be to base the DSL on one of the languages which have very little by way of their own syntax, and thus are more flexible in adapting to a domain. Languages such as LISP and FORTH have hardly any syntax, and have both been used for describing complex domains. It used to be said of FORTH that the task of programming in forth was mainly one of writing a language in which the solution is trivial to express.

I’m afraid I don’t have a final answer to this, though. I have no tool, language or framework to sell, just a hope that somebody, somewhere, is working on this kind of next-generation BDD approach.

For more information about low-syntax languages, you might want to read some of my articles on Raspberry Alpha Omega, such as What is a High Level Language and More Language Thoughts and Delimter-Free Languages.

Also, for Paul Marrington’s take on internal vs external DSLs for testing, see BDD – Use a DSL or natural language?.

I just watched an interesting and entertaining talk about behavior-driven development (BDD or ATDD). Gojko Adzic – BDD: Busting the myths on Vimeo on Vimeo

Well worth a look, particularly if you are worried that BDD or automated acceptance tests do not make sense, or are not helping your business.

test-Driven development (TDD) is a very powerful and beneficial approach to developing software. I routinely recommend it as a solution to untested and hard-to maintain software. However, every rule has its exceptions and there are occasions when a pure TDD approach is not very helpful.

Mark Needham wrote a recent blog post on the subject where he describes a kind of “Spike Driven Development”: Coding: Spike Driven Development at Mark Needham.

I tend to think of this as some sort of experiment-driven development. I start by choosing a goal. Typically this is either finding out about some murky or unknown aspect of a system such as interfacing with a third party, or improving a variable quantity such as performance. This in turn leads to thinking about ways to assess whether we have done enough to meet the goal and how much time to allocate to the exercise. Finally, a period of playing, following where each trial leads. The important bit is to repeatedly check progress against the goal – that’s what links it to the aspects of TDD which work well.

It’s happened on most reasonable sized projects I have worked on. The benefits of test coverage an continuous integration are obvious and pay back immediately. But, somehow, as the project grows and diversifies, a point is reached where the complexity and run time of the CI process begins to slow down development rather than assist it.

Jez Humble has put together some interesting thoughts on how to deal with this issue. Read more at Deployment pipeline anti-patterns.

There has been a lot of speculation and worry about Oracle’s acquisition of Sun, so it’s nice to see some positive aspects. Here’s one about an improved attitude to automated testing.

One of the results of the Oracle purchase of Sun has been an increased focus on testing — not that we didn’t test GlassFish before, but it was mostly manual in my area of the server.

from Jason Lee’s blog Coming Up for Air.

A conference presentation from RubyFringe, designed to be contentious. There are some good points, particularly about the way that different approaches to testing can complement each other, but I think he misses the point about TDD when he lumps it in with developer unit testing and ignores the design aspects of the technique.

InfoQ: Testing is Overrated.

I sometimes have to suppress a shudder when people use the term “best practice”. Despite a positive sounding name, the idea of “best practice” is almost always used in a way that is restrictive rather than enabling. Declaring one approach or solution as “best practice” by implication shuts out other answers.

I will admit that for some (very narrow) fields there can be a common understanding of the one best way to do something, but this is often so well understood that it does not even get a name. Walking on your feet is generally better than walking on your hands or knees, for example, but I have never met anyone who referred to foot-walking as “best practice”.

In my world of software development, where the landscape changes at a moment’s notice, naming something as “best practice” is tantamount to declaring it obsolete. Yet large numbers of software developers still numbly follow the its lead.

James Bach tackles a similar issue in testing James Bach’s Blog » Blog Archive » The Great Implication of Context-Driven Methodology.

Every now and then we discuss ways of better automating the manual tests which accompany our web applications. This is especially poignant right now as both the development and test teams have been recently reduced in size. We have had some success with Watir in the past, but it was always dependent on Internet Explorer and Windows. So it’s cool to read that there is now an equivalent for Chrome.

ThoughtWorkers on Open Source: Announcing ChromeWatir release 1.0.0

I love this idea. Pseudo-random test cases to simulate mass usage by “realistic but sometimes thoughtless” users. Especially when augmented with audit trails so that failures can be easily repeated for fixing.

DanNorth.net » Monkey business value

This impatient monkey – dancingmango

A hot topic in software development circles at the moment is the interaction and demarcation between “developers” and “testers”. Development uses an agile approach but it’s sometimes hard to see how this sits with the testing folks, particularly as most stories seem to move snappily through development then pile up in testing. Sometimes it seems as if a team needs more testers. Sometimes it seems as if a team should reduce or dispense with testers altogether. Sometimes it seems as if roles and responsibilities should change completely.

I wish I knew an answer to this, but at least it is encouraging that others are also considering and writing about these issues.

InfoQ: The Correct Ratio of Agile Testers to Developers? It Depends.

The British Computer Society (BCS) is supposedly the professional institution in the UK which represents anyone working in the field of Information Technology (IT). I have been an associate member for many years, and most years I consider upgrading my membership to become a full member but have never actually done so. Usually the problem I face is that, despite having worked in software development for at least 15 years, I don’t actually know anyone else in the society to act as my sponsor.

This year, though, I face a different problem – my disappointment with the attitude of the society, particularly as expressed in two recent articles, published on their web site and notified to members via an email “magazine”.

The two articles in question are:

• Employee-driven Corporate Change – Reverse Ludditism?
• ‘Strictly Testing Only’, please – Agile doesn’t work!

I urge you to read them and make up your own mind.

My problem with both of these articles is that they seem to express a one-sided attitude which I had hoped had been banished from the BCS many years ago.

When I first encountered the BCS (during my university time in the mid 1980s) it was seen as the bastion of IT middle management, and in particular middle management in large corporate environments. There seemed little or no representation of or for the people actually doing the work of producing and maintaining information systems, and an almost pathological ignorance of the concerns of contractors and people working for small businesses.

Over the years, my concerns waned. The society seemed to re-invent itself to become more in-line with its charter and thus more inclusive of software developers and contractors. It still struggled with the idea of small IT businesses, and of small departments or lone IT workers in other organisations, but I had hopes that this would improve, too.

But then I received these two articles, and all my concerns about the emphasis of the society came flooding back. Both articles appear riddled with the view that developers are lazy and selfish creatures who will always choose to produce rubbish unless forced by managers or testers. This is an appalling stereotypical slur, and completely at odds with the stated, inclusive, intent of the society.

I understand that both these articles were published through the society’s “blog” initiative, and thus have not had the editorial oversight that one might expect from a more formal publication. However, this does not exempt them from reflecting on the BCS. Since reading these articles a few days ago I have already had one forwarded to me by a colleague. The damage has been done.

Maybe I won’t be upgrading my membership this year, either.

Our agile team is finding some things challenging. In particular deciding how to prioritise and work on “bugs” in the midst of a pool of prioritised and scheduled feature stories.

“Agile In Action” has a nice summary of an approach to software development. Most agile practitioners won’t find anything to object to.

AGILE IN ACTION: The Energized Way

Our first problem is with the second bullet “Work with clients every day“. As a team we would love to work with clients every day, but there seems to be a thick layer of representatives and proxies in between us and real customers. This is made especially difficult as we are currently serving the needs of several customers with a single product, and resolving customer differences is proving tricky.

Our second problem is with “Fix defects as soon as they’re discovered“. In principle this seems obvious, but the trouble we are having rests on the definition of a defect. As an agile team we keep up-front specification to a minimum, and in-effect treat every delivery as a prototype ready for customer feedback. Plenty of people in the company have opinions on such prototypes – things they think it should do, things they think it should not do, and things they think it does wrong. Any of these could be considered as defects (and indeed many of them are raised in our bug tracking system.) If we stopped new work to make all these changes we would (a) greatly reduce our feature velocity, (b) bypass the prioritisation process used to “Deliver the client’s highest-value stuff first“, and (c) leave us stuck in a mire of conflicting opinions.

We certainly do not ever want to deliver “broken” software, but its a fact of life that some “bugs” are lower in priority than others. Some “bugs” are also lower in priority than new features, but this is more of a business decision than a development decision. Working out how deliver prompt, appropriate and minimal software in the face of such a slew of opinions is proving contentious.

I’d be interested in reading any suggestions or answers to these problems.

This looks like an interesting project. I’m slightly worried by the way that it seems to embody the one class === one test assumption, but if that doesn’t get in the way of other forms of unit testing it could be useful.

moreUnit – Home

I’m currently working on some software which sends notifications to users (using SMS, email, or whatever) and have faced the inevitable problems with testing it. On balance I’d prefer not to receive a test SMS on my mobile phone every time our continuous integration system runs an end-to-end test.

Gojko Adzic has some thoughts about how to make such a system more straightforward to test.

Gojko Adzic » How to test e-mail notifications properly

I develop almost all my code these days using Test Driven Development (TDD). By taking this approach I never bother about unit test code coverage, but some developers seem very concerned by it.

One thing that I have never considered doing is retroactively adding tests just to increase some code-coverage metric. Dan Manges has written in interesting explanation of why that might be, using some examples in Ruby.

Dan Manges’s Blog – Increasing Code Coverage May Be Harmful

As we attempt to spread agile, continual, iterative, processes throughout the company, there is a growing confusion as to when things are actually ready for release. The old “waterfall” notion of a complex and detailed advance specification, backed up by months of laborious manual testing is no longer applicable.

In most ways this is a very good thing. A whole lot of unnecessary rigidity and busy-work has been removed. However, one aspect we miss is some sort of final “quality gate” to approve a release. We are in danger of ending up either with a queue of potential releases jostling for testing priority, with none of them ever actually being released before they are overtaken by a new candidate, or with untested and potentially broken builds being released to live deployments.

This is obviously also on other people’s minds, too. InfoQ has recently published an article on “The Power of Done” which summarises some attempts to chart an agile course through this dangerous situation.

InfoQ: The Power of Done

As software testing spreads out in scope from the old notion of manual exercising of a system into areas such as developer unit tests and automated acceptance tests, the issue of ownership becomes more important. In this short and pithy post Kristan Vingrys states his opinion.

Vinktank | Who owns automated acceptance tests?

I’ll put my cards on the table. At heart I am a “classic TDD” guy. My first attempt at any new code is to test return values or side-effects rather than immediately jump to setting up mock objects and testing interactions. Although I do use mock objects as a testing technique from time to time I have never liked any of the popular mock frameworks. The nearest I have found to what I would like is probably Mockito, but even so I still use my own recording framework.

thekua.com@work » The world mocks too much

This slightly provocative post examines the role of QA (a.k.a testing) in various view of a project lifecycle and considers how we might react to a QA team which reported no bugs.

Ryan Greenhall: What is The Role of QA?

A fascinating counterpoint to Gojko Adzic‘s writings on acceptance testing in an agile process.

thekua.com@work » Automated story-based acceptance tests lead to unmaintainable systems

Update: here’s some more discussion on this topic, and how it is affected by the nature of user stories

User Stories are Just Schedulable Change