Frank Carver's musings about software and life
There has been a lot of speculation and worry about Oracle’s acquisition of Sun, so it’s nice to see some positive aspects. Here’s one about an improved attitude to automated testing.
One of the results of the Oracle purchase of Sun has been an increased focus on testing — not that we didn’t test GlassFish before, but it was mostly manual in my area of the server.
from Jason Lee’s blog Coming Up for Air.
A conference presentation from RubyFringe, designed to be contentious. There are some good points, particularly about the way that different approaches to testing can complement each other, but I think he misses the point about TDD when he lumps it in with developer unit testing and ignores the design aspects of the technique.
I sometimes have to suppress a shudder when people use the term “best practice”. Despite a positive sounding name, the idea of “best practice” is almost always used in a way that is restrictive rather than enabling. Declaring one approach or solution as “best practice” by implication shuts out other answers.
I will admit that for some (very narrow) fields there can be a common understanding of the one best way to do something, but this is often so well understood that it does not even get a name. Walking on your feet is generally better than walking on your hands or knees, for example, but I have never met anyone who referred to foot-walking as “best practice”.
In my world of software development, where the landscape changes at a moment’s notice, naming something as “best practice” is tantamount to declaring it obsolete. Yet large numbers of software developers still numbly follow the its lead.
James Bach tackles a similar issue in testing James Bach’s Blog » Blog Archive » The Great Implication of Context-Driven Methodology.
Every now and then we discuss ways of better automating the manual tests which accompany our web applications. This is especially poignant right now as both the development and test teams have been recently reduced in size. We have had some success with Watir in the past, but it was always dependent on Internet Explorer and Windows. So it’s cool to read that there is now an equivalent for Chrome.
ThoughtWorkers on Open Source: Announcing ChromeWatir release 1.0.0
I love this idea. Pseudo-random test cases to simulate mass usage by “realistic but sometimes thoughtless” users. Especially when augmented with audit trails so that failures can be easily repeated for fixing.
A hot topic in software development circles at the moment is the interaction and demarcation between “developers” and “testers”. Development uses an agile approach but it’s sometimes hard to see how this sits with the testing folks, particularly as most stories seem to move snappily through development then pile up in testing. Sometimes it seems as if a team needs more testers. Sometimes it seems as if a team should reduce or dispense with testers altogether. Sometimes it seems as if roles and responsibilities should change completely.
I wish I knew an answer to this, but at least it is encouraging that others are also considering and writing about these issues.
InfoQ: The Correct Ratio of Agile Testers to Developers? It Depends.
The British Computer Society (BCS) is supposedly the professional institution in the UK which represents anyone working in the field of Information Technology (IT). I have been an associate member for many years, and most years I consider upgrading my membership to become a full member but have never actually done so. Usually the problem I face is that, despite having worked in software development for at least 15 years, I don’t actually know anyone else in the society to act as my sponsor.
This year, though, I face a different problem – my disappointment with the attitude of the society, particularly as expressed in two recent articles, published on their web site and notified to members via an email “magazine”.
The two articles in question are:
I urge you to read them and make up your own mind.
My problem with both of these articles is that they seem to express a one-sided attitude which I had hoped had been banished from the BCS many years ago.
When I first encountered the BCS (during my university time in the mid 1980s) it was seen as the bastion of IT middle management, and in particular middle management in large corporate environments. There seemed little or no representation of or for the people actually doing the work of producing and maintaining information systems, and an almost pathological ignorance of the concerns of contractors and people working for small businesses.
Over the years, my concerns waned. The society seemed to re-invent itself to become more in-line with its charter and thus more inclusive of software developers and contractors. It still struggled with the idea of small IT businesses, and of small departments or lone IT workers in other organisations, but I had hopes that this would improve, too.
But then I received these two articles, and all my concerns about the emphasis of the society came flooding back. Both articles appear riddled with the view that developers are lazy and selfish creatures who will always choose to produce rubbish unless forced by managers or testers. This is an appalling stereotypical slur, and completely at odds with the stated, inclusive, intent of the society.
I understand that both these articles were published through the society’s “blog” initiative, and thus have not had the editorial oversight that one might expect from a more formal publication. However, this does not exempt them from reflecting on the BCS. Since reading these articles a few days ago I have already had one forwarded to me by a colleague. The damage has been done.
Maybe I won’t be upgrading my membership this year, either.
Our agile team is finding some things challenging. In particular deciding how to prioritise and work on “bugs” in the midst of a pool of prioritised and scheduled feature stories.
“Agile In Action” has a nice summary of an approach to software development. Most agile practitioners won’t find anything to object to.
AGILE IN ACTION: The Energized Way
Our first problem is with the second bullet “Work with clients every day“. As a team we would love to work with clients every day, but there seems to be a thick layer of representatives and proxies in between us and real customers. This is made especially difficult as we are currently serving the needs of several customers with a single product, and resolving customer differences is proving tricky.
Our second problem is with “Fix defects as soon as they’re discovered“. In principle this seems obvious, but the trouble we are having rests on the definition of a defect. As an agile team we keep up-front specification to a minimum, and in-effect treat every delivery as a prototype ready for customer feedback. Plenty of people in the company have opinions on such prototypes – things they think it should do, things they think it should not do, and things they think it does wrong. Any of these could be considered as defects (and indeed many of them are raised in our bug tracking system.) If we stopped new work to make all these changes we would (a) greatly reduce our feature velocity, (b) bypass the prioritisation process used to “Deliver the client’s highest-value stuff first“, and (c) leave us stuck in a mire of conflicting opinions.
We certainly do not ever want to deliver “broken” software, but its a fact of life that some “bugs” are lower in priority than others. Some “bugs” are also lower in priority than new features, but this is more of a business decision than a development decision. Working out how deliver prompt, appropriate and minimal software in the face of such a slew of opinions is proving contentious.
I’d be interested in reading any suggestions or answers to these problems.
This looks like an interesting project. I’m slightly worried by the way that it seems to embody the one class === one test assumption, but if that doesn’t get in the way of other forms of unit testing it could be useful.
I’m currently working on some software which sends notifications to users (using SMS, email, or whatever) and have faced the inevitable problems with testing it. On balance I’d prefer not to receive a test SMS on my mobile phone every time our continuous integration system runs an end-to-end test.
Gojko Adzic has some thoughts about how to make such a system more straightforward to test.
I develop almost all my code these days using Test Driven Development (TDD). By taking this approach I never bother about unit test code coverage, but some developers seem very concerned by it.
One thing that I have never considered doing is retroactively adding tests just to increase some code-coverage metric. Dan Manges has written in interesting explanation of why that might be, using some examples in Ruby.
As we attempt to spread agile, continual, iterative, processes throughout the company, there is a growing confusion as to when things are actually ready for release. The old “waterfall” notion of a complex and detailed advance specification, backed up by months of laborious manual testing is no longer applicable.
In most ways this is a very good thing. A whole lot of unnecessary rigidity and busy-work has been removed. However, one aspect we miss is some sort of final “quality gate” to approve a release. We are in danger of ending up either with a queue of potential releases jostling for testing priority, with none of them ever actually being released before they are overtaken by a new candidate, or with untested and potentially broken builds being released to live deployments.
This is obviously also on other people’s minds, too. InfoQ has recently published an article on “The Power of Done” which summarises some attempts to chart an agile course through this dangerous situation.
As software testing spreads out in scope from the old notion of manual exercising of a system into areas such as developer unit tests and automated acceptance tests, the issue of ownership becomes more important. In this short and pithy post Kristan Vingrys states his opinion.
I’ll put my cards on the table. At heart I am a “classic TDD” guy. My first attempt at any new code is to test return values or side-effects rather than immediately jump to setting up mock objects and testing interactions. Although I do use mock objects as a testing technique from time to time I have never liked any of the popular mock frameworks. The nearest I have found to what I would like is probably Mockito, but even so I still use my own recording framework.
This slightly provocative post examines the role of QA (a.k.a testing) in various view of a project lifecycle and considers how we might react to a QA team which reported no bugs.
A fascinating counterpoint to Gojko Adzic‘s writings on acceptance testing in an agile process.
thekua.com@work » Automated story-based acceptance tests lead to unmaintainable systems
Update: here’s some more discussion on this topic, and how it is affected by the nature of user stories
User Stories are Just Schedulable Change
An article with some good tips on avoiding a clash of cultures between developers and testers on an agile project. (note, there may be a click-through ad on this link)
It seems a comon problem. One of the first steps in implementing an agile process (such as scrum) is to put in place a fixed iteration cycle in development, but but then find difficulties fitting post-development testing (a.k.a “system testing” or “integration”) into the mix.
The main problem with testing after development is that any fixes to problems identified in the follow-on testing have a tendency to become tangled in upcoming work for the next iteration. This in turn can lead to no iteration ever being “clean” enough to actually release.
One possibility for a solution is to reconsider the development-led nature of an iteration, and instead lead it from a set of acceptance tests which are estimated, developed and executed alongside the scheduling and implementation of code changes.
Gojko Adzic, one of my current favourite bloggers on the subject of testing, has attempted to map out an iteration structure which incorporates this approach:
Gojko Adzic » Fitting agile acceptance testing into the development process
This looks interesting. The role of testing (at least, testing beyond unit-level regression testing and TDD) is the subject of much discussion in software development at the moment. Maybe I should actually attend this talk at Skills Matter in London on 18 September.
When I first saw this it looked great: a ruby REST wrapper which supports a lot of useful test and integration possibilities. However, the deeper I looked, the more disappointed I became. I’m now saddened to believe that this is based on yet another misunderstanding of what REST is.
As far as I can tell, the ActiveResource concept, on which this approach is based, is merely an attempt to impose a constrained CRUD data-model over HTTP. There is no concept of content negotiation – all data is XML according to pre-assumed schemas. There is no support for automatic discovery and use of hrefs between resources. There is even the suggestion that it’s a good idea to casually extend the HTTP protocol with extra custom methods.
All of these are typical problems found in projects which use the name REST without the key bits which make it really work. The end result is just another fragile, application-specific RPC protocol. Sigh.
nutrun » Blog Archive » Testing web services with ActiveResource