However, it has been almost impossible to take that expertise, hard work, and custom code and easily/cheaply turn it into usable web applications for general public use. Low-cost hosting providers have generally shunned Java support. The most reasonable one I have found is LunarPages who actually support both ad-hoc JSP and deployment of custom web applications as war files, even though you have to search pretty deep in their web site to find out, and pay an extra dollar or two a month for the privilege.

So for my own projects I have been looking for an alternative for a long time. A way of developing and testing web apps on my various development boxes (currently running Ubuntu, MacOS X and Windows XP) and easily deploying to a low-cost hosting provider.

By far the most popular web development language is PHP. It’s available pretty much everywhere. It’s so focussed on web app development that the primary unit of coding is the web page. Believe me, I have tried to like PHP, but it’s just so clumsy. After a few paces the application begins to get tangled, development speed drops, and bugs creep in. Its web focus makes it tricky to unit test, and end-to-end testing seems to require a full-fat HTTP server. Not for me.

Beyond the traditional stomping ground of PHP, perl, and other CGI fodder is the new range of “trendy” languages. Ruby, Python, Erlang and even the venerable smalltalk are trying to position themselves as the thinking-person’s web development tool.

Erlang syntax is a bit too odd for me right now, although I may come back to it later. Smalltalk is interesting, but carries with it so much history and is hardly a popular choice for low-cost hosting providers. That leaves a short-list of Python and Ruby.

As languages I like both Python and Ruby. They have broadly similar design goals, both have keen developer communities with plenty of open source resources, and both are commonly found on linux-based web hosts. In order to decide between them I took a look around their frameworks and APIs.

Python has Google on its side. It’s the “native language” of the hugely-scalable Google AppEngine. However, a scan around the web looking for ways of writing web applications led largely to two: Zope and Django. (for balance, there are plenty of others, but these are the standout examples). Zope is old and sprawling. It has some great ideas but is hardly an obvious choice for small, tactical web apps. Django is a bit lighter, but still seems to assume a lot and require a lot of relatively fiddly config.

In the Ruby world the blindingly obvious choice for a web application framework is Rails. Sometimes it seems as if “Ruby” is just another way of saying “Rails”. However Rails, like Django, assumes a lot about the eventual application. Rails also makes a lot of use of code generation, which I simply do not like. I would always prefer that a framework eliminate boilerplate rather than just generate it for me. Looking a little further beyond Rails I came across some really interesting alternatives, and the one which really sparked my interest is Sinatra. For me this one framework made the difference. It’s so streamlined that a basic web app is as simple as:

require ‘sinatra’
get ‘/’ do
‘Hello world!’
end

Best of all, simply running the above file using ruby hello.rb starts up a web server and begins serving pages on port 4567. No extra config or faffing.

Looking around for low cost hosting I found that the provider I already use (Dreamhost) supports Ruby web hosting using Passenger. An ssh to the Dreamhost server for “gem install sinatra” followed by a little bit of FTP and my first Ruby/Sinatra application was live!

A key part of any web application is the pages, and a key part of generating pages is a good template language. As a general-purpose templating language, I still prefer my own Stringtree Templater, part of the Mojasef Java framework. So far I have not found a similar templating solution for any other language. However, for the limited and specific requirement of generating web pages, I am becoming quite fond of Haml. It’s not at all a general templating solution, but it does massively simplify the generation of web pages.

Finally, after years of CVS and Subversion, I have made the jump to distributed version control. Not only is it more trendy, but the ability to work on multiple code branches, on multiple machines, with or without an internet connection, and easily share, identify and merge when necessary has been a key benefit. Occasionally I wonder if one of the other distributed VCS (Mercurial, Bazaar, darcs, etc.) would have been a better choice, but I needed to settle on one and get to grips with it. So I chose git.

There have been a few speed-bumps. Attempting to learn git from manual entries and simplistic tutorials gives very little help on deciding what is worth doing. After about a week of following the techniques in A Git Workflow for Agile Teams, though, I began to get comfortable. Add to that some lovely web deployment tricks from A web-focused Git workflow and a selection of GUI tools ( Git Cola, GitX, and Git GUI ) to get round all that tedious “git add” stuff and simplify infrequent operations, and git becomes a very useful and productive tool.

The upshot is that I currently feel very productive.

My toolbox has a few other things in it, but they are still candidates for change if I find anything better. I am managing my projects and git repositories using Unfuddle but it has a few shortcomings including a ticket system which is a pale and clumsy cousin of bugzilla. For editing I use generic syntax-highlighting text editors (gedit on Linux, TextEdit on Windows and Smultron on Mac) but none of them have the nice autocompletion and refactoring tools I grew addicted to in Eclipse. Suggestions welcome!

Finally, a few spare links relating to the tools mentioned above.

Sinatra: 29 Links and Resources For A Quicker, Easier Way to Build Webapps

HAML reference

Rack

Mini reviews of 19 Ruby template engines

The Forgotten Ruby Web Frameworks

Since then I have progressively realized that I may have been a bit optimistic. A few odd bugs have been found in the 1.4 code, and a few key improvements have been made to the Java 5 code which by rights should also be implemented for 1.4. So I have decided to try and fix the situation.

As of today, the tag “1.4-final” in both Stringtree and Mojasef should be considered as deprecated, and any code which used to use that tagged version should instead use the branches (“Stringtree for Java 1.4″ and “Mojasef for Java 1.4″) I have just created. Initially, the code is identical, but the plan is to go through and apply known bug fixes and fully-backward-compatible internal improvements to the branched versions.

Stephen Colebourne’s Weblog

First, let’s set the scene. In TDD the rules are very simple: No production code is written without a failing test; just enough code is written to satisfy the current set of tests; duplication is mercilessly refactored. TDD is a very powerful and effective technique which can improve both the quality and delivery time of software. HTTP and REST interfaces provide access to a system or component using the HTTP protocol. This access may be by fetching HTML or WML pages, for example, or it may be by sending, receiving and updating resources in some other format such as XML or JSON.

For a first scenario imagine we have an existing HTTP interface for which we want to write a client using TDD. This service is relatively simple to start with. To send a message requires a POST request to a particular URL with two parameters “destination” and “text”. If the destination is valid, the text message is sent to the destination and a “200″ status code is returned. If the destination is not recognized, a “404″ status code is returned. This interface is so simple that it is tempting to just write the client, test it manually and move on. But if we were to do that, then we might not think of all the test case possibilities, and there would be no regression tests to help protect the software from unexpected side-effects later. So we opt to do the right thing, and use TDD.

For the sake of simplicity, I will assume we are using the Stringtree HTTPClient as described in an earlier post. So let’s start with a simple test to make sure the code compiles and the test libraries and stuff are in place:

import org.stringtree.http.*;
 
public class MessageSendingTest extends junit.framework.TestCase {
  public void testExceptionFromUnknownURL() {
    Form form = new Form();
    form.put("destination", "name@example.com");
    form.put("text", "hello from HTTP");
    HTTPClient client = new HTTPClient();
    try {
      client.post("http://localhost:9999/send", form);
      fail("should throw an exception");
    } catch (Exception e) {}
  }
}

This test serves two purposes. First, compiling it helps drive the structure of the test case, what classes need to be imported and so on. Second it documents the expected behaviour of our client if the server is not running or not where it is expected to be. I always like to start with this sort of “null” test case wherever possible. Once we get this test working we are ready to move on to the next step, testing against an actual server.

It might be possible to test against a real server, perhaps by manually examining log files, or checking the actual receipt of a message. Feel free to do that, but understand that that is really integration testing rather than unit testing, and neither helps us design and debug the client, nor document and enforce the proper behaviour of our code once we move on to other work. To gain these benefits we need a server which may be started, examined and stopped quickly and under the control of a simple unit test. This is where Mojasef comes in.

Mojasef is a server and web application framework designed for efficient test-driven development of web applications. Although usually used for building applications, it is just as suitable for building test harnesses. In this case we will build a simple application which implements the HTTP interface described above. And yes, we will do it using TDD, so we leave the MessageSendingTest for the moment and start a new one to drive our test harness without requiring HTTP. As usual, I start with a “null” test to get started:

public class MessageHarnessTest extends junit.framework.TestCase {
  MessageHarness harness;
 
  public void setUp() {
    harness = new MessageHarness();
  }
 
  public void testNoAction() {
    assertFalse(harness.wasCalled());
  }
}

This won’t compile without a MessageHarness class, so let’s make the minimum necessary.

public class MessageHarness {
  public boolean wasCalled() {
    return false;
  }
}

Good, our test now runs, so let’s add some behaviour. First another test:

  public void testCalledWithoutArguments() {
    harness.send();
    assertTrue(harness.wasCalled());
  }

To make this compile we need to add another method to MessageHarness

  public void send() {
  }

Our new test fails, so we need to make it work.

public class MessageHarness {
  private boolean called = false;
 
  public boolean wasCalled() {
    return called;
  }
 
  public void send() {
    called = true;
  }
}

That seems enough to start with. Now back to the main test case. Add a new test to call the application we have just built over HTTP and check that all the communication stuff works. With a bit of refactoring to keep the tests clean and duplication low.

import org.stringtree.http.*;
import org.stringtree.mojasef.standalone.InlineServer;
 
public class MessageSendingTest extends junit.framework.TestCase {
  HTTPClient client;
  Form form;
  InlineServer server;
  MessageHarness app;
 
  public void setUp() throws Exception {
    client = new HTTPClient();
    form = new Form();
    app = new MessageHarness();
    server = new InlineServer(app, "9998");
    server.start();
  }
 
  public void tearDown() {
	server.halt();
  }
 
  public void testExceptionFromUnknownURL() {
    form.put("destination", "name@example.com");
    form.put("text", "hello from HTTP");
    try {
      client.post("http://localhost:9999/send", form);
      fail("should throw an exception");
    } catch (Exception e) {}
  }
 
  public void testHarnessWiring() throws Exception {
    form.put("destination", "unknown@example.com");
    form.put("text", "hello from HTTP");
    assertFalse(app.wasCalled());
    client.post("http://localhost:9998/send", form);
    assertTrue(app.wasCalled());
  }
}

This probably needs a bit of explanation. The MessageHarness class we produced above is a Mojasef web application. Really, without any other configuration, base classes or nonsense. It’s just a “plain old Java object” (POJO). The job of the Mojasef code is to take an object of that class and make its methods available as HTTP URLs.

To serve it over HTTP we could use a MojasefServlet and run it in any Servlet Container, but for this test we want an ultra-light server which can be started and stopped in a test. So we use the InlineServer provided in the Mojasef jar. It’s a real HTTP server which you can use from a regular web browser, but it’s mainly designed for streamlined use in tests.

This server is configured to serve the MessageHarness application on port 9998. Having configured the server we start it in startUp and stop it in tearDown to ensure that nothing is left behind between tests.

The new test makes a HTTP request to the freshly started server, then checks (by calling the MessageHarness method we created earlier) that the test harness application was successfully called. This verifies that the server has been correctly started and called.

It’s still not really testing the message API. To do that we need more tests.

Now, we know from the introduction above that our resource should return a HTTP 404 (“not found”) error if we try to send to an unrecognised address.

  public void test404FromUnknownDestination() throws Exception {
    form.put("destination", "unknown@example.com");
    form.put("text", "hello from HTTP");
    Document result = client.post("http://localhost:9998/send", form);
    assertEquals("404", result.getHeader(HTTPClient.HTTP_RESPONSE_CODE));
  }

Making this test work is pretty simple, especially as we have no other tests which say anything about the HTTP Response code. The only “cleverness” is that we now need to use some slightly smarter features of the Mojasef code.

import org.stringtree.finder.StringKeeper;
import org.stringtree.mojasef.HTTPConstants;
 
public class MessageHarness {
  private boolean called = false;
 
  public boolean wasCalled() {
    return called;
  }
 
  public void send(StringKeeper context) {
    called = true;
    context.put(HTTPConstants.RESPONSE_CODE, "404");
  }
}

Mojasef will try a variety of ways to call a public method. We have seen the basic one – a method with no parameters mapped directly to a URL. This is another one – a method taking a “context” as an argument. This context will be pre-loaded with any POST or URL parameters, as well as any request headers and plenty of other stuff which is not of interest right now. This context also provides a way of setting information to be sent back to the client.

Now, our application code is passing all its tests, but it does so by returning a “404″ error code for all requests. This is not really desirable behaviour, so we need another test:

  public void test200FromKnownDestination() throws Exception {
    form.put("destination", "known@example.com");
    form.put("text", "hello from HTTP");
    Document result = client.post("http://localhost:9998/send", form);
    assertEquals("200", result.getHeader(HTTPClient.HTTP_RESPONSE_CODE));
  }

This test fails, so we need to modify the application to add some understanding of the supplied form data.

import org.stringtree.finder.StringKeeper;
import org.stringtree.mojasef.HTTPConstants;
 
public class MessageHarness {
  private boolean called = false;
 
  public boolean wasCalled() {
    return called;
  }
 
  public void send(StringKeeper context) {
    called = true;
    String destination = context.get("destination");
    if ("known@example.com".equals(destination)) {
        context.put(HTTPConstants.RESPONSE_CODE, "200");
    } else {
        context.put(HTTPConstants.RESPONSE_CODE, "404");
    }
  }
}

Now we reach an interesting point in the development process. It is perfectly feasible to continue in this vein, step-by-step bringing the application into existence. However, even the streamlined InlineServer still needs to set itself up, bind and unbind ports, deal with network traffic and so on. This is considerably slower than a regular method call, and as more tests are added the test case will continue to slow down.

The advantages of the Mojasef approach to web applications really begin to pay off at this point. The tests we already have prove the basics of HTTP Communication are working, so further tests do not really need to bother with HTTP, provided the tests are testing exactly the same code. Luckily (as pointed out earlier) a Mojasef application is just a regular Java class, which needs no modification to be tested using regular JUnit tests. Here are the same tests as above, but implemented as simple method calls:

import org.stringtree.finder.StringKeeper;
import org.stringtree.mojasef.HTTPConstants;
import org.stringtree.mojasef.HTTPConstants;
 
public class MessageProcessingTest extends junit.framework.TestCase {
  StringKeeper context;
  MessageHarness app;
 
  public void setUp() {
    context = new MapStringKeeper();
    app = new MessageHarness();
  }
 
  public void test404FromUnknownDestination() throws Exception {
    context.put("destination", "unknown@example.com");
    context.put("text", "hello from HTTP");
    app.send(context);
    assertEquals("404", context.get(HTTPConstants.RESPONSE_CODE));
  }
 
  public void test200FromKnownDestination() throws Exception {
    context.put("destination", "known@example.com");
    context.put("text", "hello from HTTP");
    app.send(context);
    assertEquals("200", context.get(HTTPConstants.RESPONSE_CODE));
  }

I shall stop there, leaving the development of the rest of the application to interested readers.

From the above steps, I hope you can see that the normal process of Test-Driven Development is quite possible for both web applications, and web-application clients. When using a framework which gets all the fiddly stuff out of the way it can even be as simple, quick and productive as in-process development.

JSON Schema Proposal

Apparently, Java 1.4 will officially reach its end of service life (EOSL) at the end of October 2008. That of itself does not imply that nobody will continue to use it, but it does help third-party providers such as myself decide when to stop active support.

So, this is my decision. Some time after the start of November I will make a branch of the final Java 1.4-compatible version of all my public projects. Any future changes will no longer guarantee Java 1.4 support.

This does not imply, of course, that all my public code will suddenly be rewritten; dripping with annotations and generics and so on. The process will be incremental and progressive. The main point is that I will no longer be ensuring Java 1.4 compatibility in future releases.

As an aside – if you are still using Java 1.4 and would like to suggest a change to any of my software, please ensure that you get in touch in time for me to include it before November!

For a long time I interpreted this goal as implying that all Stringtree code should run on all Java versions from Java 1.2 onwards. Java 1.4, however, introduced some compelling new features including built-in regular-expression handling. For a few years I still tried to ensure that most code was still 1.2-compatible (for example by using Ant to swap in a third-party regular-expression library while building a jar file), while also providing a Java 1.4 version. Eventually, use of Java versions prior to 1.4 declined enough that I felt comfortable removing the complicated pre-1.4 version.

For the last few years I have been very careful to keep all my Stringtree code compatible with all versions of Java from 1.4 upwards. Now, however, the pressure is building again to move over to Java 5. In my day-to-day coding I develop with Java 5 and make increasing use of Java 5 features such as the enhanced for loop, the Iterable interface, enums, generics, autoboxing, varargs and so on. It would be very nice to be able to update the Stringtree codebase to use these features too.

Occasionally a Java 5-specific detail has crept in to a Stringtree library, and I have soon received comments or emails pointing this out. I haven’t noticed this for a while, which might indicate either that I have been especially careful, or that I there are no longer any/many people developing with Stringtree code who are still limited to Java 1.4.

If you are reading this and you still require Java 1.4 support, please let me know. Likewise, if you have thrown off the shackles of 1.4 within the last year or so or are desperately hoping for a Java 5 Stringtree that would be good to know too.

Is it time for Java 5 yet?

If you need a simple and fast parser for textual data, then this should be all you need. For XML documents containing opaque binary data in a CDATA block, this may not be ideal. Currently CDATA blocks are loaded as String objects, and this can lead to incorrect data for bytes which do not represent valid characters in the current character set.

I am currently planning for the next version of the Stringtree XMLReader to offer the option of extracting a CDATA block as an unprocessed byte array.

Recently I have been working with systems which talk to each other using REST/HTTP. Providing services and resources is pretty simple using Mojasef, but accessing such resources and consuming such services (in client code and in tests) has always seemed a bit more clumsy than it should be. I tried Apache HTTPClient and HttpUnit, but both seemed cumbersome for simple tasks, and bring in several hundred KB or more of dependencies, which can really bloat a small client application. I’m sure there are others, but I got bored with looking, and instead wrote my own simple HTTP Client which does the things I need without dragging in tons of extra stuff.

The Stringtree HTTP Client consists of just four classes, with no dependencies other than the standard Java APIs:

• org.stringtree.http.HTTPClient
• org.stringtree.http.Form
• org.stringtree.http.Document
• org.stringtree.http.NameValue

The point of these classes is to allow simple construction and calling of all valid HTTP requests, including the ability to set and read headers and cookies, simulate the submission of an HTML form, and support both textual and binary content data.

As a very simple example, consider the following code which issues a GET request to a specified URL:

HTTPClient client = new HTTPClient();
Document response = client.get("http://localhost:8080/?a=b");
System.out.println("response content type=" + response.getHeader("Content-Type"));
System.out.println("response content=" + response.getContentAsString());

The above code example shows basic usage of the Stringtree HTTP Client. In more general terms, usage is as follows:

1. Create an object of the HTTPClient class.
2. Set any long-lived settings, such as cookies or a user-agent.
3. Call one of the request methods with appropriate parameters.
4. Read returned headers and content as required from the returned Document object.
5. Repeat from (3) for each new request.

A slightly more complex example using a POST request to submit an HTML form might look like:

HTTPClient client = new HTTPClient("Mozilla/5.0 (example)");
client.setCookie("username", "Frank");
Form form = new Form();
form.put("name", "Widget");
form.put("category", "thing");
Document response = client.post("http://localhost:8080/update", form);
boolean ok = "200".equals(response.getHeader(HTTPClient.HTTP_RESPONSE_CODE));

All the request methods return an org.stringtree.http.Document object. This object represents the structure of an HTTP request or response: a collection of name/value headers (which may contain duplicate names), and a block of “content” which may be considered as text or as a sequence of bytes. The HTTPClient code does make one simplifying concession; as seen in the above POST example the HTTP response code is added as a pseudo-header with the name “http.response.code”.

This should be enough to get started playing with the Stringtree HTTP Client. In the next post I will discuss the possibilities for creating and configuring an HTTPClient object in full detail.

Recently, however, one of my users has been bugging me to make the Stringtree and Mojasef classes and sources available in a Maven-style repository, for easier integration with Maven (or Maven-like) build tools and workflows. So I have done it. The current versions of key Stringtree and Mojasef files are now available from the Stringtree repository at the following URLs:

• http://stringtree.org/repository/org/stringtree/stringtree/2.0.10/stringtree-2.0.10-sources.jar
• http://stringtree.org/repository/org/stringtree/stringtree/2.0.10/stringtree-2.0.10.jar
• http://stringtree.org/repository/org/stringtree/stringtree/2.0.10/stringtree-httpclient-2.0.10.jar
• http://stringtree.org/repository/org/stringtree/stringtree/2.0.10/stringtree-json-2.0.10.jar
• http://stringtree.org/repository/org/stringtree/mojasef/2.0.b1/mojasef-2.0.b1-sources.jar
• http://stringtree.org/repository/org/stringtree/mojasef/2.0.b1/mojasef-2.0.b1-withdependencies-sources.jar
• http://stringtree.org/repository/org/stringtree/mojasef/2.0.b1/mojasef-2.0.b1-withdependencies.jar
• http://stringtree.org/repository/org/stringtree/mojasef/2.0.b1/mojasef-2.0.b1.jar

The Stringtree and Mojasef build scripts now include a “publish” target which uploads such artefacts to appropriate places in the repository, so future versions should continue to be available.

This is the first time I have done this, so I would welcome comments from any readers who actually use Maven (or buildr, or anything else which supports this repository format). In particular, I am interested in opinions on whether it is OK to simply replace artefacts with the same names and locations for minor tweaks and bug-fixes, or whether even the smallest change should result in the generation and upload of a new version with a new name and location.