Frank Carver's musings about software and life
A fascinating presentation from Barcelona Ruby Conference. Fred George talks through the history and examples of his thinking about system architectures composed of micro services.
I found this particularly interesting as it has so many resonances with systems I have designed and worked on, even addressing some of the tricky temporal issues which Fred has avoided.
Thanks to Steve Cresswell for pointing this one out to me.
Yesterday evening I went along to the Ipswich Ruby User Group, where Dan Higham gave an enthusiastic presentation about VMware CloudFoundry. The product looked interesting enough (and appropriate enough to my current project) that I decided to spend a few hours evaluating it. On the whole I’m impressed.
After poking around the web site a bit I decided to download the “micro cloud foundry” version, which does not need dedicated hardware, as it runs as a virtual machine on a development box. Once I had passed the first hurdle of requiring registration before even starting a download, then waited for about 40 minutes for the VM image to download, I had a go at running it up.
Now, I must say that I went a bit “off road” at this point. Naturally enough the web site recommends only VMWare virtual containers for the task, but the last time I used VMWare player (admittedly a few years ago) I found it clumsy and intrusive, and did not want to clutter my relatively clean development box. I already have virtualbox installed, and use it every day, so I thought I’d see how well the image runs in this container. Starting the VM was just a matter of telling virtualbox about the disk image, allocating some memory (I have a fair amount on this machine, so I gave it 4G to start with) and kicking it off.
Initially, everything seemed great. The VM started OK, and presented a series of menus pretty much as described in the “getting started guide”. I had been warned that setting it up might take a while, so I was not worried that it twiddled around for 15 minutes or so before telling me that it was ready. The next step according to the guide was to go to the system where the application source code is developed and enter vmc target http://api.YOURMICROCLOUDNAME.cloudfoundry.me (the micro cloud foundry VM has no command line, it’s all remotely administered) to connect the management client to the VM. This was a bit of a gotcha, as the “vmc” command needs a separate installation, found elsewhere on the site. Essentially “vmc” is a ruby tool, installed as a gem. In this case I already had ruby installed (it’s a ruby project I’m working on), so it was just a matter of sudo gem install vmc. Once I had installed vmc, I tried to use it to set the target as suggested, but the request just got rejected a somewhat confusing error message. On the surface it did not appear to be a network issue – I could happily “ping” the pseudo-domain, but vmc would not connect. After some looking around, both on the web, and digging in the “advanced” menus of the micro cloud foundry VM, I eventually realized that the error message in question was not actually coming from the micro cloud foundry at all but from a server running on my development system!
To make sense of why, I need to describe a bit about my development set up. The basic hardware is a generic Dell dektop with its supplied Windows 7 64-bit OS. I don’t particularly like using Windows for development (and did not want to wipe the machine, because I also need to use Windows-only software for tasks such as video and audio production), so I do all my development on one of a selection of virtual Ubuntu machines running on virtualbox. This is great in so many ways. I have VM images optimised for different work profiles, and run them from a SSD for speed. Best of all, I can save the virtual machine state (all my running servers, open windows and whatnot) when I stop work, and even eject the SSD and take it to another machine to carry on if needs be.
So, the problem I was seeing was due an interaction between the “clever” way that micro cloud foundry sets up a global dynamic dns for the VM, and the default virtualbox network settings. To cut a long story short, both my development VM and the micro cloud foundry VM were running in the same virtualbox, and both using the default “NAT” setting for the network adapter. Somewhat oddly, virtualbox gives all its VM images the same IP-address, and all the incoming packets were going to the development VM. More poking around the web, and I found that a solution is to set up two network adapters in virtualbox for the micro cloud foundry. Set the first one to “bridge” mode, so it gets a sensible IP address and can receive its own incoming packets, and set the second one to NAT, so it can make requests out to the internet. I left the development VM with just a “NAT” connector, and it seems happy to connect to both the web and to the micro cloud foundry VM via the dynamic dns lookup.
Of course, it was not all plain sailing from there, though. The first issue was that I kepy getting VCAP ROUTER: 404 - DESTINATION NOT FOUND as a reponse. A message that was obviously coming from somewhere in cloud foundry, but gave no obvious hint what was wrong. After a lot of trying stuff and searching VMware support, FAQ and Stack Overflow, I came to the conclusion that this is largely an intermittent problem. After a while things just seemed to work better. My guess is that when the micro cloud foundry VM first starts it tries to load dependencies and apply updates in the background. This is probably a quick process inside VMware’s own network, but out here at the end of a wet bit of string, things take a while to download. Eventually, though, things settled down and I was able to deploy some of the simple examples. Hooray! I have subsequently found that the micro cloud foundry VM needs a few tens of minutes to settle down in a similar way every time it is started from cold. Good job I can pause the virtual machine in virtualbox.
The process for deploying (and re-deploying) applications which use supported languages and frameworks is largely smooth and pleasant. It does not use version control (like Heroku, for example) but a specific set of tools which deploy from a checked-out workspace. If you want to deploy direct from VCS, it’s easy enough to attach a a little deploy script to a hook, though.
Once I got past paying with the the examples, I tried to deploy one of my own apps. It’s written in Ruby, uses Sinatra as a web framework and Bundler for dependency management, so it should be supported. But it does not work at all on cloud foundry. It works fine when I run “rackup” on my development box, and it works fine when I deploy it to Dreamhost, but on cloud foundry – nothing. Now, I can understand that there may be all sorts of reasons why it might not work (the apparent lack of a file system on the cloud foundry deployment, for one), but my big problem is that I have so far not discovered any way of finding what is actually wrong. An HTTP request just gives “not found” (no specific errors, stack traces or anything useful). Typing vmc logs MYAPP correctly shows the bundled gems being loaded, and the incoming HTTP requests reaching WEBrick, but no errors or other diagnostic output. I can only assume that the auto-configuration for a Sinatra app has not worked for my app, but there seems to be no way of finding out why.
To me, this lack of debuggability is the single biggest problem with cloud foundry. I hope it is just that I have not found out how to do it. If there is really no way at all of finding out what is going on on the virtual server we are back to “suck it and see” guesswork, which is so bad as to be unusable. I am simply not willing to spend hours (days? weeks?) changing random bits of my code and re-deploying to see if anything works.
If anyone reading this knows a way to find out what cloud foundry expects from a Sinatra app, and how to get it to tell me what is going on, please let me know. If not, I may have to abandon using cloud foundry for this project, and that would be a real shame.
I’m still collecting resources for a potential Sinatra/Redis project and this looks like a useful tutorial:
Building a Twitter Filter With Sinatra, Redis, and TweetStream @ DigitalHobbit.
I’m not really an eager version-watcher, but I do have a soft spot for the ruby web framework Sinatra. It’s nice to see that the Sinatra team are confident enough to give it a “1.0″.
I have been doing a lot of work with virtual machines recently. It’s a great way to sidestep machine-specific issues (Windows, I’m looking at you) and set up a consistent development/deployment environment.
However, downloading, installing, updating and setting up each new image has been increasingly tedious. Vagrant looks like a brilliant idea to enable automation of the whole process. I would try this out immediately, except that most of my Ruby development environments are already running inside a Virtualbox VM, and it’s not very likely that one will fit inside another 
I originally chose to work in Ruby for its ease of deployment to low-cost shared hosting services, but as time has progressed I have become more comfortable with the language and a selection of tools and libraries. One that I like a lot is the Sinatra web framework. Interestingly it now appears that if I wanted, I could run Sinatra applications in a Java web container such as Tomcat, Resin, or Jetty using Warbler, which bundles JRuby and an application into a standard war file for drop-in deployment.
This is an interesting challenge to my own pure-Java Mojasef framework, which offers several similarities with the approach taken by Sinatra. Sinatra wins on documentation, though 
A good guide to getting started with Sinatra and Warbler can be found at Sinatra on Java | Coreguardian.
I’m currently on my fourth or fifth project using the Sinatra web framework. Each project has taught me more about Sinatra, and Ruby, and all the other little things that go into making a web application using these tools. One thing which has been puzzling me, though, is how best to structure the directories and files which comprise each project.
I have been developing Java web applications for so long that I have a familiar and generally useful project structure which I can quickly build from memory. With these new tools I’m not so confident that I would not be storing up problems if I attempt to create my own structure.
So I was intrigued to encounter Monk recently. A system for generating project starting points for projects using a lot of the tools I have come to enjoy. It’s presumably aimed at the simplicity of starting a Rails project, but for a different set of tools.
On my Ubuntu system it was a bit tricky to get going. The documentation just recommends “sudo gem install monk” but this required some extra installations and some general fiddling around. Once installed it happily generated a project structure, which (after a few more installs) would run its tests using rake and start up a server serving “Hello, World” if asked.
I’ll give it a try on my next project, and see if it helps. I might even use the recommended No-SQL storage engine Redis instead of Sqlite or PStore which I have used in the past.
Read some more at » Monk – Quickstart Sinatra Projects Gittr.
Like many teams, I’m sure, we are trying to squeeze every drop of effectiveness out of our time. Manual build and deployment not only takes up valuable time, but also acts as a drag on the development process. Anything which pulls developers out of “the zone” is a bad thing for productivity.
We usually use the familiar “ant” build tool for building deployable artefacts, but have found it increasingly fiddly as our expectations have grown. And ant is hardly useful at all for real-world deployment tasks.
It may be the time to re-consider other build and deployment options.
Now this is an interesting result. It appears that even when running in a virtual machine, Linux can be a faster development system than the Windows the VM is running on!
This looks very interesting, although it is at an early stage right now. A system for building and managing cloud-hosted applications, using XMPP as a communication subsystem.
InfoQ: Engine Yard Releases Cloud Management Framework Vertebra
A conference presentation from RubyFringe, designed to be contentious. There are some good points, particularly about the way that different approaches to testing can complement each other, but I think he misses the point about TDD when he lumps it in with developer unit testing and ignores the design aspects of the technique.
I’m a strong believer that manual steps should be automated wherever possible. One of the areas of our current product which seems to require an inordinate amount of manual “faffing” is provisioning and deployment.
Chef, a ruby system for distributing and automating just these kinds of tasks seems an interesting solution.
It’s a long-running argument. Are modern, dynamic, languages such as Ruby and Python really much more concise than more mainstream languages such as Java? What constitutes “big” in each case? Stephan tries to flush out some hard facts, and gets a lot of comments.
I develop almost all my code these days using Test Driven Development (TDD). By taking this approach I never bother about unit test code coverage, but some developers seem very concerned by it.
One thing that I have never considered doing is retroactively adding tests just to increase some code-coverage metric. Dan Manges has written in interesting explanation of why that might be, using some examples in Ruby.
So the question is: are dynamic-typed languages (ruby, python, smalltalk, etc.) easier or harder to maintain than static-typed languages (Java, C#, etc.)? It seems there are arguments both ways.
On the one hand, dynamic languages tend to be a bit more concise, and reducing code size is a great way to simplify maintenance. On the other hand static languages typically support better and more intelligent tooling, which is also a great way to simplify maintenance.
Ola Bini has some thoughts, but has so far been unable to come up with any hard data one way or another. is there any?
The Maintenance myth | Ola Bini: Programming Language Synchronicity
A thoroughly enjoyable presentation from RubyFringe, one of the best I have ever watched.
It starts with a few minutes of demonstration of a reasonably cool music tool, but the real meat is in the presentation which follows – a fast paced, insightful and downright humorous rant about a wide range of topics from Leonardo daVinci to mountain lions, web applications and venture capital.
I guess this a bit old by now, but I hope they continue with this at least for the moment: a roundup of some of the interesting things going on in the Git world.
My current work is centred around Java, and all our applications are deployed in java containers. It might one day be useful to deploy JRuby applications, and in particular rails apps, to these same containers. Here’s a list of things to beware of in such situations.
When I first saw this it looked great: a ruby REST wrapper which supports a lot of useful test and integration possibilities. However, the deeper I looked, the more disappointed I became. I’m now saddened to believe that this is based on yet another misunderstanding of what REST is.
As far as I can tell, the ActiveResource concept, on which this approach is based, is merely an attempt to impose a constrained CRUD data-model over HTTP. There is no concept of content negotiation – all data is XML according to pre-assumed schemas. There is no support for automatic discovery and use of hrefs between resources. There is even the suggestion that it’s a good idea to casually extend the HTTP protocol with extra custom methods.
All of these are typical problems found in projects which use the name REST without the key bits which make it really work. The end result is just another fragile, application-specific RPC protocol. Sigh.
nutrun » Blog Archive » Testing web services with ActiveResource
Even though this article uses the term “web 3.0″, it’s still worth reading. The idea of moving from specific “cloud computing” implementations to a more generic and pluggable approach is a neat idea. Using ruby as a scripting language for “agents” and Erlang for the infrastructure seems a good balance, too.
InfoQ: Vertebra: EngineYard’s Next Generation Cloud Computing Platform