Ruby Pond

Getting started with MongoDB

2010-07-03T00:00:00+00:00

MongoDB is a document database, which if you’ve not come across before is best visualised as a way of persisting a complex Hash object. It varies from a key-value store (like memcache and redis) as the “documents” you store aren’t retrieved by their key alone, but can be extracted by querying the values. The objects can be of any structure as you don’t define a schema upfront, and can be nested to an infinite depth.

The beauty in a design like this is that you can have a flexible way of storing whatever you want, with all dependent objects nested within each other so you don’t need to join across multiple tables to get the information you need. In situations where the most often use-case requires more than one query or a table join to retrieve everything you want (a blog post along with a series of comments is a good example) this can be a real performance win. You’d actually store the comments within the blog post itself rather than in a separate table or collection.

Where I’ve found it useful, is in storing the data from the Twitter Streaming API. I’ve had grand plans to build an app on top of this data and API, but life continues to get in the way and in the interim period Twitter keeps adding new features and richer data. Thanks to MongoDB I don’t need to update my code to stay in step with their change, I just dump the data straight into the DB and it will automatically include the new fields that have been included. Even better, is that I can write queries against this new data immediately and don’t have to worry about migrations.

Installing MongoDB

Installing MongoDB is easy. On OSX I just used Homebrew:

brew install mongodb

On other platforms there are binaries available to download.

By default mongod wants to store data in the /data/db directory so you’ll need to create that first:

mkdir -p /data/db

Once that’s done you can start the service by just running mongod. Open another terminal session and run mongo to connect to your local server. You’ll be taken to a Mongo shell session where you can start issuing Javascript commands to talk to your database. So start by doing the following:

db.mydatabase.save({ name: "Steve" })

The command above with save the document {name: "Steve"} into the database called mydatabase. But that doesn’t exist, does it? Well, it does now. If you try and issue a command to a database that doesn’t exist, Mongo will create the database for you and then run the command against it. If you take a look in /data/db now you should see a couple of files, one of them is probably in the range of 64MB to 2GB. Hang about, what?! 2GB to store that one small document?

Mongo will pre-allocate disk for storing these objects, that means the next record you insert wont increase the size of these files. It will continue appending documents into these pre-allocated files until they are full, at which point it will pre-allocate another file of the same size and repeat the process.

Storing the Twitter Stream in MongoDB

As I said, I’ve been using it as a flexible store for persisting the Twitter stream. I’ve created a Tweet model to store each tweet that looks like the following:

require 'mongo'
require 'twitter-text'

class Tweet  
  def self.create!(tweets)
    collection.insert(tweets)
  end

  private
    def self.establish_connection
      Mongo::Connection.new.db("twitter")
    end

    def self.db
      @db ||= establish_connection
    end

    def self.collection
      @collection ||= db.collection("tweets")
    end
end

I’m just using the native ruby driver and not using one of the wrapper libraries, and to be honest I have no plans to as the interface is so simple I don’t really see the point. You just need to open a connection to a database (in this code above it’s called twitter) and then identify the collection this model is writing to (the equivalent of a table for those transitioning from SQL, above it’s called tweets). Remember that if the database and collection doesn’t exist, Mongo will just make it for us. Using the ruby code we’ve been building on from previous posts, we can take the data we’ve been receiving and store it with the following:

Twitter.stream("mytwittername", "secret") do |status|
  Tweet.create!(status)
end

Simple! Now you’re storing the tweets as quickly as they are arriving. That in itself isn’t very interesting though, so lets see how we’d go about retrieving some of our saved data.

Querying in MongoDB

To do the equivalent of a select * from where ... in Mongo you need to pass a JSON object to the find command. Something like the following:

collection.find(:user => { :screen_name => "glenngillen" })

What might not be entirely obvious here is that the Twitter data comes back in a format like the following:

{ :user => { :screen_name => "glenngillen", 
             :profile_image_url => "http://rubypond.com/image.png",
             :followers_count => 1000000 },
  :text => "This is the text from my tweet"
}

And you can see from the query above, I’m able to query based on value nested down within the :user key. Much like SQL, you can write queries to return data that is in a given list of values, is greater than or less than a certain value, you can even match based on a regular expression. For more examples of how to query the data, head over to the MongoDB query documentation

Taking it further

In coming posts I’ll expand on the previous examples, show you how to easily:

Supply additional options to the stream to filter it down to just the tweets you are interested in
Setup MongoDB to store the data you need
Provision Amazon EC2 instances to help you deal with processing load
Get Chef involved to handle the provision and setup of your EC2 instances automatically
Use RabbitMQ to dispatch work to multiple servers
Load balance your MongoDB instances across EC2

Managing email lists with Mailchimp and MonkeyWrench

2010-07-02T00:00:00+00:00

Most of my projects and clients have a need to send email to the users to inform them of updates to the system, inform them of special marketing offers, or other such broadcast messages. Unfortunately, these type of messages can flag up warnings with many mail servers as your message may never arrive at the recipient. There are numerous providers out there that do what they can to make it easier, and most do a respectable job (I think I’ve used almost all of them at some stage). Mailchimp (or here via an affiliate link) has proven to be the best for my purposes to date.

They have a really rich and well documented API, a pretty good admin interface (although the bar is set pretty low here by most competitors), and they’re free to use until you get over 500 users. And even then they’re pretty good value for what you get.

Introducing MonkeyWrench

Over a year ago now I started first started integrating my apps to using Mailchimp, and while it was quite easy I did occasionally run into problems. The ruby APIs were just thin layer atop the Mailchimp API and it didn’t feel natural, plus there were a range of edge cases that were non-obvious unless you were hanging out in the dev forums or using the API a lot (like the fact that you can’t batch subscribe members to a list and send them a welcome email).

So I wrote MonkeyWrench to overcome these shortcomings. It feels more natural to use within a ruby app, and it will automatically process large actions in batches, subscribe members as a list in batch if possible but do them individually if that is what is required. It’s all an attempt to make interacting with with your users as simple as possible.

It’s in production use for a number of clients, and significant portions of the codebase have been contributed/improve upon by David Heath who I’ve been working with at Wordtracker.

To give an example of how to use it, I’m going to just paste some of the README in here.

Getting Started

To get started, you need to first connect to the appropriate datacenter with your API key:

MonkeyWrench::Config.new(:datacenter => "us1", 
                         :apikey => "your-api-key-goes-here")

From there you’ve got a rich API for managing Lists and Members. To subscribe a new user to a list simply do the following:

list = MonkeyWrench::List.find_by_name("My Example List")
list.subscribe("foo@bar.com")

Taking it Further

Code is available at GitHub, documentation is an evolving affair (all of the List methods are documented). I’m more than happy to receive patches for any functionality that isn’t covered, we’ve only really implemented what we’ve needed to date. Just fork it, make a test, and send me a pull request with the patch.

Filtering the Twitter Streaming API

2010-06-30T21:00:00+00:00

A couple of months ago I gave a brief introduction on how I’ve been parsing the Twitter Streaming API in my ruby applications. Part of the new featureset of the streaming API is the ability to filter the stream to only include tweets that include your preferred users, a set of keywords, from within a specific geographic location, or a combination of the above. There are limits on how many filter rules you can provide, and they vary depending on your access level so check the documentation for more details

Extending the Twitter class created in the last post should be fairly straight forward. First there is a new URL:

url = URI.parse("http://#{username}:#{password}@stream.twitter.com/1/statuses/filter.json")

And we have to POST to this address instead of requesting with GET:

Yajl::HttpStream.post(url, :symbolize_keys => true)

We also need to pass through the predicates we want to filter on. I’ve opted for building up a list of the settings, only including them if it’s been supplied. I’ve also given the options different names to the Twitter specified ones to try and prevent me confusing them in the future (does follow mean “follow this user” or “follow these keywords”? Avoid the confusion by calling them users and keywords instead):

params = []
params << "follow=#{[*filters[:users]].join(",")}" if filters[:users]
params << "track=#{[*filters[:keywords]].join(",")}" if filters[:keywords]
params << "locations=#{[*filters[:locations]].join(",")}" if filters[:locations]

You’ll notice above that I’m actually splatting the value of each setting into a Hash and then calling join on that. The reason is so I can pass through just a single value (:users => 12) or a list of values (:users => [12,13]) and they’ll both work the same way.

All that is left is to wrap it all up in a new method, and add it to our class:

require 'uri'
require 'yajl/http_stream'

class Twitter
  MAX_ALLOWED_ERRORS = 1200
  
  def self.filter_stream(username, password, filters = {}, &block)
    url = URI.parse("http://#{username}:#{password}@stream.twitter.com/1/statuses/filter.json")
    params = []
    params << "follow=#{[*filters[:users]].join(",")}" if filters[:users]
    params << "track=#{[*filters[:keywords]].join(",")}" if filters[:keywords]
    params << "locations=#{[*filters[:locations]].join(",")}" if filters[:locations]
    consecutive_errors = 0
    while consecutive_errors &lt; max_allowed_errors  do
      begin
        Yajl::HttpStream.post(url, params.join("&"), :symbolize_keys => true) do |status|
          consecutive_errors = 0
          yield(status)
        end
      rescue Yajl::HttpStream::InvalidContentType
        consecutive_errors += 1
      end
      sleep(0.25*consecutive_errors)
    end
  end
end

Taking it further

As previously mentioned, I’ll extend on this series to cover:

Consuming the Twitter Stream
Supply additional options to the stream to filter it down to just the tweets you are interested in
Setup MongoDB to store the data you need
Provision Amazon EC2 instances to help you deal with processing load
Get Chef involved to handle the provision and setup of your EC2 instances automatically
Use RabbitMQ to dispatch work to multiple servers
Load balance your MongoDB instances across EC2

Bash script to check timestamp on cron logfile

2010-06-30T00:00:00+00:00

For one of my current clients we’ve been making some changes to a script that runs nightly via cron. Things like moving over to using bundler, not running it via script/runner to avoid loading the full Rails stack, and a couple of other things that have introduced the risk that a dependency may not load in production or be available in the context/user in which the script is run. Having been caught out once, it’s time to put some monitoring on the sucker to make sure it doesn’t happen again.

Redirecting script output to a logfile

For most cron jobs we pipe the output to a log file somewhere to help keep out inboxes sane. It also means we can then monitor the log for events, and fire off an appropriate message (IM/jabber, email, sms) depending on the severity. So we redirect STDERR to STDOUT, and then append that to a file:

/path/to/myscript >> /var/log/myscript 2>&1

In addition to that though, we want to know the exit code for the last time it ran, so I’ll append that to the end of the command

 /path/to/myscript >> /var/log/myscript 2>&1; echo $? > /var/log/myscript-error-code

Returning the exit code

We’re using Zabbix to monitor services, so what I need is a means of raising an error if the exit code from the last run was not zero. That could fairly trivially be done by simply doing cat /var/log/myscript-error-code but I know were going to want to extend that just a little, so I’m going to put in in a shell script:

#!/bin/bash

LOG_DIR=/var/log
ERROR_CODE_FILE=myscript-error-code

cat $LOG_DIR/$ERROR_CODE_FILE

Checking that the script has been run in the past 24 hours

There are a number of servers this codebase is deployed to, but we only want it to run on one of them nightly. That also introduces the risk that the servers wont know which one should run the script, and as a result it doesn’t run at all. To mitigate against that, we want to make sure the file has exit code is no more than 24 hours old:

MINS_SINCE_UPDATE=-1500
if [[ -z $(find $LOG_DIR -cmin $MINS_SINCE_UPDATE -name $ERROR_CODE_FILE) ]];
then
  echo "Error"
fi

I’ve used the cmin option because using atime is problematic with monitoring things that should occur in less than a day, and I want to be confident in the difference between it running 23 hours ago, and 25 hours ago. The -z option is testing if the following command returns an empty response.

Calculating the difference between now and the file timestamp

So I can immediately tell whether the problem lies with cron/scheduling or the script I’m going to make the monitoring send a more meaningful error when it is a scheduling error:

LAST_MODIFIED=`stat -c %y $LOG_DIR/$ERROR_CODE_FILE | cut -f1 -d '.'`
HOUR_SINCE=$(((`date -d "$LAST_MODIFIED" +%s` - `date +%s`)/3600))
echo "Import last run $LAST_MODIFIED ($HOUR_SINCE hours ago)"

Here I’m using stat to return the date the file was last updated. I pass that into date to convert it to seconds since epoch, and then subtract the current seconds since epoch. Divide those seconds by 3600 and you get how long in hours since that file was updated.

Putting it all together

So the final script looks something a little like this:

#!/bin/bash

LOG_DIR=/var/log
ERROR_CODE_FILE=myscript-error-code
MINS_SINCE_UPDATE=-1500

if [[ -z $(find $LOG_DIR -cmin $MINS_SINCE_UPDATE -name $ERROR_CODE_FILE) ]];
then
  echo "Error"
  LAST_MODIFIED=`stat -c %y $LOG_DIR/$ERROR_CODE_FILE | cut -f1 -d '.'`
  HOUR_SINCE=$(((`date -d "$LAST_MODIFIED" +%s` - `date +%s`)/3600))
  echo "Import last run $LAST_MODIFIED ($HOUR_SINCE hours ago)"
else
  cat $LOG_DIR/$ERROR_CODE_FILE
fi

All that is left is to plug it in to Zabbix, and hope that it never fails again.

Building ruby-debug on Ruby 1.9.x

2010-06-26T00:00:00+00:00

If you’ve tried to install ruby-debug with ruby 1.9.x there are two things you need to know. First, you need to be using the ruby-debug19 gem and not the regular ruby-debug one. Second, you’ll probably need to point to the ruby source for the version of 1.9 that you are using. And that is as simple as:


gem install ruby-debug19 -- --with-ruby-include=`cd ~/.rvm/src/ruby-1.9.2-preview3; pwd`

Obviously, replace 1.9.2-preview3 above with whatever release you are using.

You’re welcome.

Consuming the Twitter Stream

2010-04-28T00:00:00+00:00

I’ve got a bunch of code I’ve been using for various projects for years that I’ve been sitting on with plans of documenting and sharing, so first to get that treatment is some code to handle parsing of the Twitter Stream. Apt timing given the recent creation of dev.twitter.com and the conclusion of the first @Chirp developer conference.

What is the Twitter Streaming API

Essentially it’s a feed of the public twitter stream (so not private tweets or direct messages), cleaned for relevance (so hopefully known spammers are excluded), and down–sampled to a manageable size based on your level of access and requirements (the default current sampling rate is 5% of statuses). It’s different to just requesting the rss/xml/json of the public timeline as it is an open connection that is constantly streaming data to you, so you only need to start another request if the connection is dropped for some reason.

How to process the incoming stream

I’ve been using the ruby-yajl library because it’s a C-based JSON parser which is a bit quicker than the standard JSON parsing libraries, but also because it has a built-in HttpStream class. Lets take a look at some code:


url = URI.parse("http://#{username}:#{password}@stream.twitter.com/1/statuses/sample.json")
Yajl::HttpStream.get(url) do |status|
  puts status.inspect
end

The code above will connect to the given url, and iterate over each JSON object it gets back. In the example, I’m just outputting the object so we can see what it is.

Of course things will sometimes go wrong, connections will be dropped, etc. at which point Yajl typically complains that it can’t process the stream. So to get around it I rescue the exception and wrap the whole thing in a loop:


loop do
  url = URI.parse("http://#{username}:#{password}@stream.twitter.com/1/statuses/sample.json")
  begin
    Yajl::HttpStream.get(url) do |status|
      puts status.inspect
    end
  rescue Yajl::HttpStream::InvalidContentType
  end
end

Now being a good citizen requires that if you start getting errors you need to back off your retry rate, so best to put that in place so it’s not forgotten. Keep track of the number of consecutive errors, and switch to a while loop:


max_allowed_errors = 1200
consecutive_errors = 0
while consecutive_errors < max_allowed_errors do
  url = URI.parse("http://#{username}:#{password}@stream.twitter.com/1/statuses/sample.json")
  begin
    Yajl::HttpStream.get(url) do |status|
      consecutive_errors = 0
      puts status.inspect
    end
  rescue Yajl::HttpStream::InvalidContentType
    consecutive_errors += 1
  end
  sleep(0.25*consecutive_errors)
end

And you’re more–or–less done. Of course it would make sense to wrap this code up into a re-usable class, so:

require 'uri'
require 'yajl/http_stream'

class Twitter
  MAX_ALLOWED_ERRORS = 1200

  def self.stream(username, password, &block)
    url = URI.parse("http://#{username}:#{password}@stream.twitter.com/1/statuses/sample.json")
    consecutive_errors = 0
    while consecutive_errors &lt; max_allowed_errors  do
      begin
        Yajl::HttpStream.get(url, :symbolize_keys => true) do |status|
          consecutive_errors = 0
          yield(status)
        end
      rescue Yajl::HttpStream::InvalidContentType
        consecutive_errors += 1
      end
      sleep(0.25*consecutive_errors)
    end
  end
end

You may have noticed a couple of other changes in the code above. I’m no longer outputting the status but rather I’m yielding it to a block. I’ve also set an extra parameter on the Yajl::HttpStream.get call to automatically symbolize the keys of all the objects returned.

Using out new Twitter stream

Putting the code into action, I use it like the following:


Twitter.stream("mytwittername", "secret") do |status|
  Tweet.create!(status)
end

The benefit of having our stream method yield it to a block is that you now have the flexibility to do almost anything with each returned status. In my simple example above I’ve got it storing straight to a MongoDB collection I’ve created, but I could just as easily add some additional information to the status object before saving:


Twitter.stream("mytwittername", "secret") do |status|
  Tweet.create!(status.merge(
    :weather => current_weather(status[:location]),
    :mood => determine_mood(status[:text])
  )
end

Or store the status, in addition to adding it to a queue for post-processing and aggregation by our army of machines in the cloud:


Twitter.stream("mytwittername", "secret") do |status|
  Tweet.create!(status.merge(
    :weather => current_weather(status[:location]),
    :mood => determine_mood(status[:text])
  )
  TweetProcessors.queue(status)
end

Taking it further

In coming posts I’ll expand on the previous examples, show you how to easily:

Supply additional options to the stream to filter it down to just the tweets you are interested in
Setup MongoDB to store the data you need
Provision Amazon EC2 instances to help you deal with processing load
Get Chef involved to handle the provision and setup of your EC2 instances automatically
Use RabbitMQ to dispatch work to multiple servers
Load balance your MongoDB instances across EC2

Upgrading Engine Plugins to work with Rails 3

2010-03-21T12:34:00+00:00

I’ve just finished upgrading an application to run on Rails 3, and for the most part it’s been a relatively pain free process. I refused to use the rails upgrade plugin just so that I’d be forced to work through the problems and experience first hand what has changed. It turns out, not a huge amount. The most significant change though was the need to refactor my Rails 2.3 Engine plugins to work with Rails 3 as a gem.

Creating a gem

There are countless ways to create a ruby gem these days but the problem is that they are all pretty bat-shit mental. Kudos to James (coincidentally the very same James Adam who created Rails Engines) for trying to make people wake up and realise that creating a gem really isn’t that hard. If you’ve already got experience in doing it some way or another you can skip this bit, otherwise follow along and discover how simple it is to roll one by hand.

First you need to create yourself a gemspec file, place it in the root directory of your engine:


# my_rails_engine.gemspec
Gem::Specification.new do |s|
  s.version = '0.0.1'
  s.name = "my_rails_engine"
  s.files = Dir["lib/**/*", "app/**/*", "config/**/*"]
  s.summary = "A summary about my engine"
  s.description = "Some more details about what my engine will do"
  s.email = "glenn@rubypond.com"
  s.homepage = "http://rubypond.com"
  s.authors = ["Glenn Gillen"]
  s.test_files = []
  s.require_paths = [".", "lib"]
  s.has_rdoc = 'false'
  if s.respond_to? :specification_version then
    current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
    s.specification_version = 2
    if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.2.0') then
      s.add_runtime_dependency('another_gem_here', '>= 0.4')
    else
      s.add_dependency('another_gem_here', '>= 0.4')
    end
  else
    s.add_dependency('another_gem_here', '>= 0.4')
  end
end

Most of the fields are relatively self explanatory. The if s.respond_to? block could potentially be removed. It’s there to add in dependencies in a way that will work with various versions of rubygems, but given this is a Rails 3 Engine, which requires Bundler, which is in turns dependent on rubygems 1.3.6+ then you could assume you’ve at least got that and only specify dependencies using add_runtime_dependency.

Next up, create a Rakefile in the same directory to help package up the gem for you:


# Rakefile
require 'rubygems'
require 'rake'
require 'rake/gempackagetask'

spec = nil
File.open('my_rails_engine.gemspec', 'r'){|f| spec = eval(f.read)}

Rake::GemPackageTask.new(spec) do |pkg|
  pkg.need_zip = false
  pkg.need_tar = false
end

Now from the command line you can type:


rake gem

And congratulations! You just created your first gem (it’s in pkg/my_rails_engine-*.gem) without the need to download all sorts of generators or other packages. It wasn’t that hard was it?

Migrating a Rails 2.3 Engines Plugin to a Rails 3 Engines Gem

I’ll admit right now that I’ve not dug too deep into the internals of Rails 3 or the new engines/railtie setup yet. But to me they’re already feeling a lot more like merb slices and django apps than just a plugin. They’re basically a rails application in their own right, complete with routes, initializers, locales, and the models/views/controllers that you’d expect. That also means they carry many of the same expectations in terms of directory layout and the existence of certain files. There’s a number of initializers that need to be available for the Railtie/Engine to load, so I simply created a new Rails 3 project elsewhere and copied the relevant files over.


rails my_new_rails3_app
cp -r my_new_rails3_app/config/initializers my_rails_engine/config

If you have any routes in your engine, it’s worth updating those too. They need to live in the /config/routes.rb file within you engine and should be amended to use the new syntax for defining them (read the comments in routes.rb in the my_new_rails3_app we made before to see examples).

And finally, we need to remove the existing init.rb file as it’s no longer used. Instead create /lib/my_rails_engine.rb with something like the following:


require "rails"
module MyRailsEngine
 class Engine < Rails::Engine
    engine_name :my_rails_engine
  end
end

For some reason the best documentation I can find at present on configuring your engine appears to live in this gist. To summarise the most important bit, you can override the path to various options like so:


class MyRailsEngine < Rails::Engine
  paths.app                 = "app"
  paths.app.controllers     = "app/controllers"
  paths.app.helpers         = "app/helpers"
  paths.app.models          = "app/models"
  paths.app.metals          = "app/metal"
  paths.app.views           = "app/views"
  paths.lib                 = "lib"
  paths.lib.tasks           = "lib/tasks"
  paths.config              = "config"
  paths.config.initializers = "config/initializers"
  paths.config.locales      = "config/locales"
  paths.config.routes       = "config/routes.rb"
end

Installing your Rails Engine within your Rails App

Now that we’ve got our Engine refactored into a gem, it’s time to get it working within our app. First thing, it’s no longer a plugin so if you’ve left it in it’s previous location (/vendor/plugins) you need to move it out. You could of course run rake gem as we did early, then gem install it onto your system, and finally include it with Bundler with a line like the following in your Gemfile:


gem "my_rails_engine"

Alternatively, and probably useful at least during development, you can specify the path to the gem within your bundle like so:


gem "my_rails_engine", :path => "lib/my_rails_engine"

And you’re done. Start your app up, and start debugging any other problems you might have. I know for my engine I had to update both HAML and Devise to be the latest pre-releases, so it would be worth checking if you need to do the same.

JRuby, and why bouncers are assholes

2009-07-30T13:58:30+01:00

I'm normally not one to use this site as a soap box or an outlet for opinion pieces. But I figured it's easier than being continually baited into replying on all the various blogs/forums/groups out there. There's a whole heap of talk in the ruby community at the moment on JRuby and it's place in the ecosystem. Most of it has been kicked off in response to Charles Nutter's post on JRuby's importance to Ruby. I was fortunate enough to see Charles' presentation at Rails Underground last week and have a chat to him, and he had some great insights and examples of why JRuby actually is pretty damn awesome.

The sky is falling

The only real problem I have with Charles' post is the inference that all is lost for the Ruby community if we don't embrace JRuby quickly. It's not, it won't suddenly disappear. What we will possibly lose is the migration of great java developers like Jim Weirich and Charles. I'm willing to excuse a bit of sensationalism in his post though, we're effectively talking about his livelihood here. He needs to get people to pay attention, he needs people to use JRuby, he needs more of us to embrace it.

Whatever his motivations, he's right.

Java sucks

Yes, we know. But you've missed the point. This isn't about giving us java, it's about giving java ruby. Or better yet, giving ruby the JVM. Just try and think through the argument logically; why on earth would a java developer who is unhappy with java for whatever reason move to a new language, that gave you all the same obstacles? They wouldn't (no matter what your opinion of the average java dev is). It's about combining the flexibility of ruby with the good bits of java, the bits that are good enough to make people want to stick with java.

Java/The JVM is slow

"Good morning Mr. Kettle, I'm the pot… and you my friend look black". Really? People that are happy enough with the performance of ruby 1.8 are pointing the "unperformant" stick at other interpreters? Give me a break. The fact is that the JRuby implementation (in most cases) is as good as 1.8, and in many is as good as 1.9. The fact that such a minor section of the community seems to be aggressively migrating to 1.9 highlights the fact that performance really isn't such a high concern for you in real life work, it's just a great scape goat for debates. Oh, and worried about that additional few secs startup overhead from starting the JVM for little scripts? Then don't start it up

Ruby/rails is a competitive advantage

Bull. Shit. Your competitive advantage is your expertise and the ability to meet your client requirements, as efficiently as possible. Period. Ruby and Rails have helped you have an edge for the past few years, but there's many alternatives that are closing the gap. Most of my clients don't care what technology I use so long as I do the best job for them and make the most efficient use of their money. And why should they care? The few clients I have worked for that would care, would probably still prefer I use java over ruby.

The EngineYard deal is acknowledgement from Sun that even they think JRuby is dead

Wrong. JRuby is never going to be a core part of Sun's business. There are so many people that are happy with the status quo that they will continue coding away in java until such a day it might be declared deceased. Why would they have any desire to invest both time and money to stop such a minor outflow from their business? They wouldn't, they've got bigger problems to deal with. On the other hand, EngineYard is all about infrastructure… and infrastructure for hosting ruby apps specifically. Higher performance ruby implementations means they need to invest less in hardware for the same return. It has a direct and appreciable impact on their bottom line. Sun was a great fit initially to allow the guys access to the java team so we could get the most out of the JVM. EngineYard have more to gain out of a successful JRuby at this point than Sun do, so they're a much better fit.

We don't want more java developers or the java community

This is the one that irks me the most. I can only recall two occasions in life where one group of people proclaims they are too cool to associate with another group of people. Bouncers at nightclubs, and "that" group in high school. Bouncers are assholes, and those kids from high school probably grew up to become bouncers.

If you've got a problem with someone else, that's your own insecurities and your problem to deal with, don't infect the rest of us with it. What's the issue here? You don't like their code? Then fork it, fix it, or do it yourself and the problem magically goes away. If you don't want to do that, then shut your pie hole and pay some respect to those that actually do the work. You don't like the competition? Go read a book or five, find some better developers to pair with, stop trying to hide.

If we're using a technology that really is as great as we make out, we should see it as inevitable that the community is going to grow significantly in size at some point. I think Malcolm Gladwell refers to it as the tipping point. The technical ability of the average developer may drop a little. Commercial interests may infest us and move conferences to Vegas. But for every 10 hacks you still end up with a Jim Weirich, Giles Bowkett, David A. Black, etc. And for every large conference you end up with a Rails Underground, Scotland on Rails, or Ruby Hoedown.

Looking at the world through ruby tinted glasses

The fact is that for a long time the performance of ruby has sucked. We've worked around it and justified it for a long time, and for the vast majority of people it didn't suck bad enough to cause problems. Heck, we were all happy using webrick until Zed Shaw came along and showed us a better way. But 3rd party library support has also sucked. Trying to create PDFs was a nightmare for ages. Trying to work with Microsoft document formats still is. I can think of at least 5 occasions where I've solved a simple problem in another language because doing it in ruby would have been far too difficult.

For all the talent the community has, there are so many areas where good support is still lacking that it's unrealistic to think there are enough talented people who care and have the time to do anything about it. Combine that with the fact java has at least (commercially) a 10 year head start and the gap between the two languages is huge. Java has plenty to offer us.

Why can't it all just be peace, love, rainbows, and freakin flying ponies?

All this arguing seems to do is instill a whole "us and them" mentality. There's no need for it to be that way. Embrace JRuby, take advantage of the performance of the JVM, get access to a whole swathe of new libraries that go far beyond what your current list of gems currently offers you… and get to keep using the language you love.

One of the things I admire most about the ruby community is the ability it has to support innovations and allow the proverbial cream to rise to the top. WebBrick was replaced by Mongrel, then we had Thin, and now we've got Passenger. SOAP became REST. AttachmentFu became Paperclipped, etc. I'd hate if in this instance it wasn't allowed to happen because of a whole heap of misconceptions about what JRuby was and what it had to offer.

Or maybe we just need to get a mascot like Django and chill the fuck out.

Rails Underground - Day 2

2009-07-25T09:59:19+01:00

update: videos are now online

I was a little late to the kick-off of the keynote on day 2 with Yehuda. For the most part if is another discussion on what's happening in Rails 3 so I don't think I've missed anything I hadn't heard before. He then started to go into more detail on how things have changed than he has previously, probably due to the progress since I last saw him in March…

Yehuda Katz - Keynote

Yehuda went through the upcoming changes in approach and philosophy on rails Dev. Having well defined contracts, consistent conventions, and allowing things be be changeable.

One of the changes of not is splitting off ideas and interfaces into modules which can then be imported into a class at runtime. He used the analogy that your parents didn't define everything you could do when you were born, you can learn new things. So to when you define ta class you don't have to define the complete interface up front, nor if you want to make changes to you need to re-implement the entire class. Instead you can have sub-sections of interfaces included or changed at runtime where appropriate.

In Rails 3 this translates into better separation of the components and clearly defined interfaces. No longer are you forced to use ActionView, you just need to use something that is "ActionView compliant". The same goes for ActiveRecord, your models are just "ActionModel compliant".

An ActionView compliant interface just needs to implement the following 3 methods:

for_controller(paths, assigns, controller)
render_template(opts)
render_partial(opts)

An ActiveModel compliant interface needs the following 4 methods:

new_record?
valid?
errors
model_name

That last model_name method needs to return plural, partial name, and a few other things rails expects internally. It'll only be 10-20 lines of code to implement manually, or there is a module to include if you just want the default behaviour.

Next up is a cleaner separation of components, one which we got a demo of was splitting out the ActiveModel::Validations. Now you can create your own ActiveModel compliant persistance/ORM layer, and pull in the existing validations for free regardless of where it's stored (YAML, CouchDB, etc.).

By implementing the above you get a system that fully works with ActionPack, so all the usual form and error helpers will just work.

The other benefit is that pulling in support for these doesn't need you to load the entire Rails stack and rob your machine of RAM, it's only a 1.5mb hit.

Next was the new ActionController::Metal demo. At first I wondered why I'd bother, it was slightly more vebose than just using Rackable or rolling it manually, and seemed to carry more overhead than Sinatra. But it gives you the ability to opt into additional features easily. If you wall before/after filters you can pull in the callbacks module. If you want better rendering options (various formats, etc) pull in the rendering module, if you want different layouts pull in that module. ActionController::Base is basically just ActionController::Metal with all of the modules pulled in, so you've now got the way to strip back to metal, or anything in between.

The final piece is ActiveSupport. If for some reason you want to pull in the entire ActiveSupport behemoth you just call:

require 'active_support/all'

Otherwise you can call in the various newly separated modules, one of note was

require 'active_support/ruby/shim'

Which backports a bunch of useful ruby1.9 features on datatypes that could be easily implemented in pure ruby. So you can use them now, and when you upgrade to ruby1.9 you get the performance improvement of having them implemented in C.

The end result is a new level of granularity. Just because you want to use DataMapper doesn't mean you have to say goodbye to all of ActiveRecord. So you can opt-out of just the tiny pieces you don't want. Conversely, you can start completely bare and opt-in on various features to build your own framework on top of rails. You could now build Sinatra on top of rails, and when the complexity of your app increases you can much more easily pull in the additional features you need. All in all, it sounds really cool. Cool enough that I might even try using it on my next app.

Dr. Nic Williams - Dead Simple JavaScript Unit Tests in Rails with Blue Ridge and Screw.Unit

Dr. Nic has dropped a very important bombshell right at the start of his talk. The while Australia may indeed be losing the current 5 game series, we are still holders of The Ashes. Now on with the less important aspects of the presentation.

Nic has a bunch of open source plugins and projects which he could have come and given a presentation on, but he's so impressed with BlueRidge that he feels it's the most important thing to happen to the Rails ecosystem since Cucumber. At a high level it allows you to do both in browser and headless testing, it fits in with the existing rake tasks, and means that javascript testing should just become part of the usual TDD/BDD development approach and not be an afterthought in Cucumber.

Generating a javascript test with BlueRidge creates a spec file, and a fixture file which will give you some test HTML. The idea being that most of your javascript ultimately want to modify the DOM, so run the app and copy out the rendered HTML you need to use as your stub.

An example of a test is as follows:

require("spec_helper");
require("../../public/javascripts/accounts.js");

Screw.Unit(function() {
  describe('accounts/new initially has fields', function() {
    it("should have name field", function() {
      expect($('#account_name').size()).to(equal, 1);
    });
  });

  describe('accounts/new to fail if missing name', function() {
    it("should have erroneous name field", function() {
      $('#account_submit').click();
      expect($('div.fieldWithErrors #account_name').size()).to(equal, 1)
    });
  });
});

The first test will check that an account_name field exists, and in TDD it shouldn't. So next you should go insert the HTML you need into your stub HTML file. (note: you'll also need to ensure that the form doesn't actually submit on the test, so you need to add a function to return false when the button is clicked).

When you run the tests you should find that you get nicely green/red tests in browser for any passing/failing tests.

require("spec_helper");
require("../../public/javascripts/accounts.js");

Screw.Unit(function() {  
  describe('accounts/new to fail if missing name', function() {
    it("should have erroneous name field", function() {
      $('#account_submit').click();
      expect($('div.fieldWithErrors #account_name').size()).to(equal, 1)
    });
  });
});

Next it is checking that when you submit the form, it wraps the empty form input with the standard rails error div. Run the test, get a failure. Implement the code and the tests go green. You'll probably notice that the test passes but you don't get the red box, so you need to include the CSS in the test (just below the require) to actually see what the user would:

css("../../public/stylesheets/scaffold.js");

Next Dr. Nic went through a big gotcha, in that that tests don't run in transactions. Which means the DOM isn't reset back to it's an initial state, and he highlighted it by adding an additional test to check that when you input a name that you don't get the error div. Unfortunately the test fails, as the div had been inserted by previous test.

The way to deal with it was using a before block in your test:

require("spec_helper");
require("../../public/javascripts/accounts.js");
css("../../public/stylesheets/scaffold.js");

Screw.Unit(function() {  
 before(function() {
   $('#account_name').val('');
   $('div.fieldWithErrors').remove();
 });

 describe('accounts/new to fail if missing name', function() {
   it("should have erroneous name field", function() {
     $('#account_submit').click();
     expect($('div.fieldWithErrors #account_name').size()).to(equal, 1)
   });
 });
});

If someone can come up with an easy way to snapshot the DOM/body.innerHTML and all attached events and substitute it back in after each test that would be awesome.

Defining your own matches is also pretty simple. You simply add an object with match (to do the comparison) and failure_message to display the feedback and push it onto the matchers hash/dictionary.

If anybody is wondering if they should test more, you need to watch the last 30secs of the presentation once the videos are available. Great job Nic :)

Paolo Negri - Divide and conquer riding rabbits and trading gems

Paolo had a problem where he had 1,000,000 search phrases and wanted to compare the results between Google and Bing (warning, don't try this at home… it's a breach of the terms of service). Trying to fetch 2m pages is quite a time consuming task, enter a distributed approach.

How many nodes will you need? How many workers? What mechanism should you use to distribute the work? RabbitMQ is an open source implementation of AMQP, written in Erlang. The fact it's written in Erlang is relevant is it inherently supports concurrency, fault tolerance, and distribution of tasks.

To get started you need to install the rabbitmq package and a gem:

sudo apt-get install rabbitmq
sudo gem install tmm1-amqp

There's a link to the code at the bottom so rather than transcribe it all I'll just talk through it.

First you create a worker which does the setup and then subscribe to the queue. The queue is where RabbitMQ will store all outstanding work, and the workers will request new work whenever they're free for more processing. At the other end of the process what happens is that your app pushes a message to an exchange, the exchange pops it on a queue, and then workers come take it off the queue. You can have multiple exchanges, multiple queues, and multiple workers. Queues and messages are resident in RAM, however you can persist them to disk for further fault tolerance (for a performance hit).

There are problems though, if a worker has a problem you could lose some messages. To get around it you can send an ACK message from the worker to prefetch a message. You then process the message, and acknowledge completion. Dead client connections go unacknowledged and the work is handed off elsewhere.

Another problem is there is no easy way to control the workers. The solution to this is to great a system queue. You give each worker a unique ID and explicitly define which exchange you want to use, and make it use the system queue. You then subscribe the worker to that queue. If you want to do something to the worker you simply post the appropriate command onto the system queue, and the worker will then pick up that command as soon as it can. You can also multicast messages to affect all workers.

Next up was drilling into some of the detail of fetching the pages Google/Bing. For anyone who's not done something like this, you really need to use EventMachine and EventMachine::Deferrables. It means you're not blocked waiting for socket responses, but can fire callbacks when you get a response back and you can do other tasks (fetch other pages) concurrently in the one process.

Problem #3 was working out how many workers you had, and keeping track of what they were doing. The solution was somewhat similar to the system approach, setup a heartbeat queue and have the workers post to the queue what they're doing. Then create a different worker which subscribes to that queue, takes all the messages off, and posts them somewhere for you to interrogate or view later (like a web page?).

For doing TDD with RabbitMQ there is moqueue which gives you easy testing of workers. There is also a web front-end for monitoring your RabbitMQ clients and make sure the system is alive called alice. For testing with EventMachine you've got em-spec.

Slides are available at SlideShare. Code is on Github

Pat Allan - "Sphinx - Beyond the basics"

Sphinx is a full-text search engine, and Thinking Sphinx is an ActiveRecord plugin which will use Sphinx. Once you've set it up if you want to search you just do:

Article.search "my name"

You'll also need to use a few rake tasks to create/update the indexes. Now the basics are out of the way.

Facets

Pat's written some ruby code to provide a means of offering facets (search summaries) like other engines like Solr offer. To use you do

indexed field, :facet => true
has_attribute , :fa

rebuild your indexes, then you're good to search.

@facets = Article.facets 'pancakes'
@facets == { :author => { "Pat Allan" => 12,
                         "Glenn Gillen" => 1}.
            :tags => { "breakfast" => 2,
                       "brunch" => 3}}

You can then drill into the results to return just the Articles for a specific author:

@facet.for(:author => "Pat Allan")

Excerpts

It got a bit of a cheer, apparently some people have been hanging out for them. They're basically the bold bits on a Google page of results, where the actual search terms are highlighted within the results. The syntax is:

@articles = Article.search 'pancakes'
@article.excerpts.body

Geo-searching

Pat thinks this could be the killer feature. The only gotcha is that the datatypes need to be in radians as floats. To define lats and longs on your models just do:

has latitude
has longitude

If you've already got lat and long data as degrees and decimals you can do:

has 'RADIANS(latitude)',
    :as => :latitude,
    :type => :float
has 'RADIANS(latitude)',
    :as => :latitude,
    :type => :float

Once that's done, you can find all pubs that serve pancakes near a given point:

@pubs = Pub.search 'pancakes'
  :geo => [@lat, @lng],
  :order => '@geodist ASC'

Once you've got your pubs you probably want

@pubs.each_with_geodist do |pub, distance|
end

International Sphinx

Using the character set table means you can now handle international characters easier, so an "e" with an accent can be treated as a plain old english "e". Same with other similar characters.

With word stemming you can now define a custom stemmer to help you handle stemming in various languages without too much trouble.

Different Delta Approaches

In most rails apps you want information to be available as quickly as possible, so when users update a field you don't want to have to rebuild the whole index. To get around that you can do the following:

define_index do
  set_property :delta => true
end

The downside is you end up with a new column on the table, and a little additional HTTP overhead.

Instead you can skip the extra column and use the updated_at field to manage deltas:

define_index do
  set_property :delta => :datetime, :threshold => 2.hours
end

The downside is they're not updated immediately, they're updated every 2 hours. Another alternative is to use the delayed option which pushes the request out of process using DelayedJob.

define_index do
  set_property :delta => :delayed
end

Removes indexing from HTTP requests, and the results are available soon(ish). The downside is it needs a constantly running rake task (using delayedjob) and a long queue of updates could take a while to get through.

Multi-server deployment

Basically, set the remote_spinx value to true, use the DelayedJob approach, and make sure Spinx and the database are on the same server. You'll probably want the DelayedJob and ThinkingSphinx on that server too. Pat's not actively using multi-server setups so is looking for feedback on this.

Sphinx Scopes

People have been using named_scopes with Sphinx and it's caused some problems because of the difference between SQL and Sphinx. The new sphinx scopes are an implementation that works just like named scopes, except they chain together properly and work with pagination.

Brendan Lim - Mobilize Your Rails Application

There are an estimated 4 billing mobile users, which means there are more phones than there are personal computers. Approximately half of those users have mobile web access. With that much reach you need to make your apps accessible to as many of these users as possible.

THe problem with trying to do just one webpage for all devices is problematic because of resolution differences, javascript, flash, bandwidth, etc.

Enter MobileFu. It'll detect if a user is mobile, can add custom styling based on user agent and takes you one step closer to one webpage. You use it like:

class MyController < ActionController::Base
  has_mobile_fu

  def index
    respond_to do |format
      format.html
      format.mobile
    end
  end
end

Which will output the page in a mobile friendly format. Some of the magic happens in the stylessheet_include_tag:

stylessheet_include_tag "foo"

will look for foo_iphone.css, foo_android.css, etc. so you can have custom styles for each device. You've also got various helper methods to take actions within you view:

is_mobile_device?
is_device?('blackberry')
in_mobile_view?

Rails iUI is a wrapper for the iUI user interface framework. It takes care of detecing rotating an iPhone and various other helpers. To use it you do the following in your controller:

class MyController < ActionController::Base
  acts_as_iphone_controller

  def index
    respond_to do |format|
      format.html
      format.iphone
    end
  end
end

And then within your view:

<%=include_iui_files%>

To display iPhone styled default interface objects you've the following methods:

iui_toolbar( initial_caption, search_url)
button_link_to()
iui_list(items, opts)
iui_grouped_list(items, opts, &group_by_block)

Again you need add some stuff into your views to make it work:

<body <%=register_orientation_change%>>
observe_orientation_change(url_for_options)

you'll receive params[:position] in the request to the url you put in observe_orientation_change, 0 is upright. 90 and -90 is the two various landscape modes (depending on rotation).

Next up is SMS. You can use Clickatell's API. Once you connect to the API you just do:

api.send_message(phone_number, message)

Brendan also wrote something called SMSFu. The problems are that you need to know the recipients carrier and it doesn't support as many carriers. But because it sends SMS via email it's free to send (to supported carriers). To send you just do:

deliver_sms(phone_number, carrier, message)

Now how do you receive SMS? Well you can get a shortcode (an inbound SMS number like 80800), but they're quite expensive to setup with monthly costs associated. Alternatively there are providers (like TextMarks) where you're given a keyword that users need to prepend on the front of their message, and you share a shortcode. TextMarks then send the next set of words (up to 9) as an array to you to do what you want with.

If you're wanting to receive MMS, then MMS2R is the gem to use. It strips out any carrier advertising and other stuff that may be added to the core attachment that you actually want. You'll get access to the message subject, body, and default_media (the attachment).

Jim Weirich, DHH, Obie Fernandez, Geoffrey Grosenbach, Jonathan Siegel - Q&A Session

Bah, I can't keep pace with questions and responses in a fashion that does them justice. Best wait for the videos.

Eleanor McHugh - The Ruby Guide to *nix Plumbing

Unix isn't just a server based operating system, it's a philosophy. It consists of a whole heap of little tools that all receive a little piece of input and do just a little piece of work. And they can all work together to achieve much bigger tasks.

Eleanor went through a bit of an explanation of how she felt the fascination on better threading in ruby is a bit stupid, and that the introduction of the threading approach we have in 1.8 was a backward step from 1.6. Most problems you can get around by forking the process, and threading is very rarely the appropriate solution to the problem.

The shell system is simply a process which provides you an interactive environment, job/process management, and a scripting language. However, you can get around using the standard shell scripting languages by using the under-documented shell.rb.

A little known fact is that the IO/File new methods don't just accept a string, but will also accept a file descriptor (which is just a number). So if you open 0, 1, or 2 you'll end up writing directly to STDIN, STDOUT, and STDERR respectively.

Taking the approach further means that you can effectively communicate with any process in a similar fashion by sending signals to the appropriate process descriptors.

When working with sockets in ruby, Eleanor said you should turn off reverse DNS lookups as they will greatly slow down network connection and tie up your processes needlessly.

There's also a whole host of examples of using low level system calls and examples in the slides which I've no chance of transcribing quickly enough with the right context.

Lindsay Holmwood - Behaviour driven monitorins with cucumber-nagios

Everyone is probably familiar these days with what Cucumber is and the descriptive syntax it gives you for defining tests. You can combine it with webrat and mechanize to give you a way of monitoring external websites. To get started you need to install the gem and setup a new project:

sudo gem install auxesis-cucumber-nagios
cucumber-nagios-gen project mysites
cd mysites
rake deps

You then create a new feature, for the sake of example create one to test the navigation:

cucumber-nagios-gen feature rails-underground.com navigation

which will create the file features/rails-underground.com/navigation.feature. From there you can use all the usual cucumber/webrat commands to go to a page, click links, test for the result text. To run the test you:

cucumber-nagios features/rails-underground.com/navigation.feature

cucumber-nagios will then take care of taking the cucumber results and outputting them in a nagios plugin format.

The caveats are that you're going to run problems if you site is javascript dependent. But you're embracing progressive enhancement and your site works fine without javascript, right? You could also run into some problems if you're trying to run multiple scenarios within the same feature file.

The big benefits of this approach is that you end up making your system monitoring actually test what you care about. Traditionally these tests are just a ping or socket connection. While if you've got a server down your tests will definitely fail, but there is a whole host of other cases where your site might be responsive but not usable by a real person (DB down?). Lindsay argues that this approach is really (and should be) continuous integration for systems monitoring.

Some cool examples of people extending it include telephony-systems-test which allows you to test an Asterisk/Adhearsion telephony dial plan. cucumber+dash which pull metrics out of the hosted Dash metrics system to alert you of application performance problems.

Lindsay went through what seemed like an awesome replacement for nagios that he's been working on, which is handling upto 6,000 tests per second. It's super top secret at the moment and didn't want specifics to go outside the room. He's auxesis on github, you join the dots.

I've also got summaries for Rails Underground - Day 1

Rails Underground - Day 1

2009-07-24T09:54:50+01:00

update I've also got summaries for Rails Underground - Day 2, plus videos are now online

Today kicks off the first day of talks for Rails Underground in London. Thankfully they've got usable wifi, and two days of what sound like mostly excellent talks. So over the next couple of days I'll hopefully be able to post summaries of the various talks I'm able to squeeze into.

So first up…

Fred George - Rails is a Hammer (Keynote)

Taking a queue from Martin Fowler's "Patterns of Enterprise Application Architectures" Fred goes in to discuss the various trade offs that happen in different application styles. Essentially, approaches that get you going quickly usually become exponentially more difficult as the complexity of your problem increases. Rails with it's "convention over configuration" offers a great way to prototype very quickly, but by Fowler's assumption this should get slower and slower to extend as your app complexity grows. So Fred decided to start from scratch, take his learnings from smalltalk and see how things would turn out on a new project.

Models were just pure ruby, storing to a YAML persistance layer. Sinatra took control of the controllers, with HAML/SASS for the view layer. I turns out that with very little code, YAML provided an easy way to persist the objects quickly to disk. Incrementing filenames on each save also meant that object state was versioned, and actions were simple to undo/redo (and note we're talking full objects being serialised to disk with fill state, not just instance variables/attributes).

Then to link the model with the view comes in Sinatra. It's so light and easy, does nothing more than it needs to, it's essentially just a set of regexps that push a request to the appropriate code. And how much more do you really need to do in most controllers?

Next was the view, and for anyone using HAML there is nothing groundbreaking to learn here. It's awesome, heaps better than the prescribed ERB, and you should be using it (my conclusion on Fred's discussion and my own experience, don't flame him directly). It's not just less typing, the prescriptive structure makes the code more readable and being able to use variables in CSS is incredibly liberating and DRY.

Some good debate post keynote on whether it was really a comparison of rails vs alternatives, or just a test project using stuff rails already offers you without using rails.

Gwyn Morfey - Refactoring with Fire

A high paced and amusing presentation about trying to refactor old apps. Why would you need to consider a rewrite? "Maybe it was PHP on Rails, or some guys first Rails project". The options are burn it to the ground and start again, or try and refactor it. The default option should always be to try and refactor. It's usually tested, and it works, and if you re-write it you're basically shutting down and delivering absolutely nothing for an extended period town. At which point you eventually release, with the same feature set. And even if you're re-writing in an agile fashion, your first releases still have only a minor percentage of the features expected by the users and will be considered a fail.

You refactor unless:

You have a very, very, very good reason to rewrite

You rewrite if:

You can drastically cut scope
You won't stall the business
You have access to the original coder
The existing business logic is fundamentally wrong (the requirements have changed?)

When going through the estimation phase, it's often not accurate to just take a cursory glance at the code and see how easy it is to understand. So much can be hidden under the surface that it's usually best to try a spike at implementing something new and see how easy it is.

If you're considering a re-write, you need to approach it as a complete outsider and say "I know nothing, what does this thing need to do". Otherwise you open yourself up to the stealth feature that was hidden deep in the code that you lost in your re-write. Boot it up, use the application… and watch the client use the application.

To make it work and keep the client happy, make sure that any re-write is actually adding value to the existing system. Deliver early, deliver often.

That's it on re-writing, next is how to make refactoring manageable.

Gwyn also added in a shameless plug for his "noisy partials" plugin which will insert HTML comments in the rendered views to help you identify where exactly things are being rendered. The "partial dependencies" plugin will draw a nice pretty dependency graph showing the links between all the partials in a project.

Split the long methods into smaller workable and meaningful chunks. You can then work on refactoring these bite sized pieces while you're working through the codebase. If there's code you don't understand, delete it and see what breaks. If nothing breaks, kiss it goodbye… "Deleted code is debugged code". Make sure that you're only working on the small bites, and get it back to a working state before you continue. Otherwise you'll forget what it was you were originally working on, and before you know it you'll have upgraded your entire rails stack and plugins and nothing works. And don't refactor code just because you hate it, much sure you hate it and it's in your way.

Check out Sequel as the database adapter if you're trying to move legacy data between SQL database schemas.

Desi McAdam - Working with Legacy Rails Apps - Technical Debt Hell and how to work your way out of it

Another talk on dealing with old rails projects and code, am I detecting a general theme here? So what constitutes legacy apps here?

Old Rails versions
No tests
Complexity in architecture and design
Violates best practices (fat controllers, poor separation, etc.)

Essentially, code that is difficult to change. It all boils down to technical debt.

At Hash Rocket they have rescue missions which they do for clients they inherit with these problems. First off they'll spend a couple of days doing a code audit. As Gwyn pointed out though, much of the detail isn't uncovered until you try and implement something.

So what to do and how to make it workable?

Pair all the f*ing time. You need someone to check you and stop you from being sucked in to writing the same style of bad code. If you look at it long enough you'll start to find it normal.
Be fearless. Take the risks and delete the code.
Write tests. Use cucumber to wrap the main functionality to help give yourself a sense of stability.
Stay focussed. At the point of change wrap tests around the current functionality, BDD the changes. And don't get distracted (at which point it an amp starts squealing in the background and disrupts the talk… brilliant timing)
Refactor. Always refactor with tests, and do it as you go. It helps estimate velocity, removes technical debt, and alleviates the broken windows problem.
Use Git (or anything that lets you branch easily)
Use your customer. Deliver early and often, and get them to help you test. They've got the best idea of what it's meant to do.

The specific things hash Rocket do in their approach is:

Update the app to the latest rails version
Update the plugins and external dependencies (or remove/un-fork where possible).
Fix the easy stuff. Clean the formatting, remove comment code as you're unlikely to get much value from is.
Use the free stuff. Replace custom code with built-in rails functionality, use new features (like named scopes) to decrease complexity.

Obie Fernandez - Blood, Sweat, and Rails

Obie got introduced to ruby via Aslak of Cucumber fame back in 2004, with rails in 2005 for a client in London.

Sales & Marketing

First piece of advice, controversy can be useful. One of the first things Obie did was writing the top 10 reasons why Java sucks. Just make sure you can take the abuse, especially if you deserve it… he's willing to admit his initial ideas for Rails Maturity Model sucked. Confidence always win. On the flip side of that, don't ever seem desperate. The moment you go in and convey, even subconsciously, that you need the work you make the client wonder why you need the work. They'd much rather work with someone who is in demand. Referrals will be the lifeblood of your business, so go ahead and videotape your clients. Get them used to it, video meetings, testimonials, and help document the project. It's a tactical nuclear weapon.

Make it personal, pick up the phone and call a potential client. It's way too easy to just fire off an email or IM.

Legal Matters

You need to write bulletproof contracts. Keeping them simple is not ideal, too many potential problems. Master services agreements, warranties, digital signing/acceptance, etc. all need to be included. Don't negotiate under pressure. You'll end up agreeing to things that you shouldn't to and it will cost you more in the long run. And when things go wrong, lawyers cost a lot of money.

Finances

Don't undercapitalise. Don't do fixed bid contracts. None of Hash Rocket's contracts state a deliverable, you're buying time and expertise. There is no definition of scope. By documenting scope you're breaking the philosophy of agile and you lose the ability to adjust scope. You'll probably need to gather some momentum before these are applicable. Make sure you charge what you are worth (a slide showing that Obie billed out at $250/hr, other devs at $190/hr for short term work $150/hr for anything greater than 6 months). Allow some budget for non-paying clients. When you're extending credit, some times they're just not going to pay (either because they're bastards, they ran out of funding, they've drained the kids college fund, etc.). Allow some budget for product development, everyone wants to be 37signals eventually. Don't invoice manually, Harvest is a good example of how to handle it easily.

Agile Practices

You shouldn't have to defend agile. Just start from the assumption that "this is how we work" and that it's that the normal case, then you're not having to justify a change in approach. Always do agile by the book first, don't try and tweak it and then wonder why it doesn't work. Storycard the work, either on index cards or in something like Pivotal Tracker.

Client Relationships

Execution is absolutely critical. If you're not consistently striving for excellence then it's all worthless. Going hand-in-hand with that is perception is reality. If you can't relay to the client that you're doing an excellent job then it's worthless also. Bend the rules, the client just wants a result and doesn't really care how you go about it. In some cases you should even break the rules. Another simple one is to establish contact, especially with big value contracts, call them every day. Mind their budget, especially if you've made the jump to time based contracts. The responsibility falls back on to you to make sure you're not burning through all their cash so have transparency on who's working and what the cash burn rate is. Do what you can to win their people over. Fire clients if they deserve to be fired. Hire people on a contract to perm basis (bah! I say they should all be contractors if they're any good). Keep your employees constantly learning. And make them pair all the time. Empower your employees for change, be open for them to challenge the generally accepted practices (like pairing all the time). You need to actively work to make that possible. Make the work environment appealing and always have fun. Keep everyone in the loop all the time.

Charles Nutter - The Present and Future of JRuby (and the Future of Rails as it relates to JRuby)

The current version of JRuby is 1.3.1, it's ruby 1.8.6 compatible (give or take, it can't do continuations and some other things). Has some ruby 1.9 support (somewhere 75%-90% done).

It's roughly equivalent in performance terms to ruby 1.9, with real native threads and runs rails fine. It takes at least 0.5secs to start up, but can take several seconds so it's not great. Once up, it should be faster than 1.9 in almost all cases. Most ruby application bottlenecks are in the core classes and not ruby itself (string manipulations, working with hashes, etc.) and the JRuby performance in these classes is mixed which makes providing meaningful benchmarks almost impossible. Some cases are great, others are bad. Also because of the iterative optimisations that the JVM does internally a single run of an application doesn't show real world performance. To demonstrate Charles ran a fractal generation program 5 times and you could see how much quicker the latter runs were.

It's the only ruby implementation with true native threads. Again another demonstration showing how with JRuby you can effectively max out two cores, yet with regular ruby, even 1.9, you only ever really get the equivalent of 1 core working (spread across the two cores). Next was a really cool demo that I'll do no justice here. Basically using FFI to call C functions and have them then execute ruby callbacks.

Next was a demo of starting an app (Typo) using Glassfish. Just download the gem, go into the app and:

"glassfish -e production"

If you're running threadsafe rails it will start just one instance of the JVM and keep memory usage low. In any event, it takes care of concurrency and all the usual problems for you.

Ruby 1.9 support

It's at about maybe 80% of 1.9.1. The 1.9.2 release adds a bunch more, but they'd like some help (so get in touch if you're interested/able). He also gave a cool example of a change to regexps in 1.9 that I'd not seen. If you want to extract a grouped match from a regexp pattern you'd normally do something like:

str = "Welcome to Rails Underground"
matched = str.match(/Welcome to Rails (.*)/)
matched[1] == "Underground"

The problem is if you change the regexp to:

matched = str.match(/(Welcome) to Rails (.*)/)

matched[1] now becomes "Welcome" rather than underground, and your code brakes. In ruby 1.9 you can name grouped matches:

str = "Welcome to Rails Underground"
matched = str.match(/Welcome to Rails (?<conf>.*)/)
matched[:conf] == "Underground"

What's missing?

One thing that isn't missing is 1.8 compatibility, it's a mostly solved problem. There are some edge cases that wont work, but it's got the best coverage of all the alternative estimates. On the todo list is better performance, better java integration. To highlight some of the upcoming performance improvements Charles went through the intermediate code the new compiler generates which was quite fascinating to see. Makes me glad I don't have to build compilers for a living though. As a result there is also a "ruby2java" executable which generates java classes from ruby code now. And to further help with the java and ruby integration there is a "become_java" method you can call on a ruby class to turn it into an object that can be used natively within java.

The other tasks are rubifying the java libraries. Hibernate, Ant, Maven, etc. It's basically an attempt to keep all of the rich functionality and performance that java offers, without the inherent java ugliness. The big news is that Hibernate is basically done now and just needs to be wrapped up in a nice ruby DSL. But you can use Hibernate as a persistance layer in JRuby now.

How can you help

Use JRuby every chance you get
Help improve JRuby (especially 1.9)
Start evangelising ruby at java conferences
Study Groovy, Scala, Clojure, etc
Study java libraries and help rubify them

Gwyn Morfey (filling in for Laurie Young) - Agile Deployment with Ruby

You're going to need a lot of servers, so you may as well make the job as easy as possible. You've got lost of serving options, you should be using passenger though. There's now Ruby Intelligent Packaging called RIP which is great and you should use it, except it doesn't work… so just keep an eye on it. You also need to have continuous integration setup and working. CruiseControl.rb isn't great but it works.

As far as actually deploying, capistrano is there. But use webistrano, it's a thin layer over capistrano. The main benefit is you get an audit history so you can see who deployed what, where, and when. Next is puppet to manage your server config. It's not as quick is making disk images, but it's more flexible and much easier to maintain (Personally, I'd advocate Chef instead of puppet).

You need multiple servers (either actual dedicated iron for each, or virtualised environments) to deploy to. A test server for external testers and/or clients to run through the completed development prior to a production deployment. A pre-production server that closely mirrors the production server. It's there as a final sanity check prior to going live, and there to check there isn't some configuration difference on the production box that will trip you up. Make sure you test both the up and down migrations, it should work, but it often doesn't. And there is nothing worth than discovering the rollback won't work once you get to production.

On production, make sure you've got notifications setup. "Exception Notifier", "Get Exceptional", "HopToad", "New Relic" test them all and make a call, just use one. Service monitoring, make sure things stay up with monit or god. To help with peak loads consider a Content Delivery Network like Limelight Networks.

You'll probably need a debug server. If you push things all the way to production and you get errors, it's quite possibly data related. You need to copy the logs, assets, and all other data over to try and work through. In practice, New Bamboo don't run a separate debug server but just re-provision pre-production very quickly (it's there, it's got the production code already, it's almost ready to go).

Martin Kleppmann - Sales & payments in your app

The Ruby Invoicing Framework tries to give a solid foundation where you can build a solid web application. It offers a few classes:

Invoice - you owe us money
Credit note - oops you billed to much
Payment -thanks for the cash

They all inherit from LedgerItem which throws is a naming throw back to accounting terms, but it essentially offers you access to discrete billing line items and two companies, one on each side of the transaction. What it gives you for free is an automatically rendered invoice (an actual invoice you can post/email to the client) with all the legally required fields included.

Accountant need to know the exact dates of transactions, that bank statements reconciling, and VAT/sales or other applicable taxes.

insert various slides about accounting practices and how ledger items are calculated. I'd expect anybody who runs their own business already knows all this stuff

I've had to roll most of this stuff myself over the past 12-18 months, and for the most part our schemas and approach is almost identical. Just to save myself the maintenance headache, I'll probably look at porting over to this invoicing gem. I also need to look at the existing open standards (UBL, XBRL-GL, OAccounts, and OASIS) and see how they fit in.

George Palmer - CouchDB and Ruby

I've covered the background on what CouchDB is in previous posts, or it's otherwise readily available with a quick search so I'll leave you to find the really high level stuff out yourself. Documents are stored as JSON, you get subsets of documents via views. Done.

Unlike relational databases which you pay a performance hit on indexes when you insert data, CouchDB makes you pay the penalty on the first read of a view. That means if you path insert a bunch of data, the next request to see that information is going to be slow. In a typical read heavy web application, this isn't that noticeable. If you're inserting data often, then there are ways to mitigate the problem.

So when should you use CouchDB? If you want a schema-less database. FriendFeed was used as an example, given the amount of data they store adding a new column in MySQL could take several hours. Often it also relates more directly to your real-world models. George went through an example from constructing his "5ft Shelf" site and the complexities associated with books (numerous ISBNs, different titles in different countries, difference retail prices in different locales, different editions, etc.) and how difficult it is to map all these permutations in a relational system. The final scenario is when you know you're going to need replication or sharding, either for offline capability or scaling.

You shouldn't use it is when your problem domain is very fixed. Finance is a good example, real estate transactions are another.

Enter George's couch_foo plugin which is a way of interfacing with CouchDB in an ActiveRecord fashion. Because everything is ultimately stored as JSON, there's no real concept of datatypes. So long as the attributes can be translated to JSON and reconstructed from JSON you're good to use any datatype/object you type. But the plugin does natively support validations, associations, callbacks, and pretty much everything you'd expect from AR. One gotcha is that if you try and order by an attribute that isn't exposed by the key, it will return all the results and then order them in ruby. It's a double sting if you then try and limit, as you've pulled back a heap of records you never needed.

Performance wise, George said there's a whole heap of naive benchmarks claiming CouchDB is faster than this or that. It's a different approach, some things are going to be quicker, some are going to be slower. The latest CouchDB release (0.9) does offer some speed improvements over previous versions though.