Ruby Pond

JRuby, and why bouncers are assholes

Thu, 30 Jul 2009 12:58:30 GMT

I’m normally not one to use this site as a soap box or an outlet for opinion pieces. But I figured it’s easier than being continually baited into replying on all the various blogs/forums/groups out there. There’s a whole heap of talk in the ruby community at the moment on JRuby and it’s place in the ecosystem. Most of it has been kicked off in response to Charles Nutter’s post on JRuby’s importance to Ruby. I was fortunate enough to see Charles’ presentation at Rails Underground last week and have a chat to him, and he had some great insights and examples of why JRuby actually is pretty damn awesome.

The sky is falling

The only real problem I have with Charles’ post is the inference that all is lost for the Ruby community if we don’t embrace JRuby quickly. It’s not, it won’t suddenly disappear. What we will possibly lose is the migration of great java developers like Jim Weirich and Charles. I’m willing to excuse a bit of sensationalism in his post though, we’re effectively talking about his livelihood here. He needs to get people to pay attention, he needs people to use JRuby, he needs more of us to embrace it.

Whatever his motivations, he’s right.

Java sucks

Yes, we know. But you’ve missed the point. This isn’t about giving us* java, it’s about giving *java ruby. Or better yet, giving ruby the JVM. Just try and think through the argument logically; why on earth would a java developer who is unhappy with java for whatever reason move to a new language, that gave you all the same obstacles? They wouldn’t (no matter what your opinion of the average java dev is). It’s about combining the flexibility of ruby with the good bits of java, the bits that are good enough to make people want to stick with java.

Java/The JVM is slow

“Good morning Mr. Kettle, I’m the pot… and you my friend look black”. Really? People that are happy enough with the performance of ruby 1.8 are pointing the “unperformant” stick at other interpreters? Give me a break. The fact is that the JRuby implementation (in most cases) is as good as 1.8, and in many is as good as 1.9. The fact that such a minor section of the community seems to be aggressively migrating to 1.9 highlights the fact that performance really isn’t such a high concern for you in real life work, it’s just a great scape goat for debates. Oh, and worried about that additional few secs startup overhead from starting the JVM for little scripts? Then don’t start it up

Ruby/rails is a competitive advantage

Bull. Shit. Your competitive advantage is your expertise and the ability to meet your client requirements, as efficiently as possible. Period. Ruby and Rails have helped you have an edge for the past few years, but there’s many alternatives that are closing the gap. Most of my clients don’t care what technology I use so long as I do the best job for them and make the most efficient use of their money. And why should they care? The few clients I have worked for that would care, would probably still prefer I use java over ruby.

The EngineYard deal is acknowledgement from Sun that even they think JRuby is dead

Wrong. JRuby is never going to be a core part of Sun’s business. There are so many people that are happy with the status quo that they will continue coding away in java until such a day it might be declared deceased. Why would they have any desire to invest both time and money to stop such a minor outflow from their business? They wouldn’t, they’ve got bigger problems to deal with. On the other hand, EngineYard is all about infrastructure… and infrastructure for hosting ruby apps specifically. Higher performance ruby implementations means they need to invest less in hardware for the same return. It has a direct and appreciable impact on their bottom line. Sun was a great fit initially to allow the guys access to the java team so we could get the most out of the JVM. EngineYard have more to gain out of a successful JRuby at this point than Sun do, so they’re a much better fit.

We don’t want more java developers or the java community

This is the one that irks me the most. I can only recall two occasions in life where one group of people proclaims they are too cool to associate with another group of people. Bouncers at nightclubs, and “that” group in high school. Bouncers are assholes, and those kids from high school probably grew up to become bouncers.

If you’ve got a problem with someone else, that’s your own insecurities and your problem to deal with, don’t infect the rest of us with it. What’s the issue here? You don’t like their code? Then fork it, fix it, or do it yourself and the problem magically goes away. If you don’t want to do that, then shut your pie hole and pay some respect to those that actually do the work. You don’t like the competition? Go read a book or five, find some better developers to pair with, stop trying to hide.

If we’re using a technology that really is as great as we make out, we should see it as inevitable that the community is going to grow significantly in size at some point. I think Malcolm Gladwell refers to it as the tipping point. The technical ability of the average developer may drop a little. Commercial interests may infest us and move conferences to Vegas. But for every 10 hacks you still end up with a Jim Weirich, Giles Bowkett, David A. Black, etc. And for every large conference you end up with a Rails Underground, Scotland on Rails, or Ruby Hoedown.

Looking at the world through ruby tinted glasses

The fact is that for a long time the performance of ruby has sucked. We’ve worked around it and justified it for a long time, and for the vast majority of people it didn’t suck bad enough to cause problems. Heck, we were all happy using webrick until Zed Shaw came along and showed us a better way. But 3rd party library support has also sucked. Trying to create PDFs was a nightmare for ages. Trying to work with Microsoft document formats still is. I can think of at least 5 occasions where I’ve solved a simple problem in another language because doing it in ruby would have been far too difficult.

For all the talent the community has, there are so many areas where good support is still lacking that it’s unrealistic to think there are enough talented people who care and have the time to do anything about it. Combine that with the fact java has at least (commercially) a 10 year head start and the gap between the two languages is huge. Java has plenty to offer us.

Why can’t it all just be peace, love, rainbows, and freakin flying ponies?

All this arguing seems to do is instill a whole “us and them” mentality. There’s no need for it to be that way. Embrace JRuby, take advantage of the performance of the JVM, get access to a whole swathe of new libraries that go far beyond what your current list of gems currently offers you… and get to keep using the language you love.

One of the things I admire most about the ruby community is the ability it has to support innovations and allow the proverbial cream to rise to the top. WebBrick was replaced by Mongrel, then we had Thin, and now we’ve got Passenger. SOAP became REST. AttachmentFu became Paperclipped, etc. I’d hate if in this instance it wasn’t allowed to happen because of a whole heap of misconceptions about what JRuby was and what it had to offer.

Or maybe we just need to get a mascot like Django and chill the fuck out.

Rails Underground - Day 2

Sat, 25 Jul 2009 08:59:19 GMT

update: videos are now online

I was a little late to the kick-off of the keynote on day 2 with Yehuda. For the most part if is another discussion on what’s happening in Rails 3 so I don’t think I’ve missed anything I hadn’t heard before. He then started to go into more detail on how things have changed than he has previously, probably due to the progress since I last saw him in March…

Yehuda Katz - Keynote

Yehuda went through the upcoming changes in approach and philosophy on rails Dev. Having well defined contracts, consistent conventions, and allowing things be be changeable.

One of the changes of not is splitting off ideas and interfaces into modules which can then be imported into a class at runtime. He used the analogy that your parents didn’t define everything you could do when you were born, you can learn new things. So to when you define ta class you don’t have to define the complete interface up front, nor if you want to make changes to you need to re-implement the entire class. Instead you can have sub-sections of interfaces included or changed at runtime where appropriate.

In Rails 3 this translates into better separation of the components and clearly defined interfaces. No longer are you forced to use ActionView, you just need to use something that is “ActionView compliant”. The same goes for ActiveRecord, your models are just “ActionModel compliant”.

An ActionView compliant interface just needs to implement the following 3 methods:

for_controller(paths, assigns, controller)
render_template(opts)
render_partial(opts)

An ActiveModel compliant interface needs the following 4 methods:

new_record?
valid?
errors
model_name

That last model_name method needs to return plural, partial name, and a few other things rails expects internally. It’ll only be 10-20 lines of code to implement manually, or there is a module to include if you just want the default behaviour.

Next up is a cleaner separation of components, one which we got a demo of was splitting out the ActiveModel::Validations. Now you can create your own ActiveModel compliant persistance/ORM layer, and pull in the existing validations for free regardless of where it’s stored (YAML, CouchDB, etc.).

By implementing the above you get a system that fully works with ActionPack, so all the usual form and error helpers will just work.

The other benefit is that pulling in support for these doesn’t need you to load the entire Rails stack and rob your machine of RAM, it’s only a 1.5mb hit.

Next was the new ActionController::Metal demo. At first I wondered why I’d bother, it was slightly more vebose than just using Rackable or rolling it manually, and seemed to carry more overhead than Sinatra. But it gives you the ability to opt into additional features easily. If you wall before/after filters you can pull in the callbacks module. If you want better rendering options (various formats, etc) pull in the rendering module, if you want different layouts pull in that module. ActionController::Base is basically just ActionController::Metal with all of the modules pulled in, so you’ve now got the way to strip back to metal, or anything in between.

The final piece is ActiveSupport. If for some reason you want to pull in the entire ActiveSupport behemoth you just call:

require 'active_support/all'

Otherwise you can call in the various newly separated modules, one of note was

require 'active_support/ruby/shim'

Which backports a bunch of useful ruby1.9 features on datatypes that could be easily implemented in pure ruby. So you can use them now, and when you upgrade to ruby1.9 you get the performance improvement of having them implemented in C.

The end result is a new level of granularity. Just because you want to use DataMapper doesn’t mean you have to say goodbye to all of ActiveRecord. So you can opt-out of just the tiny pieces you don’t want. Conversely, you can start completely bare and opt-in on various features to build your own framework on top of rails. You could now build Sinatra on top of rails, and when the complexity of your app increases you can much more easily pull in the additional features you need. All in all, it sounds really cool. Cool enough that I might even try using it on my next app.

Dr. Nic Williams - Dead Simple JavaScript Unit Tests in Rails with Blue Ridge and Screw.Unit

Dr. Nic has dropped a very important bombshell right at the start of his talk. The while Australia may indeed be losing the current 5 game series, we are still holders of The Ashes. Now on with the less important aspects of the presentation.

Nic has a bunch of open source plugins and projects which he could have come and given a presentation on, but he’s so impressed with BlueRidge that he feels it’s the most important thing to happen to the Rails ecosystem since Cucumber. At a high level it allows you to do both in browser and headless testing, it fits in with the existing rake tasks, and means that javascript testing should just become part of the usual TDD/BDD development approach and not be an afterthought in Cucumber.

Generating a javascript test with BlueRidge creates a spec file, and a fixture file which will give you some test HTML. The idea being that most of your javascript ultimately want to modify the DOM, so run the app and copy out the rendered HTML you need to use as your stub.

An example of a test is as follows:

require("spec_helper");
require("../../public/javascripts/accounts.js");

Screw.Unit(function() {
  describe('accounts/new initially has fields', function() {
    it("should have name field", function() {
      expect($('#account_name').size()).to(equal, 1);
    });
  });

  describe('accounts/new to fail if missing name', function() {
    it("should have erroneous name field", function() {
      $('#account_submit').click();
      expect($('div.fieldWithErrors #account_name').size()).to(equal, 1)
    });
  });
});

The first test will check that an account_name field exists, and in TDD it shouldn’t. So next you should go insert the HTML you need into your stub HTML file. (note: you’ll also need to ensure that the form doesn’t actually submit on the test, so you need to add a function to return false when the button is clicked).

When you run the tests you should find that you get nicely green/red tests in browser for any passing/failing tests.

require("spec_helper");
require("../../public/javascripts/accounts.js");

Screw.Unit(function() {  
  describe('accounts/new to fail if missing name', function() {
    it("should have erroneous name field", function() {
      $('#account_submit').click();
      expect($('div.fieldWithErrors #account_name').size()).to(equal, 1)
    });
  });
});

Next it is checking that when you submit the form, it wraps the empty form input with the standard rails error div. Run the test, get a failure. Implement the code and the tests go green. You’ll probably notice that the test passes but you don’t get the red box, so you need to include the CSS in the test (just below the require) to actually see what the user would:

css("../../public/stylesheets/scaffold.js");

Next Dr. Nic went through a big gotcha, in that that tests don’t run in transactions. Which means the DOM isn’t reset back to it’s an initial state, and he highlighted it by adding an additional test to check that when you input a name that you don’t get the error div. Unfortunately the test fails, as the div had been inserted by previous test.

The way to deal with it was using a before block in your test:

require("spec_helper");
require("../../public/javascripts/accounts.js");
css("../../public/stylesheets/scaffold.js");

Screw.Unit(function() {  
 before(function() {
   $('#account_name').val('');
   $('div.fieldWithErrors').remove();
 });

 describe('accounts/new to fail if missing name', function() {
   it("should have erroneous name field", function() {
     $('#account_submit').click();
     expect($('div.fieldWithErrors #account_name').size()).to(equal, 1)
   });
 });
});

If someone can come up with an easy way to snapshot the DOM/body.innerHTML and all attached events and substitute it back in after each test that would be awesome.

Defining your own matches is also pretty simple. You simply add an object with match (to do the comparison) and failure_message to display the feedback and push it onto the matchers hash/dictionary.

If anybody is wondering if they should test more, you need to watch the last 30secs of the presentation once the videos are available. Great job Nic :)

Paolo Negri - Divide and conquer riding rabbits and trading gems

Paolo had a problem where he had 1,000,000 search phrases and wanted to compare the results between Google and Bing (warning, don’t try this at home… it’s a breach of the terms of service). Trying to fetch 2m pages is quite a time consuming task, enter a distributed approach.

How many nodes will you need? How many workers? What mechanism should you use to distribute the work? RabbitMQ is an open source implementation of AMQP, written in Erlang. The fact it’s written in Erlang is relevant is it inherently supports concurrency, fault tolerance, and distribution of tasks.

To get started you need to install the rabbitmq package and a gem:

sudo apt-get install rabbitmq
sudo gem install tmm1-amqp

There’s a link to the code at the bottom so rather than transcribe it all I’ll just talk through it.

First you create a worker which does the setup and then subscribe to the queue. The queue is where RabbitMQ will store all outstanding work, and the workers will request new work whenever they’re free for more processing. At the other end of the process what happens is that your app pushes a message to an exchange, the exchange pops it on a queue, and then workers come take it off the queue. You can have multiple exchanges, multiple queues, and multiple workers. Queues and messages are resident in RAM, however you can persist them to disk for further fault tolerance (for a performance hit).

There are problems though, if a worker has a problem you could lose some messages. To get around it you can send an ACK message from the worker to prefetch a message. You then process the message, and acknowledge completion. Dead client connections go unacknowledged and the work is handed off elsewhere.

Another problem is there is no easy way to control the workers. The solution to this is to great a system queue. You give each worker a unique ID and explicitly define which exchange you want to use, and make it use the system queue. You then subscribe the worker to that queue. If you want to do something to the worker you simply post the appropriate command onto the system queue, and the worker will then pick up that command as soon as it can. You can also multicast messages to affect all workers.

Next up was drilling into some of the detail of fetching the pages Google/Bing. For anyone who’s not done something like this, you really need to use EventMachine and EventMachine::Deferrables. It means you’re not blocked waiting for socket responses, but can fire callbacks when you get a response back and you can do other tasks (fetch other pages) concurrently in the one process.

Problem #3 was working out how many workers you had, and keeping track of what they were doing. The solution was somewhat similar to the system approach, setup a heartbeat queue and have the workers post to the queue what they’re doing. Then create a different worker which subscribes to that queue, takes all the messages off, and posts them somewhere for you to interrogate or view later (like a web page?).

For doing TDD with RabbitMQ there is moqueue which gives you easy testing of workers. There is also a web front-end for monitoring your RabbitMQ clients and make sure the system is alive called alice. For testing with EventMachine you’ve got em-spec.

Slides are available at SlideShare. Code is on Github

Pat Allan - “Sphinx - Beyond the basics”

Sphinx is a full-text search engine, and Thinking Sphinx is an ActiveRecord plugin which will use Sphinx. Once you’ve set it up if you want to search you just do:

Article.search "my name"

You’ll also need to use a few rake tasks to create/update the indexes. Now the basics are out of the way.

Facets

Pat’s written some ruby code to provide a means of offering facets (search summaries) like other engines like Solr offer. To use you do

indexed field, :facet => true
has_attribute , :fa

rebuild your indexes, then you’re good to search.

@facets = Article.facets 'pancakes'
@facets == { :author => { "Pat Allan" => 12,
                         "Glenn Gillen" => 1}.
            :tags => { "breakfast" => 2,
                       "brunch" => 3}}

You can then drill into the results to return just the Articles for a specific author:

@facet.for(:author => "Pat Allan")

Excerpts

It got a bit of a cheer, apparently some people have been hanging out for them. They’re basically the bold bits on a Google page of results, where the actual search terms are highlighted within the results. The syntax is:

@articles = Article.search 'pancakes'
@article.excerpts.body

Geo-searching

Pat thinks this could be the killer feature. The only gotcha is that the datatypes need to be in radians as floats. To define lats and longs on your models just do:

has latitude
has longitude

If you’ve already got lat and long data as degrees and decimals you can do:

has 'RADIANS(latitude)',
    :as => :latitude,
    :type => :float
has 'RADIANS(latitude)',
    :as => :latitude,
    :type => :float

Once that’s done, you can find all pubs that serve pancakes near a given point:

@pubs = Pub.search 'pancakes'
  :geo => [@lat, @lng],
  :order => '@geodist ASC'

Once you’ve got your pubs you probably want

@pubs.each_with_geodist do |pub, distance|
end

International Sphinx

Using the character set table means you can now handle international characters easier, so an “e” with an accent can be treated as a plain old english “e”. Same with other similar characters.

With word stemming you can now define a custom stemmer to help you handle stemming in various languages without too much trouble.

Different Delta Approaches

In most rails apps you want information to be available as quickly as possible, so when users update a field you don’t want to have to rebuild the whole index. To get around that you can do the following:

define_index do
  set_property :delta => true
end

The downside is you end up with a new column on the table, and a little additional HTTP overhead.

Instead you can skip the extra column and use the updated_at field to manage deltas:

define_index do
  set_property :delta => :datetime, :threshold => 2.hours
end

The downside is they’re not updated immediately, they’re updated every 2 hours. Another alternative is to use the delayed option which pushes the request out of process using DelayedJob.

define_index do
  set_property :delta => :delayed
end

Removes indexing from HTTP requests, and the results are available soon(ish). The downside is it needs a constantly running rake task (using delayedjob) and a long queue of updates could take a while to get through.

Multi-server deployment

Basically, set the remote_spinx value to true, use the DelayedJob approach, and make sure Spinx and the database are on the same server. You’ll probably want the DelayedJob and ThinkingSphinx on that server too. Pat’s not actively using multi-server setups so is looking for feedback on this.

Sphinx Scopes

People have been using named_scopes with Sphinx and it’s caused some problems because of the difference between SQL and Sphinx. The new sphinx scopes are an implementation that works just like named scopes, except they chain together properly and work with pagination.

Brendan Lim - Mobilize Your Rails Application

There are an estimated 4 billing mobile users, which means there are more phones than there are personal computers. Approximately half of those users have mobile web access. With that much reach you need to make your apps accessible to as many of these users as possible.

THe problem with trying to do just one webpage for all devices is problematic because of resolution differences, javascript, flash, bandwidth, etc.

Enter MobileFu. It’ll detect if a user is mobile, can add custom styling based on user agent and takes you one step closer to one webpage. You use it like:

class MyController < ActionController::Base
  has_mobile_fu

  def index
    respond_to do |format
      format.html
      format.mobile
    end
  end
end

Which will output the page in a mobile friendly format. Some of the magic happens in the stylessheet_include_tag:

stylessheet_include_tag "foo"

will look for foo_iphone.css, foo_android.css, etc. so you can have custom styles for each device. You’ve also got various helper methods to take actions within you view:

is_mobile_device?
is_device?('blackberry')
in_mobile_view?

Rails iUI is a wrapper for the iUI user interface framework. It takes care of detecing rotating an iPhone and various other helpers. To use it you do the following in your controller:

class MyController < ActionController::Base
  acts_as_iphone_controller

  def index
    respond_to do |format|
      format.html
      format.iphone
    end
  end
end

And then within your view:

<%=include_iui_files%>

To display iPhone styled default interface objects you’ve the following methods:

iui_toolbar( initial_caption, search_url)
button_link_to()
iui_list(items, opts)
iui_grouped_list(items, opts, &group_by_block)

Again you need add some stuff into your views to make it work:

<body <%=register_orientation_change%>>
observe_orientation_change(url_for_options)

you’ll receive params[:position] in the request to the url you put in observe_orientation_change, 0 is upright. 90 and -90 is the two various landscape modes (depending on rotation).

Next up is SMS. You can use Clickatell’s API. Once you connect to the API you just do:

api.send_message(phone_number, message)

Brendan also wrote something called SMSFu. The problems are that you need to know the recipients carrier and it doesn’t support as many carriers. But because it sends SMS via email it’s free to send (to supported carriers). To send you just do:

deliver_sms(phone_number, carrier, message)

Now how do you receive SMS? Well you can get a shortcode (an inbound SMS number like 80800), but they’re quite expensive to setup with monthly costs associated. Alternatively there are providers (like TextMarks) where you’re given a keyword that users need to prepend on the front of their message, and you share a shortcode. TextMarks then send the next set of words (up to 9) as an array to you to do what you want with.

If you’re wanting to receive MMS, then MMS2R is the gem to use. It strips out any carrier advertising and other stuff that may be added to the core attachment that you actually want. You’ll get access to the message subject, body, and default_media (the attachment).

Jim Weirich, DHH, Obie Fernandez, Geoffrey Grosenbach, Jonathan Siegel - Q&A Session

Bah, I can’t keep pace with questions and responses in a fashion that does them justice. Best wait for the videos.

Eleanor McHugh - The Ruby Guide to *nix Plumbing

Unix isn’t just a server based operating system, it’s a philosophy. It consists of a whole heap of little tools that all receive a little piece of input and do just a little piece of work. And they can all work together to achieve much bigger tasks.

Eleanor went through a bit of an explanation of how she felt the fascination on better threading in ruby is a bit stupid, and that the introduction of the threading approach we have in 1.8 was a backward step from 1.6. Most problems you can get around by forking the process, and threading is very rarely the appropriate solution to the problem.

The shell system is simply a process which provides you an interactive environment, job/process management, and a scripting language. However, you can get around using the standard shell scripting languages by using the under-documented shell.rb.

A little known fact is that the IO/File new methods don’t just accept a string, but will also accept a file descriptor (which is just a number). So if you open 0, 1, or 2 you’ll end up writing directly to STDIN, STDOUT, and STDERR respectively.

Taking the approach further means that you can effectively communicate with any process in a similar fashion by sending signals to the appropriate process descriptors.

When working with sockets in ruby, Eleanor said you should turn off reverse DNS lookups as they will greatly slow down network connection and tie up your processes needlessly.

There’s also a whole host of examples of using low level system calls and examples in the slides which I’ve no chance of transcribing quickly enough with the right context.

Lindsay Holmwood - Behaviour driven monitorins with cucumber-nagios

Everyone is probably familiar these days with what Cucumber is and the descriptive syntax it gives you for defining tests. You can combine it with webrat and mechanize to give you a way of monitoring external websites. To get started you need to install the gem and setup a new project:

sudo gem install auxesis-cucumber-nagios
cucumber-nagios-gen project mysites
cd mysites
rake deps

You then create a new feature, for the sake of example create one to test the navigation:

cucumber-nagios-gen feature rails-underground.com navigation

which will create the file features/rails-underground.com/navigation.feature. From there you can use all the usual cucumber/webrat commands to go to a page, click links, test for the result text. To run the test you:

cucumber-nagios features/rails-underground.com/navigation.feature

cucumber-nagios will then take care of taking the cucumber results and outputting them in a nagios plugin format.

The caveats are that you’re going to run problems if you site is javascript dependent. But you’re embracing progressive enhancement and your site works fine without javascript, right? You could also run into some problems if you’re trying to run multiple scenarios within the same feature file.

The big benefits of this approach is that you end up making your system monitoring actually test what you care about. Traditionally these tests are just a ping or socket connection. While if you’ve got a server down your tests will definitely fail, but there is a whole host of other cases where your site might be responsive but not usable by a real person (DB down?). Lindsay argues that this approach is really (and should be) continuous integration for systems monitoring.

Some cool examples of people extending it include telephony-systems-test which allows you to test an Asterisk/Adhearsion telephony dial plan. cucumber+dash which pull metrics out of the hosted Dash metrics system to alert you of application performance problems.

Lindsay went through what seemed like an awesome replacement for nagios that he’s been working on, which is handling upto 6,000 tests per second. It’s super top secret at the moment and didn’t want specifics to go outside the room. He’s auxesis on github, you join the dots.

I’ve also got summaries for Rails Underground - Day 1

Rails Underground - Day 1

Fri, 24 Jul 2009 08:54:50 GMT

update I’ve also got summaries for Rails Underground - Day 2, plus videos are now online

Today kicks off the first day of talks for Rails Underground in London. Thankfully they’ve got usable wifi, and two days of what sound like mostly excellent talks. So over the next couple of days I’ll hopefully be able to post summaries of the various talks I’m able to squeeze into.

So first up…

Fred George - Rails is a Hammer (Keynote)

Taking a queue from Martin Fowler’s “Patterns of Enterprise Application Architectures” Fred goes in to discuss the various trade offs that happen in different application styles. Essentially, approaches that get you going quickly usually become exponentially more difficult as the complexity of your problem increases. Rails with it’s “convention over configuration” offers a great way to prototype very quickly, but by Fowler’s assumption this should get slower and slower to extend as your app complexity grows. So Fred decided to start from scratch, take his learnings from smalltalk and see how things would turn out on a new project.

Models were just pure ruby, storing to a YAML persistance layer. Sinatra took control of the controllers, with HAML/SASS for the view layer. I turns out that with very little code, YAML provided an easy way to persist the objects quickly to disk. Incrementing filenames on each save also meant that object state was versioned, and actions were simple to undo/redo (and note we’re talking full objects being serialised to disk with fill state, not just instance variables/attributes).

Then to link the model with the view comes in Sinatra. It’s so light and easy, does nothing more than it needs to, it’s essentially just a set of regexps that push a request to the appropriate code. And how much more do you really need to do in most controllers?

Next was the view, and for anyone using HAML there is nothing groundbreaking to learn here. It’s awesome, heaps better than the prescribed ERB, and you should be using it (my conclusion on Fred’s discussion and my own experience, don’t flame him directly). It’s not just less typing, the prescriptive structure makes the code more readable and being able to use variables in CSS is incredibly liberating and DRY.

Some good debate post keynote on whether it was really a comparison of rails vs alternatives, or just a test project using stuff rails already offers you without using rails.

Gwyn Morfey - Refactoring with Fire

A high paced and amusing presentation about trying to refactor old apps. Why would you need to consider a rewrite? “Maybe it was PHP on Rails, or some guys first Rails project”. The options are burn it to the ground and start again, or try and refactor it. The default option should always be to try and refactor. It’s usually tested, and it works, and if you re-write it you’re basically shutting down and delivering absolutely nothing for an extended period town. At which point you eventually release, with the same feature set. And even if you’re re-writing in an agile fashion, your first releases still have only a minor percentage of the features expected by the users and will be considered a fail.

You refactor unless:

You have a very, very, very good reason to rewrite

You rewrite if:

You can drastically cut scope
You won’t stall the business
You have access to the original coder
The existing business logic is fundamentally wrong (the requirements have changed?)

When going through the estimation phase, it’s often not accurate to just take a cursory glance at the code and see how easy it is to understand. So much can be hidden under the surface that it’s usually best to try a spike at implementing something new and see how easy it is.

If you’re considering a re-write, you need to approach it as a complete outsider and say “I know nothing, what does this thing need to do”. Otherwise you open yourself up to the stealth feature that was hidden deep in the code that you lost in your re-write. Boot it up, use the application… and watch the client use the application.

To make it work and keep the client happy, make sure that any re-write is actually adding value to the existing system. Deliver early, deliver often.

That’s it on re-writing, next is how to make refactoring manageable.

Gwyn also added in a shameless plug for his “noisy partials” plugin which will insert HTML comments in the rendered views to help you identify where exactly things are being rendered. The “partial dependencies” plugin will draw a nice pretty dependency graph showing the links between all the partials in a project.

Split the long methods into smaller workable and meaningful chunks. You can then work on refactoring these bite sized pieces while you’re working through the codebase. If there’s code you don’t understand, delete it and see what breaks. If nothing breaks, kiss it goodbye… “Deleted code is debugged code”. Make sure that you’re only working on the small bites, and get it back to a working state before you continue. Otherwise you’ll forget what it was you were originally working on, and before you know it you’ll have upgraded your entire rails stack and plugins and nothing works. And don’t refactor code just because you hate it, much sure you hate it and it’s in your way.

Check out Sequel as the database adapter if you’re trying to move legacy data between SQL database schemas.

Desi McAdam - Working with Legacy Rails Apps - Technical Debt Hell and how to work your way out of it

Another talk on dealing with old rails projects and code, am I detecting a general theme here? So what constitutes legacy apps here?

Old Rails versions
No tests
Complexity in architecture and design
Violates best practices (fat controllers, poor separation, etc.)

Essentially, code that is difficult to change. It all boils down to technical debt.

At Hash Rocket they have rescue missions which they do for clients they inherit with these problems. First off they’ll spend a couple of days doing a code audit. As Gwyn pointed out though, much of the detail isn’t uncovered until you try and implement something.

So what to do and how to make it workable?

Pair all the f*ing time. You need someone to check you and stop you from being sucked in to writing the same style of bad code. If you look at it long enough you’ll start to find it normal.
Be fearless. Take the risks and delete the code.
Write tests. Use cucumber to wrap the main functionality to help give yourself a sense of stability.
Stay focussed. At the point of change wrap tests around the current functionality, BDD the changes. And don’t get distracted (at which point it an amp starts squealing in the background and disrupts the talk… brilliant timing)
Refactor. Always refactor with tests, and do it as you go. It helps estimate velocity, removes technical debt, and alleviates the broken windows problem.
Use Git (or anything that lets you branch easily)
Use your customer. Deliver early and often, and get them to help you test. They’ve got the best idea of what it’s meant to do.

The specific things hash Rocket do in their approach is:

Update the app to the latest rails version
Update the plugins and external dependencies (or remove/un-fork where possible).
Fix the easy stuff. Clean the formatting, remove comment code as you’re unlikely to get much value from is.
Use the free stuff. Replace custom code with built-in rails functionality, use new features (like named scopes) to decrease complexity.

Obie Fernandez - Blood, Sweat, and Rails

Obie got introduced to ruby via Aslak of Cucumber fame back in 2004, with rails in 2005 for a client in London.

Sales & Marketing

First piece of advice, controversy can be useful. One of the first things Obie did was writing the top 10 reasons why Java sucks. Just make sure you can take the abuse, especially if you deserve it… he’s willing to admit his initial ideas for Rails Maturity Model sucked. Confidence always win. On the flip side of that, don’t ever seem desperate. The moment you go in and convey, even subconsciously, that you need the work you make the client wonder why you need the work. They’d much rather work with someone who is in demand. Referrals will be the lifeblood of your business, so go ahead and videotape your clients. Get them used to it, video meetings, testimonials, and help document the project. It’s a tactical nuclear weapon.

Make it personal, pick up the phone and call a potential client. It’s way too easy to just fire off an email or IM.

Legal Matters

You need to write bulletproof contracts. Keeping them simple is not ideal, too many potential problems. Master services agreements, warranties, digital signing/acceptance, etc. all need to be included. Don’t negotiate under pressure. You’ll end up agreeing to things that you shouldn’t to and it will cost you more in the long run. And when things go wrong, lawyers cost a lot of money.

Finances

Don’t undercapitalise. Don’t do fixed bid contracts. None of Hash Rocket’s contracts state a deliverable, you’re buying time and expertise. There is no definition of scope. By documenting scope you’re breaking the philosophy of agile and you lose the ability to adjust scope. You’ll probably need to gather some momentum before these are applicable. Make sure you charge what you are worth (a slide showing that Obie billed out at $250/hr, other devs at $190/hr for short term work $150/hr for anything greater than 6 months). Allow some budget for non-paying clients. When you’re extending credit, some times they’re just not going to pay (either because they’re bastards, they ran out of funding, they’ve drained the kids college fund, etc.). Allow some budget for product development, everyone wants to be 37signals eventually. Don’t invoice manually, Harvest is a good example of how to handle it easily.

Agile Practices

You shouldn’t have to defend agile. Just start from the assumption that “this is how we work” and that it’s that the normal case, then you’re not having to justify a change in approach. Always do agile by the book first, don’t try and tweak it and then wonder why it doesn’t work. Storycard the work, either on index cards or in something like Pivotal Tracker.

Client Relationships

Execution is absolutely critical. If you’re not consistently striving for excellence then it’s all worthless. Going hand-in-hand with that is perception is reality. If you can’t relay to the client that you’re doing an excellent job then it’s worthless also. Bend the rules, the client just wants a result and doesn’t really care how you go about it. In some cases you should even break the rules. Another simple one is to establish contact, especially with big value contracts, call them every day. Mind their budget, especially if you’ve made the jump to time based contracts. The responsibility falls back on to you to make sure you’re not burning through all their cash so have transparency on who’s working and what the cash burn rate is. Do what you can to win their people over. Fire clients if they deserve to be fired. Hire people on a contract to perm basis (bah! I say they should all be contractors if they’re any good). Keep your employees constantly learning. And make them pair all the time. Empower your employees for change, be open for them to challenge the generally accepted practices (like pairing all the time). You need to actively work to make that possible. Make the work environment appealing and always have fun. Keep everyone in the loop all the time.

Charles Nutter - The Present and Future of JRuby (and the Future of Rails as it relates to JRuby)

The current version of JRuby is 1.3.1, it’s ruby 1.8.6 compatible (give or take, it can’t do continuations and some other things). Has some ruby 1.9 support (somewhere 75%-90% done).

It’s roughly equivalent in performance terms to ruby 1.9, with real native threads and runs rails fine. It takes at least 0.5secs to start up, but can take several seconds so it’s not great. Once up, it should be faster than 1.9 in almost all cases. Most ruby application bottlenecks are in the core classes and not ruby itself (string manipulations, working with hashes, etc.) and the JRuby performance in these classes is mixed which makes providing meaningful benchmarks almost impossible. Some cases are great, others are bad. Also because of the iterative optimisations that the JVM does internally a single run of an application doesn’t show real world performance. To demonstrate Charles ran a fractal generation program 5 times and you could see how much quicker the latter runs were.

It’s the only ruby implementation with true native threads. Again another demonstration showing how with JRuby you can effectively max out two cores, yet with regular ruby, even 1.9, you only ever really get the equivalent of 1 core working (spread across the two cores). Next was a really cool demo that I’ll do no justice here. Basically using FFI to call C functions and have them then execute ruby callbacks.

Next was a demo of starting an app (Typo) using Glassfish. Just download the gem, go into the app and:

"glassfish -e production"

If you’re running threadsafe rails it will start just one instance of the JVM and keep memory usage low. In any event, it takes care of concurrency and all the usual problems for you.

Ruby 1.9 support

It’s at about maybe 80% of 1.9.1. The 1.9.2 release adds a bunch more, but they’d like some help (so get in touch if you’re interested/able). He also gave a cool example of a change to regexps in 1.9 that I’d not seen. If you want to extract a grouped match from a regexp pattern you’d normally do something like:

str = "Welcome to Rails Underground"
matched = str.match(/Welcome to Rails (.*)/)
matched[1] == "Underground"

The problem is if you change the regexp to:

matched = str.match(/(Welcome) to Rails (.*)/)

matched[1] now becomes “Welcome” rather than underground, and your code brakes. In ruby 1.9 you can name grouped matches:

str = "Welcome to Rails Underground"
matched = str.match(/Welcome to Rails (?<conf>.*)/)
matched[:conf] == "Underground"

What’s missing?

One thing that isn’t missing is 1.8 compatibility, it’s a mostly solved problem. There are some edge cases that wont work, but it’s got the best coverage of all the alternative estimates. On the todo list is better performance, better java integration. To highlight some of the upcoming performance improvements Charles went through the intermediate code the new compiler generates which was quite fascinating to see. Makes me glad I don’t have to build compilers for a living though. As a result there is also a “ruby2java” executable which generates java classes from ruby code now. And to further help with the java and ruby integration there is a “become_java” method you can call on a ruby class to turn it into an object that can be used natively within java.

The other tasks are rubifying the java libraries. Hibernate, Ant, Maven, etc. It’s basically an attempt to keep all of the rich functionality and performance that java offers, without the inherent java ugliness. The big news is that Hibernate is basically done now and just needs to be wrapped up in a nice ruby DSL. But you can use Hibernate as a persistance layer in JRuby now.

How can you help

Use JRuby every chance you get
Help improve JRuby (especially 1.9)
Start evangelising ruby at java conferences
Study Groovy, Scala, Clojure, etc
Study java libraries and help rubify them

Gwyn Morfey (filling in for Laurie Young) - Agile Deployment with Ruby

You’re going to need a lot of servers, so you may as well make the job as easy as possible. You’ve got lost of serving options, you should be using passenger though. There’s now Ruby Intelligent Packaging called RIP which is great and you should use it, except it doesn’t work… so just keep an eye on it. You also need to have continuous integration setup and working. CruiseControl.rb isn’t great but it works.

As far as actually deploying, capistrano is there. But use webistrano, it’s a thin layer over capistrano. The main benefit is you get an audit history so you can see who deployed what, where, and when. Next is puppet to manage your server config. It’s not as quick is making disk images, but it’s more flexible and much easier to maintain (Personally, I’d advocate Chef instead of puppet).

You need multiple servers (either actual dedicated iron for each, or virtualised environments) to deploy to. A test server for external testers and/or clients to run through the completed development prior to a production deployment. A pre-production server that closely mirrors the production server. It’s there as a final sanity check prior to going live, and there to check there isn’t some configuration difference on the production box that will trip you up. Make sure you test both the up and down migrations, it should work, but it often doesn’t. And there is nothing worth than discovering the rollback won’t work once you get to production.

On production, make sure you’ve got notifications setup. “Exception Notifier”, “Get Exceptional”, “HopToad”, “New Relic” test them all and make a call, just use one. Service monitoring, make sure things stay up with monit or god. To help with peak loads consider a Content Delivery Network like Limelight Networks.

You’ll probably need a debug server. If you push things all the way to production and you get errors, it’s quite possibly data related. You need to copy the logs, assets, and all other data over to try and work through. In practice, New Bamboo don’t run a separate debug server but just re-provision pre-production very quickly (it’s there, it’s got the production code already, it’s almost ready to go).

Martin Kleppmann - Sales & payments in your app

The Ruby Invoicing Framework tries to give a solid foundation where you can build a solid web application. It offers a few classes:

Invoice - you owe us money
Credit note - oops you billed to much
Payment -thanks for the cash

They all inherit from LedgerItem which throws is a naming throw back to accounting terms, but it essentially offers you access to discrete billing line items and two companies, one on each side of the transaction. What it gives you for free is an automatically rendered invoice (an actual invoice you can post/email to the client) with all the legally required fields included.

Accountant need to know the exact dates of transactions, that bank statements reconciling, and VAT/sales or other applicable taxes.

insert various slides about accounting practices and how ledger items are calculated. I’d expect anybody who runs their own business already knows all this stuff

I’ve had to roll most of this stuff myself over the past 12-18 months, and for the most part our schemas and approach is almost identical. Just to save myself the maintenance headache, I’ll probably look at porting over to this invoicing gem. I also need to look at the existing open standards (UBL, XBRL-GL, OAccounts, and OASIS) and see how they fit in.

George Palmer - CouchDB and Ruby

I’ve covered the background on what CouchDB is in previous posts, or it’s otherwise readily available with a quick search so I’ll leave you to find the really high level stuff out yourself. Documents are stored as JSON, you get subsets of documents via views. Done.

Unlike relational databases which you pay a performance hit on indexes when you insert data, CouchDB makes you pay the penalty on the first read of a view. That means if you path insert a bunch of data, the next request to see that information is going to be slow. In a typical read heavy web application, this isn’t that noticeable. If you’re inserting data often, then there are ways to mitigate the problem.

So when should you use CouchDB? If you want a schema-less database. FriendFeed was used as an example, given the amount of data they store adding a new column in MySQL could take several hours. Often it also relates more directly to your real-world models. George went through an example from constructing his “5ft Shelf” site and the complexities associated with books (numerous ISBNs, different titles in different countries, difference retail prices in different locales, different editions, etc.) and how difficult it is to map all these permutations in a relational system. The final scenario is when you know you’re going to need replication or sharding, either for offline capability or scaling.

You shouldn’t use it is when your problem domain is very fixed. Finance is a good example, real estate transactions are another.

Enter George’s couch_foo plugin which is a way of interfacing with CouchDB in an ActiveRecord fashion. Because everything is ultimately stored as JSON, there’s no real concept of datatypes. So long as the attributes can be translated to JSON and reconstructed from JSON you’re good to use any datatype/object you type. But the plugin does natively support validations, associations, callbacks, and pretty much everything you’d expect from AR. One gotcha is that if you try and order by an attribute that isn’t exposed by the key, it will return all the results and then order them in ruby. It’s a double sting if you then try and limit, as you’ve pulled back a heap of records you never needed.

Performance wise, George said there’s a whole heap of naive benchmarks claiming CouchDB is faster than this or that. It’s a different approach, some things are going to be quicker, some are going to be slower. The latest CouchDB release (0.9) does offer some speed improvements over previous versions though.

Scotland on Rails 09 - Videos are up

Thu, 21 May 2009 10:41:37 GMT

Not much more to say really, the title sums it up. For those that weren’t there or those that were that just want a recap go and check out the Scotland on rails videos now.

And I will eventually get around to doing that round-up of the post/out-of-conference activities to do should you be looking to attend the conference yourself next year.

Belated Scotland on Rails round up

Fri, 08 May 2009 13:15:08 GMT

Okay, so I’m almost a month and a half late in finishing this. And the content might already seem a little dated given RailsConf is currently on in Vegas and EuRuKo kicks off this weekend in Barcelona. But better late than never.

The Summary

So in summary, how was it? Pretty good and great value. It wasn’t blow my mind amazing, but there was enough good stuff going on to justify the trip. Plus Scotland is a beautiful country, and Edinburgh in-particular a great city, so any means of justifying a business expense to get there is worthwhile ;)

So onto the sessions I saw or chatted in depth to people about.

Sessions

Keynote - Marcel Molina Jr

I’d heard of Marcel before, I’d even seen his name grace the cover of some books I own, but I didn’t know much about him beyond that. For those like me, he was one of the original core team at 37signals that worked on Rails. And his keynote was quite a personal look at the beginnings of the framework. Digging up old usenet posts from DHH showing how he’s trying to learn ruby, to slowly building the hype around rails, to a public release. For a presentation that was essentially reading old emails and newsgroup postings I found it thoroughly enjoyable. It was particularly interesting to see just how quickly DHH went from new to ruby to releasing probably one of the most widely used tools written in the language.

Implementing Geo-awareness - Jim Remsik

I’d had big hopes for this presentation. I’ve dabbled with geo-awareness stuff before, heck I’ve even written my own rails mapping plugin because I thought YM4R and the alternatives weren’t very ruby (btw. it’s in use for the collaborative mapping/direction site I quickly rustled up for my girlfriend Map.ly). I was really looking at some idea re how to make an existing site more “geo aware”, and this presentation was little more than “how to put a google map on your site 101”. For all my complaints about the existing plugins and their approach, they’re still quite simple to use. With any level of ruby/rails knowledge, this talk could have probably been reduced to 5-10mins.

Getting Git: Getting effective project management - Scott Chacon

Scott is part of the team at Logical Awesome working on Github. He’s also done the peepcode on git, and maintains the community book. He really knows his git, and by all accounts I’m disappointed I missed this one for the geo-awareness session. Thankfully the guys I pair with most days, Graham Ashton and Jon Ramsey, went and filled me in somewhat on it. The first half was a bit of an introduction on what it is, why it’s different, how to use it, etc. that could mostly be ignored if you’ve used it at any length. Apparently the second half of the talk (which unfortunately he tried to cram into 10mins) was amazing. Showing how to use bisect to identify which commit introduced a bug as one of the highlights. The slides are available at github and hopefully the videos will be up soon.

Ruby on the couch eating potatoes - Alexander Lang

A well times presentation for my perspective. I’ve been looking at a number of relational database alternatives of late for one of my side projects, and CouchDB had come up as one of the most likely contenders. So I’d done a fair degree of research on CouchDB itself, but not much on the most suitable adapter for using it with rails.

Alex did a good job going through all the various know adapters for talking to CouchDB (including his own, CouchPotato) and pointed out the pros and cons of each. Amazingly for a presentation like this he admitted that you should avoid his. For those interested, the winner was Paul Carey’s RelaxDB. A point of contention which spilled onto the twittersphere was Alex’s recommendation to avoid George Palmer’s CouchFoo (George was in the audience). The rationale being that CouchFoo, much like CouchPotato, is too “ActiveRecord” and that document-based databases are different to relational databases so this means the approach must be inherently wrong.

Personally I think no, the rationale is wrong. Sure, you don’t want to treat CouchDB like a relational database because you’re missing the benefits gained in the document approach and kind of missing the whole point. But that doesn’t mean ActiveRecord for CouchDB is wrong. ActiveRecord does not have to mean Object Relational Mapper, it’s just up until this point it’s all people have used it for. In fact, there is is nothing in the name ActiveRecord that inherently infers it is only for relational schemas. I think if people had grown up being accustomed to AR being an Object Document Mapper instead of an ORM then this argument would be moot. In rails you interface with an object, the adapter takes care of how to store it be-it in a document store or a relational store. We’ve got a pretty well documented API for AR, with lots of validation rules and other plugins that are useful. If you then decide to go build a relational schema in your document database that isn’t the fault of CouchFoo, just like AR can’t be blamed if you decide to try and keep everything in a single big flat table in MySql.

My own conclusion after some research was the two best adapters out there were RelaxDB and CouchFoo. RelaxDB seems the best (I’ll go into details in a future post) but CouchFoo is ideal if you want to transition quickly or need access to some of the AR helpers/mixins. So all in all, a good presentation and made me go back and re-assess my own research on the topic and we ultimately shared the same conclusion albeit with different reasoning.

Merb and Rails 3.0 - Yehuda Katz

Probably not a lot to go through here which wont be out of date due to RailsConf. An interesting and high-paced presentation from Yehuda on what’s coming in rails 3.0, some of the design decisions, etc. Probably not a huge amount of new information for anybody that had been using merb or following the public updates.

In Praise of non-fixtured data - Kevin Barnes

I often wonder how I’d go on stage in front of a bunch of peers, but I’m thankful that many of the presenters at these conferences looks as nervous up there as I’d be. Kevin made me rethink if I should ever bother. A really excellent and clear presenter that was a joy to listen to.

That being said, I didn’t take from this as much as I’d hoped. It’s my own fault as the talk was exactly what was presented on the label, mostly about why you shouldn’t use fixtured data and then a look at what the alternatives are. I was looking for a bit more of a detailed comparison of the pros/cons of each of the object factory approaches (much like Alex had done in the CouchDB talk) but it was really just a cursory mention of them all and then a look at how to use the one Kevin had developed, ObjectDaddy.

Confessions of a PackRat - Scott Raymond

I only caught the tail-end of this talk, but from those I spoke to it was quite insightful. Scott went into detail about how the developed a Facebook app and then needing to scale quickly to deal with the load. Everything from scaling into EC2, switching between technologies, and having to handle people trying to game the system. The 10 mins I saw were thoroughly enjoyable.

Advanced Deployment - Johnathon Weiss

This is one that I felt wasn’t at all what was described on the label. I’d normally consider deployment something more akin to “getting your stuff onto servers”. Instead this covered lots of high-level architectural options you have available to you, the alternatives for each, but without any real explanation on why you should choose one over another. An interesting topic, but pitched to the wrong crowd with the wrong name I feel. Much better for manager types with little-to-no hands on rails experience. I think anybody who’s actually deployed an application would have known the answers to all the theoretical questions raise.

The Ruby Object Model - Dave Thomas

The guy that brought is the Pragmatic Programmers and a number of ruby books and videos from that stable gave a great presentation on the object model with ruby. The shoeless maestro strutted around the stage and made what could be a very dull and overly technical topic seem approachable and simplistic. When you call a method where does it look to first to run it? Where does it bubble up to? In what order? And how do you access an anonymous method? All this arnd more was answered! Not sure if I picked up anything new, but it was a great cementing of my understanding in a highly graphical fashion. 6 weeks on when I’m debugging the internals of something I’m still visualising Dave’s charts in my head to help me. Much of the content you can go through at leisure and in more detail in the $5 videos from pragprog.

R-house - Fernand Galiana

Fernand took the noble risk of trying to do a live demonstration of the possibilities of using home automation with ruby. Thankfully for the most part it worked, and when it didn’t he was entertaining enough to cover it and continue. And I have to appreciate the level of work that has gone into R-house and the libraries it depends on to make this happen, with such a range of different home automation equipment out there and each with there own slight nuances on how to interface it can’t have been a simple job. A great presentation, not sure how soon I’ll be able to use any of it though.

How to makes a successful Rails app - Steven A Bristol

A presentation with no slides?! The horror!! Steven was another excellent presenter, and with a great presentation to boot. He went into what it takes to actually make a successful Rails app (hint: actually releasing it is a big part), and the advice was sound. The other big one which I think many developers ignore, I know I’ve been guilty of it, is trying to do the design yourself. Pay for a few days of their time and make it look professional. He was awfully opinionated on a number of topics without any real basis. That was fine when I agreed with him, I thought he was full of shit when he didn’t justify it ;) One point that niggled at me throughout most of the presentation was that the reason I know of Steven and his company was through the the open-source Lovd by Less social networking framework they’d built in rails. I’d needed something like it once upon a time and looked at Lovd, and the code was horrible. So here I was being lectured by someone on how to make a Rails app who’d produced some of the ugliest code I’d tried to use.

Then the bombshell hit (paraphrasing), “I don’t care what the code looks like if it works, just get it out the door and see if people like it. Anybody who’s seen some of our code will know it’s horrible”. And he was right, on all counts. But… I still had hesitations reservations. This was an open source framework, so the consumers in this instance are developers. So for the same reasons that you should get a designer to make your site look good for regular punters, I think you should take the time to make your code elegant and usable for developers. He’s got a great looking product which does what it needs to, it’s a shame that it’s such a pain to use and made me question the validity of what he was saying. To that end, and rather than continuing to pay other developers to extend my own social networking engine I’ve taken it upon myself to fix everything I think is broken with Lovd. There’s still a long way to go and I’ll make a more formal announcement once it’s nearer completion but it’s already come a long way IMHO. Get it from github.

Who needs Photoshop? Image manipulation in Ruby - Cory Forsyth

Reading through this initially I thought it was going was a bit of a sensational headline for what was going to be an intermediate level look at rmagick. And given my own experience with rmagick, I couldn’t possibly stomach the thought of it ever being considered a viable photoshop replacement. I was wrong.

This was another thoroughly entertaining and informative presentation. Cory went through in detail the logic behind the one feature that has been making me want to upgrade to the latest version of photoshop, content aware scaling, complete with code samples. Next up was facial recognition which could be applied to suggest smart cropping, or apparently put party hats on cats. It’s most likely made my CS4 upgrade redundant, which paid for the trip numerous times over.

Edgecase dialog: Ruby code review - Joe O’Brian and Jim Weirich

It sounded like a boring topic, it should have been a boring topic, and a very odd note on which to end a conference. It turns out Joe and Jim are brilliant entertainers taking the audience through a hypothetical role play of a consultant coming in to refactor an existing rails code base. While it’s unlikely they’ll be invited back to do their return for the Edinburgh Fringe Festival in August, they thoroughly deserved their rapturous applause at the end. I’ve not got any notes unfortunately, I’m not sure if that’s because I was laughing too hard or most of it was common sense. In any event, it was a great reminder that you should step back and look at you code to make sure you aren’t trying to be clever for the sake of it. Simple solutions are often the best.

Seeya next year

It was well worth the trip. The conference was super cheap so it’s really not hard to justify the expense and Edinburgh is a great place to visit. I’ve been there as both a tourist and lived there for 12 months, so I’ll do a subsequent post for those looking to attend next year with my own off-conference highlights.

Capturing a form cancel

Fri, 06 Feb 2009 22:40:02 GMT

I’ve been meaning to write this one up for months now, ever since I originally posted the sexy forms in rails. The form builder I talk about in that post makes it easy to put a submit and cancel button on a the form, and a few people have asked how to handle the cancel action. Some have suggested using javascript to redirect back using an onclick, ick! What happens if the user doesn’t have javascript support? Sure you might try and justify it by saying, “Well, how many users don’t have javascript these days?” but there is absolutely no reason why this has to use javascript, so lets make it work for everybody.

It’s actually much easier than you’d think, within application.rb I’ve got the following:

class ApplicationController < ActionController::Base
  before_filter :catch_cancel, :update => [:create, :update, :destroy]
  after_filter :set_referrer, :only => [:index, :show]

  private
    def set_referrer
      session[:referrer] = url_for(params)
    end

    def catch_cancel
      redirect_to session[:referrer] if params[:commit] == "Cancel"
    end
end

The logic being, that if we’ve got a fully RESTful architecture the only pages we’ll ever want to go “back” will be the index or show actions on each controller. So we cheekily call urlfor in an after filter with the current params to get the path of the page we’ve just display the user, and stash it in the session. Theoretically it may be possible to use ENV[“HTTPREFERER”] instead but it’s not guaranteed, this approach however ensures we keep track of the page to go back to explicitly ourselves.

Then, to piece it all together there’s a before filter checking if the user has clicked the submit button labeled with “Cancel”. You’ll need to change that equality test dependent on what text you put on your button. You may also need to move the before filter into the controllers that require them, depending on the order it needs to run with other filters you have.

Hope that helps.

Ruby Manor Videos Available

Fri, 16 Jan 2009 14:38:44 GMT

For anybody that was unable to attend (like me), or for those that just want a recap then a selection of the Ruby Manor videos are now available for download. Hopefully some more will be posted soon.

For any of the bits that are missing you can read through Graham’s review of the various sessions. Ruby Manor part 1 and Ruby Manor part 2

scRUBYt! Gets Plugins!

Fri, 16 Jan 2009 01:18:45 GMT

Yes! You heard right! As you may have gathered, it’s been a rather frantic month of development for scRUBYt! and currently this is the addition I’m most proud of. One of the most common requests used to be along the lines of “when do you plan to support xxx format output?”. Now, scRUBYt! is oblivious to output formats. That’s right, it natively supports nothing, nada, zilch. But to make it useful, we’ve written a Hash output plugin which we’re shipping it with.

How to use a scRUBYt! output plugin

Firstly, you need to make sure you have the plugin you require installed. At the time of writing there will only be the two I’ve written, Hash and XmlFile. Then in your ruby file require the plugin. As the current edge release isn’t yet packaged as a gem you’ll need to test this with the github checkout and reference the output plugin explicitly:

require "plugins/scrubyt_xml_file_output/scrubyt_xml_file_output"

If you’ve been following the tutorials talking about web scraping with the new version for the past few weeks you’ll have seen how to direct output to a plugin. To request Hash output it’s:

@extractor = Scrubyt::Extractor.new :output => :hash do
    fetch "http://www.google.com/search?&q=ruby"
    result "//html/body/div[5]/div/div/h2/a"
end

and for XmlFile it is:

@file = File.open("results.xml", "w")
@extractor = Scrubyt::Extractor.new :output => :xml_file, :file => @file do
    fetch "http://www.google.com/search?&q=ruby"
    result "//html/body/div[5]/div/div/h2/a"
end

The XmlFile output takes an additional parameter which is the file to stream the results out to.

Creating your own plugin

That’s great for those of you that are happy with XML or Hash output, but what about if you want some other custom format? Well it’s time to create your own. I’ll show you the actual code that implements the XmlFile output to show you how simple it is:

require 'rexml/document'
require "#{File.dirname(__FILE__)}/inflector"
require "#{File.dirname(__FILE__)}/inflections"

class Scrubyt::Output::XmlFile < Scrubyt::Output::Plugin  
  @subscribers = {}
  on_initialize :setup_file
  before_extractor :open_root_node
  after_extractor :close_root_node
  on_save_result :save_xml


  def setup_file(args = {})
    @file = args[:file]
  end

  def open_root_node(*args)
    @file.write("<root>")
  end

  def save_xml(name, results)
    if results.is_a?(::Hash)
      @file.write results.to_xml
    else
      results.each do |result|
        @file.write result.to_xml(name)
      end
    end
  end

  def close_root_node(*args)
    @file.write("</root>")
  end
end

The require lines at the top are only needed for this output format. REXML to construct the XML tags for me, and some inflections I’ve put together to turn the Hash and Array objects into XML. Now into analysing the class proper.

class Scrubyt::Output::XmlFile < Scrubyt::Output::Plugin  
  @subscribers = {}

At the moment, you’ll need to inintialize this instance variable to be an empty Hash for the events to get attached correctly. I’m looking for a way to remove it, stay tuned. But for now you’ll need to put it in.

on_initialize :setup_file
before_extractor :open_root_node
after_extractor :close_root_node
on_save_result :save_xml

Here we’ve got four events to listen for, the concept should be familiar if you’re coming from Rails. Essentially all we are doing is saying “When we initialize run the setupfile method. Before the extractor actually starts, run the method called openrootnode. Whenever we get a result to save, call savexml. And finally, after the extractor run the method called closerootnode.”

def setup_file(args = {})
  @file = args[:file]
end

This is fairly straightforward. If you’ve got any custom logic that needs to happen when the output plugin is initialized you can place it in here. Any parameter that is passed in to Extractor.new() is passed through for you to access here.

def open_root_node(*args)
  @file.write("<root>")
end

Now just to open the XML file, and keep it somewhat consistent with the old scRUBYt! XML output we open a node within the file.

def save_xml(name, results)
  if results.is_a?(::Hash)
    @file.write results.to_xml
  else
    results.each do |result|
      @file.write result.to_xml(name)
    end
  end
end

Here is where the majority of the magic happens. The save_xml method will be passed the desired name for the result, and a hash of the results. This is essentially the same format you’d get if you used the Hash output format, except for each individual detail block rather than then entire extractor.

The reason for the if/else scenario is for when results are not part of a detail block. If you’re just returning results straight (like the Google example at the top of this post) then “results” in this context will be a list/Array of all the matching results rather than a Hash.

Passing results back to the extractor

Not everyone is going to want to stream results out to a file though, so to deal with this you can make a results method available on the instance of your plugin. As I said earlier, even Hash operates as a plugin now so we can see an example of how this work in the Hash output plugin:

class Scrubyt::Output::Hash < Scrubyt::Output::Plugin
  @subscribers = {}
  on_initialize :setup_results
  on_save_result :store_hash

  def setup_results(args = {})
    @results = []
  end

  def results
    @results
  end

  def store_hash(name, passed_results)
    @results << passed_results
  end
end

Here we setup a @results instance oninitialize, and then onsaveresults simply pushes the passedresults into @results. Confused yet? Hopefully the code is clear enough to make sense.

All that happens then is that back in your extractor definition the call to @extractor.results is passed through to the first output plugin it can find.

Naming Conventions and Namespacing

The only additional requirement for a plugin to work in scRUBYt! is that it is correctly named and namespaced. As you may have noticed the ones I’ve provided are called Scrubyt::Output::Hash and Scrubyt::Output::XmlFile, that means they can be targeted using :output => :hash and :output => :xmlfile respectively. If you wanted to call your output GlennsBadExample it would be namespaced as Scrubyt::Output::GlennsBadExample and you’d then just need to require the appropriate file and use :output => :glennsbad_example

Oh the possibilities! So what’s next?

We’re only just starting to see the possibilities that this will offer our extractors. It opens up the possibility of pushing results not only to a different format, but possibly a completely different service. It’s now trivial to create an output format that streams results directly into backgroundRB, a nanite worker, or a web service for further processing and data warehousing. By the time you read this, you’ll also be able to pass in an array of outputs like :output => [:hash, :xml_file] and have both plugins generate the appropriate format(s). For the scraper I’m currently working on where I have two different companies wanting the same data, this could be just the ticket for interfacing directly to their API as I scrape.

I’d love to hear what ideas people might have for this, or how you think it could be improved. We’re really hopeful that this is the kind of thing that makes developing and extending scRUBYt! really easy for those with more complicated needs.

Web Scraping - The Amazon Example Finale

Thu, 15 Jan 2009 20:52:12 GMT

So yesterday I showed you a fairly contrived example of how to build a web scraper using scRUBYt! to get data from Amazon (they’ve got an API that would be much easier and more robust if you need access to this info, but that’s not the point at the moment). But if you look at the results, they’re not the greatest. There is too much noise in some fields, and we probably want to share there data with another system so a ruby based Hash object isn’t going to work.

Removing empty results with scRUBYt!

For various reasons, sometimes you may not get all the data you want back for every record. It’s usually related to your result definition being too restrictive, or a change in format on a specific page. Maybe the price information is in a different DIV if it is on sale. In any event, you need to make a decision on what to do. In the new release of scRUBYt! there are three immediate options that come to mind, but we may well build more in if required. First, and what we did yesterday in the example, is to do nothing. You’ll get the nil/empty result returned back to you to handle as you see fit. Second, you can simply drop any fields that are nil (I’ve just displayed the first few results):

@extractor = Scrubyt::Extractor.new do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a" do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']", :remove_blank => true
    saving "//td[@class='price']"
    isbn "//li[text()*='ISBN-10:']"
  end
end
puts @extractor.results.inspect
=> [{:book=>[{:title=>"The Ruby Programming Language [ILLUSTRATED]  (Paperback)"}, 
             {:price=>"$26.39"}, 
             {:saving=>"$13.60\n      (34%)\n    "}, 
             {:isbn=>"ISBN-10: 0596516177"}]}
    {:book=>[{:title=>"The Ruby Programming Language (Paperback)"}, 
             {:saving=>nil}, 
             {:isbn=>"ISBN-10: 020171096X"}]}, 
    {:book=>[{:title=>"Beginning Ruby: From Novice to Professional (Beginning from Novice to Professional) (Paperback)"}, 
             {:price=>"$26.39"}, 
             {:saving=>"$13.60\n      (34%)\n    "}, 
             {:isbn=>"ISBN-10: 1590597664"}]}
    ...]

We’ve set :remove_blank to true on the price field, and as a result you’ll see that the 2nd result contains no price element. Alternatively, you could drop any detail block (in this example, any single book) which is missing the field:

@extractor = Scrubyt::Extractor.new do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a" do
    title "//h1[@class='parseasinTitle']", :required => true
    price "//b[@class='priceLarge']", :required => true
    saving "//td[@class='price']"
    isbn "//li[text()*='ISBN-10:']"
  end
end
puts @extractor.results.inspect
=> [{:book=>[{:title=>"The Ruby Programming Language [ILLUSTRATED]  (Paperback)"},
             {:price=>"$26.39"},
             {:saving=>"$13.60\n      (34%)\n    "},
             {:isbn=>"ISBN-10: 0596516177"}]},
    {:book=>[{:title=>"Beginning Ruby: From Novice to Professional (Beginning from Novice to Professional) (Paperback)"},
             {:price=>"$26.39"},
             {:saving=>"$13.60\n      (34%)\n    "},
             {:isbn=>"ISBN-10: 1590597664"}]},
    {:book=>[{:title=>"Programming Ruby: The Pragmatic Programmers' Guide, Second Edition [ILLUSTRATED]  (Paperback)"},
             {:price=>"$29.67"},
             {:saving=>"$15.28\n      (34%)\n    "},
             {:isbn=>"ISBN-10: 0974514055"}]},
    ...]

This time the “The Ruby Programming Language (Paperback)” book isn’t included in the results at all. For our purposes though, I’ve decided that I only want to know about books that I have all the details for. Instead of setting :required on every result, I can specify it on the book_detail definition which will give the same output:

@extractor = Scrubyt::Extractor.new do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a", :required => :all do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']"
    saving "//td[@class='price']"
    isbn "//li[text()*='ISBN-10:']"
  end
end

Sanitizing scRUBYt! output

The output still isn’t quite what we want. We’ve dumped the empty results, but we’ve still got that ugly “ISBN-10: ” in front of the ISBN. We could clean it up later, but it’s creating additional work for ourselves. And if we want this thing to scale (I’ve got scrapers which scrape thousands of pages off a single site) trying to keep all that data hanging around in memory isn’t going to work. So let’s do as much of possible within the scRUBYt! definition as we’re collecting the data:

@extractor = Scrubyt::Extractor.new do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a", :required => :all do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']"
    saving "//td[@class='price']"
    isbn "//li[text()*='ISBN-10:']", :script => Proc.new{|isbn| isbn.gsub("ISBN-10: ", "")}
  end
end
puts @extractor.results.inspect
=> [{:book=>[{:title=>"The Ruby Programming Language [ILLUSTRATED]  (Paperback)"},
             {:price=>"$26.39"},
             {:saving=>"$13.60\n      (34%)\n    "},
             {:isbn=>"0596516177"}]},
    ... ]

And now you’ll see that we’re getting a much cleaner ISBN result. Just create a Proc, the result will be passed into in and then do as you see fit. An if statement to check it contains something you expect, a regexp, the possibilities are endless. You can also combine this with the other options like :require and :remove_blank. So let’s really jazz this thing up. We’ll clean up the saving, and pull in the description and that’s all the data we need:

@extractor = Scrubyt::Extractor.new do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a", :required => :all  do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']"
    saving "//td[@class='price']", :script => Proc.new{|saving| saving.match(/(\$[\d\.]*)/)[1]}
    isbn "//li[text()*='ISBN-10:']", :script => Proc.new{|isbn| isbn.gsub("ISBN-10: ","")}
    description "//div[@id='productDescription']//div[@class='content']"
  end
end
puts @extractor.results.inspect
=> [{:book=>[{:title=>"Beginning Ruby: From Novice to Professional (Beginning from Novice to Professional) (Paperback)"},
             {:price=>"$26.39"},
             {:saving=>"$13.60"},
             {:isbn=>"1590597664"}, 
             {:description=>"Product Description\n  Ruby is perhaps best known as the engine powering the..."}]}, 
    {:book=>[{:title=>"Programming Ruby: The Pragmatic Programmers' Guide, Second Edition [ILLUSTRATED]  (Paperback)"}, 
             {:price=>"$29.67"}, 
             {:saving=>"$15.28"}, 
             {:isbn=>"0974514055"}, 
             {:description=>"Product Description\n  Ruby is an increasingly popular, fully object-oriented dynamic..."}]},
    ...]

Outputting results to XML

Inevitably there comes a time where you want to consume this data in something other than your ruby application. At that point, passing around a Hash is probably not the best idea. Alternatively, you might have a scraper that has to scrape hundreds to thousands of pages. Storing all the results in a Hash as you go will bring your machine to it’s knees. So here comes one of the largest changes to the way the new release of scRUBYt! works.

Previously, you always had results returned as a Hash and/or XML depending on your need. Everything was held in memory until you destroyed your extractor. Now, the standard XML option is to stream the results out to a file as they are processed and remove them from memory. There is no way to retrieve the results as XML within your program, they have to be streamed out to a file (and really, why would you want XML within your app when you can have native ruby structures instead?). So to save our scraper above out to an XML file you just pass a new output format and an instance of a File to the extractor:

@file = File.new("amazon_results.xml", "w")
@extractor = Scrubyt::Extractor.new :output => :xml_file, :file => @file do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a", :required => :all  do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']"
    saving "//td[@class='price']", :script => Proc.new{|saving| saving.match(/(\$[\d\.]*)/)[1]}
    isbn "//li[text()*='ISBN-10:']", :script => Proc.new{|isbn| isbn.gsub("ISBN-10: ","")}
    description "//div[@id='productDescription']//div[@class='content']"
  end
end

Now a call to @extractor.results at the end of the scrape will return no results. I hope that wasn’t too much and it’s given you a good view into how to create your very own web scraper. If you have any questions, head on over to the scRUBYt! forums or post them in the comments.

More web scrapers with the upcoming scRUBYt!

Tue, 13 Jan 2009 01:20:13 GMT

In the previous articles I gave a brief glimpse at the upcoming scRUBYt! release. We learned how to do some basic html scraping, and then followed it up by scraping multiple pages. As promised, in this installment I’ll go over how to get more detailed logging output to help you debug your scraper while in development and how to submit and navigate forms.

Logging Scraper Output

There’s been quite a fundamental change to the way scRUBYt! works internally of late. It doesn’t manifest itself visually in the way you interface to it, but it does mean logging the output is now much easier and cleaner. Using our most basic example from the first tutorial, you just need to pass the :log_level into the extractor:

@extractor = Scrubyt::Extractor.new(:log_level => :verbose) do
  fetch "http://www.google.com/ncr"
end

By default, the output is directed to stdout so you’d see the following on your screen:

start
fetch: http://www.google.com/ncr
end

If you’re after a more complex example, here is the scraper definition from the second part of the tutorial series and the corresponding log output:

@extractor = Scrubyt::Extractor.new(:log_level => :verbose) do
  fetch "http://www.google.com/ncr"
  fill_textfield "q", "ruby"
  submit
  page_detail "//h3[@class='r']/a" do
    title "//title"
    summary "//p", :script => Proc.new{|result| result if result.match(%r{(\w+\W+){25}})}
  end
  next_page "//a[text()*='Next']", :limit => 2
end

start
fetch: http://www.google.com/ncr
textfield: 'q' = 'ruby'
with options ''
submit
next detail: 'page' = 'http://ruby-lang.org/'
with args: ''
next detail: 'page' = 'http://en.wikipedia.org/wiki/Ruby_(programming_language)'
with args: ''

etc...

next page: /search?hl=en&ie=UTF-8&q=ruby&start=10&sa=N
fetch: http://www.google.com/search?hl=en&ie=UTF-8&q=ruby&start=10&sa=N
next detail: 'page' = 'http://www.rubycentral.com/book/'
with args: ''

etc...

end

At present the valid :log_level values are :none, :critical, :error, :warn, :info, :debug, and :verbose (in increasing order of noise). If you want to direct the log output to something other than stdout the only way at the moment is to override the Scrubyt::Logger#log method. I’m looking at ways to make it easier to substitute in a file based or other logging approach.

Making Your Scraper Navigate Forms

So now that we know how to log the output, let’s do something more useful with our scraper. We can’t really take our Google example from previous posts any further given how simple their interface is, so let us move over to Amazon. Say I wanted to grab a list the books on ruby that are for sale. Sure, I could probably get this information via an Amazon API… but that’s not really the point now is it ;)

@extractor = Scrubyt::Extractor.new(:log => :debug) do
  fetch "http://www.amazon.com/"
  select_option "url", "Books"
  fill_textfield "field-keywords", "ruby"
  submit
  book_detail "//td[@class='dataColumn']/table/tr/td/a" do
    title "//h1[@class='parseasinTitle']"
    price "//b[@class='priceLarge']"
    saving "//td[@class='price']"
    isbn "//li[text()*='ISBN-10:']"
  end
end

So let’s run through what we’ve got here, hopefully some of it looks familiar from the previous examples we’ve gone through. First we fetch the page to start with, then in the select field named “url” we choose the options that says “Books”, and we then submit the form. scRUBYt! will keep track of the last form you input any data to, so if there are multiple forms on the page then you just need to target the appropriate input fields. From there, the submit action will work out what it needs to do.

Next is to define a detail block, so we point out the XPath to the heading/link for each book on the page and say we want to navigate to that page and extract the title, price, saving, etc. I’ve been a little cheeky with the isbn definition, saying just find me any LI tag that contains the string “ISBN-10:”. If we were to look at the results generated you’d see:

puts @extractor.results.inspect
[{:book=>[{:title=>"The Ruby Programming Language [ILLUSTRATED]  (Paperback)"}, {:price=>"$26.39"}, 
          {:saving=>"$13.60\n      (34%)\n    "}, 
          {:isbn=>"ISBN-10: 0596516177"}]}, 
 {:book=>[{:title=>"The Ruby Programming Language (Paperback)"}, 
          {:price=>nil}, 
          {:saving=>nil}, 
          {:isbn=>"ISBN-10: 020171096X"}]}, 
 {:book=>[{:title=>"Beginning Ruby: From Novice to Professional (Beginning from Novice to Professional) (Paperback)"}, 
          {:price=>"$26.39"}, 
          {:saving=>"$13.60\n      (34%)\n    "}, 
          {:isbn=>"ISBN-10: 1590597664"}]}, 
 {:book=>[{:title=>"Beginning Ruby: From Novice to Professional (Kindle Edition)"}, 
          {:price=>nil}, 
          {:saving=>nil}, 
          {:isbn=>nil}]}, 
 {:book=>[{:title=>"Programming Ruby: The Pragmatic Programmers' Guide, Second Edition [ILLUSTRATED]  (Paperback)"}, 
          {:price=>"$29.67"}, 
          {:saving=>"$15.28\n      (34%)\n    "}, 
          {:isbn=>"ISBN-10: 0974514055"}]}, 
 {:book=>[{:title=>"The Ruby Way, Second Edition: Solutions and Techniques in Ruby Programming (2nd Edition) (Addison-Wesley Professional Ruby Series) (Paperback)"}, 
          {:price=>"$29.69"}, 
          {:saving=>"$15.30\n      (34%)\n    "}, 
          {:isbn=>"ISBN-10: 0672328844"}]}, 
 {:book=>[{:title=>"Ruby Way, The: Solutions and Techniques in Ruby Programming (Kindle Edition)"}, 
          {:price=>nil}, 
          {:saving=>nil}, 
          {:isbn=>nil}]}, 
 {:book=>[{:title=>"Learning Ruby [ILLUSTRATED]  (Paperback)"}, 
          {:price=>"$23.09"}, 
          {:saving=>"$11.90\n      (34%)\n    "}, 
          {:isbn=>"ISBN-10: 0596529864"}]}
]

And we have a reasonable snapshot of the data. You’ll see there though, that it’s not perfect. Firstly, we are missing information for some results. We’ve got spaces and carriage returns in the saving data, and we probably don’t need the “ISBN-10:” string at the front of the ISBN result. And what if we wanted to link to the actual result so someone could actually buy the book on Amazon?

All good questions, and all easily solvable. I’ll follow it up with a post in the next day or so and highlight some of the new ways of specifying constraints on your data in scRUBYt!. And special thanks have to go to Homeflow who have been funding at least a day of time for Peter and I each week lately, hence the increased level of development in scRUBYt!. It’s nice when you have clients who want to actively give back.

Rails Hosting - Updated

Mon, 05 Jan 2009 23:10:53 GMT

This is just a quick note so that those subscribing to my feed or checking the site for updates are aware. I’ve made a series of minor changes to the rails hosting post from last month based on the feedback provided by you guys. If you’ve got any additional experiences with different rails hosts that you’d like to share, then please send them through.

Ruby on Rails Tutorials

Mon, 05 Jan 2009 02:34:09 GMT

There has been a few occasions over the previous months when various people I meet have asked me to point them in the right direction to get started with ruby on rails. I remember when I started, it was a pretty easy question to answer… everyone just got pointed in the general direction of why_’s poignant guide, the pickaxe book, and agile web development with rails. But things have come a long way since then. There is not only a veritable cornucopia of options now out there, but for somebody who isn’t already engrossed in the community it’s almost impossible to tell which ones are still relevant and useful and which ones are so dated that they’ll make your attempts to learning markedly more difficult.

So with that in mind, I’ve attempted to compile a list of resources which are still (at least for the most part) still relevant at the time of posting. Some will have some minor inconsistencies either in implementation or current convention but I’ve made a genuine effort to review and ensure they are not way out of line. If you find in errors on that front, please let me know in the comments.

So without any further ado, here are the ruby on rails tutorials I’ve found broken into various categories (a couple are listed in more than one):

Ruby on Rails Tutorials

Getting Started and Example Applications
Models/Migrations
Controllers
Views
Routing
Mail
Testing
Performance, Administration, and Debugging
Miscellaneous
Extra Special Mention

Getting Started and Example Applications

Getting started with Rails A great guide on building your first rails app and understanding the framework.

Ruby On Rails Tutorial : The Basics A two-part tutorial to getting up and running on rails by Huw Collingbourne.

Building a blog About.com have have this guide for Rails 2.1 which runs you through the classic “15-minute blog” project.

Starting a rails project using Komodo For those of you using Komodo this is a great guide to show you how to develop within Komodo and run the tests within your IDE.

Building a bookmark manager A quick introductory guide to rails and an example app to manage your bookmarks.

Building a contact manager A ruby on rails tutorial on how to get acquainted with the framework by building a contact list manager.

A simple product catalog A tutorial from Oracle on how to create a comics catalog on top of an Oracle back-end.

The official 15-min blog screencast A rather high paced run-through of how to get a blog up and running in 15mins.

The ultimate guide to ruby & rails Quite a detailed and useful tutorial from Patrick Lenz which not only goes through rails but is a pretty decent intro to the ruby you should get your head around first too.

What I wish I knew A bit of a retrospective from someone after they got their first few rails applications under their belt.

Sending Instant Messages in Rails How to use XMPP and BackgroundRB to send messages to GMail and other Jabber based clients.

Programming Conventions A discussion on what exactly DRY means, why you should stick to conventions, and when you should look at optimization.

Models/Migrations

Bulk Insert into ActiveRecord with Rails How to quickly insert multiple records at once with ActiveRecord.

Using Oracle A tutorial from Oracle on how to create a comics catalog on top of an Oracle back-end.

Rails email validation Check that a provided email address really is legit, by doing a DNS/MX lookup on the domain.

Rail Migrations for beginners This tutorial is a little dated, but thankfully the examples all still work in Rails 2.x. What I particularly like, is that they actually cover how to add an index to a column which most don’t cover and thus many don’t bother to do.

Rails Guide to Migrations A very detailed guide on what you can and can’t do in migrations. Even I learnt a few new tricks from reading through it.

ActiveRecord Validations and Callbacks All the various validation options you have, and what method callbacks you can hook into when modifying an instance of a model.

ActiveRecord Associations How to link up the associations between the models in your applications.

Finding records in your database A tutorial on all the various ways you can find either a single record, or a collection of records, using ActiveRecord. The built in finders, and how to pass your own custom conditions in are all handled.

Controllers

Simplify Ruby on Rails controllers with make_resourceful Take advantage of the DRY approach by quickly implementing common controller actions in just a few lines of code.

Controlling Information Access A quick explanation of the why/when you should make methods in your controllers private or protected.

Displaying Useful Flash Messages How to provide more useful feedback on errors or actions to your users within your application.

Consistent and sexy rails forms How to both DRY up your code and ensure your users get a consistent experience by using a Rails Custom Form Builder.

ActionController Basics How to use filters, disabling the session, cookies, authentication, and protecting against forgery. It’s called basics, but it covers quite a lot.

Catching rails error messages An easy example of how to use the 2.x approach to catching errors in your rails applications.

Views

Introduction to form helpers A quick tutorial on when you should use the object/model based form helpers in rails, and when you should use the standard ones.

Rendering layouts and views How to render a response to the web browser within your rails app. Also goes into providing only a HEAD response, nesting layouts, using partials, redirects, and using yield.

Implementing unobtrusive javascript in rails How to stop using the built-in javascript helpers and instead use lowpro to keep your views and code clean.

Routing

Nested Resources in Rails 2.x A guide to bring you up to speed on the changes between the way resources work in 1.x and the 2.x releases.

Rails Routing from the Outside-In A very comprehensive guide on how to configure and use the routing options in rails.

Mail

Stop Rails from sending email A useful way to check that mail is getting sent, without actually sending it.

Testing

Test Rails XML Output with rspec How to easily test XML output using rspec.

Testing webservices using rspec A guide on testing the old style SOAP/XML webservices with rspec.

Using rspec have_tag How to test anything you need to in a view with rspec have_tag

Performance, Administration, and Debugging

Improve Rails Performance with Thin A tutorial on replacing mongrel or webrick with the event driven thin web server.

Let nginx automatically become aware of your Rails apps How to configure nginx to automatically become aware of any new apps you deploy and start serving them without any additional configuration.

Monitoring your Rails app with god How to use the god monitoring tool to manage, start, stop, and restart your rails app.

Monitoring BackgroundRB with god Another tutorial on using god, this one for monitoring BackgroundRB.

Improving Bulk Data Insertion with ActiveRecord Reduce the number of round-trips to the database by batch inserting records into ActiveRecord.

Background Processing with Starling How to off-load long running tasks to a queue so that you don’t tie up your webserver processes requests needlessly.

Ruby on Rails Caching Tutorial Another two-part tutorial. This one, by Gregg from RailsEnvy, runs through the various ruby on rails caching options and how to implement them to improve the performance of your site.

Ruby on Rails Rake Tutorial Gregg from RailsEnvy again running through how to make the most of rake. If you’re not already writing your own custom tasks in rake you should definitely take a look because there is probably a handful of things your could (and should) be doing in it.

Debug your app with ruby-debug I don’t know how I ever diagnosed problems in development before I learnt how to use ruby-debug. If you’ve got applications even just staring to come together, you really need to get your head around this tutorial.

19 Rails Tricks Most Rails Coders Don’t Know Not a tutorial per se, but a good collection of handy tips on how to benchmark performance, test various aspects of your app, improve it’s performance, and more.

Reading a stack trace You’re inevitably going to see a few when your developing, you better learn how to make sense of them.

Ruby on Rails Security Guide Storing and protecting the session, user management, projecting from injection, and more.

Debugging Rails Applications How to use views to help you debug your app, detecting memory leaks, and using the logger.

Benchmarking and Profiling Rails The why and how of benchmarking and profiling. Still a work in progress.

Miscellaneous

Define your own base rails framework with templates Stop manually importing and installing all the same plugins in all of your applications, define a template in just a few lines and let rails know what it should start with.

Integrate an IM Bot into your Rails App How to make an Instant Messaging BOT and use it in your rails application.

Creating Plugins How to create a plugin for use in your rails app.

Working with Rails Timezones How to use the 2.x enhancements to easily switch between timezones or display dates/times in a users local zone.

Super extra special mention

Ryan Bates is simply tireless in providing useful screencasts showing various bits of functionality and tips within rails. If I was to list each of the ones I thought would be useful here, then the list would probably be 3 times longer than it already is. Instead, make sure you check out Railscasts and scan through the archive.

Web Spider Creation with scRUBYt! - Part II

Wed, 17 Dec 2008 10:00:35 GMT

Continuing on from the previous post, Web Spidering and Data Extraction with scRUBYt!, this article will help you delve a little deeper with the scRUBYt! scraping framework both in terms of your understanding of how to use it… and actually delving deeper in your crawl to more pages.

A quick recap on the last web spider

So as far as we got last week was to go off to Google, put in a search for the word “ruby”, and then list the link text for each of the results. Here’s the code we ended up with to get that far:

@extractor = Skimr::Extractor.new(:agent => :standard) do
  fetch "http://www.google.com/ncr"
  fill_textfield "q", "ruby"
  submit
  page_title "//h3[@class='r']"
end

How to scrape deeper pages

But this is a fairly contrived example, and it’s not exactly going to save you a huge amount of time over a quick manual copy-and-paste job from the results page. But what if you didn’t want to just have a list of the links, you wanted some kind of summary or additional detail on each of the links? Lets actually go to each website, and see what kind of content they’ve got and grab something useful from it:

@extractor = Skimr::Extractor.new(:agent => :standard) do
  fetch "http://www.google.com/ncr"
  fill_textfield "q", "ruby"
  submit
  page_detail "//h3[@class='r']/a" do
    page_body "//body"
  end
end

If you offer up the XPath to a link element with a result name ending in _detail, and then pass in a block, scRUBYt! will follow the link before trying to process the block. That means we can use this technique for following each of the results Google gives us, and at each we return all of the text contain on the page between the <body> tags.

Now you could take the hash returned, and do some post processing on it to get something meaningful from the text extracted. But for the sake of example, I’ll make some big assumptions and assume that every page is going to have at least a <title> tag, and a <p> tag with more than just a few words in it.

@extractor = Skimr::Extractor.new(:agent => :standard) do
  fetch "http://www.google.com/ncr"
  fill_textfield "q", "ruby"
  submit
  page_detail "//h3[@class='r']/a" do
    title "//title"
    summary "//p", :script => Proc.new{|result| result if result.match(%r{(\w+\W+){25}})}
  end
end

As you may have noticed, you can pass a proc in as a parameter to your result definition. The output of the XPath match will be passed in to the proc, and the result ultimately returned as the final result for that definition? Make sense? If not, what I’ve done above is look for all <p> tags on the page and pass them in to my proc definition. The proc then runs a regexp against it to check that at least 25 words exist within the <p>, if there is then all the <p> content is returned otherwise nil is returned.

What if the results are paginated?

You could potentially create a highly recursive extractor to handle this, but it’s such a common case that we’ve included a method to do it for you:

@extractor = Skimr::Extractor.new(:agent => :standard) do
  fetch "http://www.google.com/ncr"
  fill_textfield "q", "ruby"
  submit
  page_detail "//h3[@class='r']/a" do
    title "//title"
    summary "//p", :script => Proc.new{|result| result if result.match(%r{(\w+\W+){25}})}
  end
  next_page "//a[text()*='Next']", :limit => 2
end

I’ve used the XPath text() function here to highlight its usefulness. I use it quite a lot as a shortcut to get things working and test, and it’s been a lifesaver in many scenarios where the markup is inconsistent or you want to keep the scraper definition generic. And thankfully, it works for this scenario. However, be wary of using it as is in production as it may have some unexpected side effects. If one of the results that came back had the word “Next” in the title, then scRUBYt! would diligently follow that link and you’d end up on the wrong page.

What’s next?

In the next installment I’ll briefly cover how to handle logging of the scrape to help you diagnose any problems, and how to handle more complex form completion.

Web Spidering and Data Extraction with scRUBYt!

Tue, 09 Dec 2008 10:14:45 GMT

Some of you may be aware that I work on (time permitting) the scRUBYt! project with Peter Szinek. Hopefully some of you have actually found an excuse to use the tool, I know there are quite a few hundred other satisfied users out there. Well Peter has been furiously working away on polishing up the lastest release, we’ve also gone back and refactored a lot of the internals and improved the test coverage of library. Given the gnarly levels of recursion in it, at times it was proving difficult to add in the new features we wanted.

The skimr branch is our first attempt at refactoring, but to do it we’ve sacrificed quite a lot. A lot of functionality is currently missing, and the syntax has changed slightly. I wouldn’t yet consider this to be a release candidate, but it has been getting used successfully for a few months now in production so I think it’s worth a look.

Whats new?

Well apart from being a lot less code, it’s significantly faster and requires much less RAM on larger web scrapes. This is in part due to the fact that you can now stream your results out to a file, rather than trying to hold your entire dataset in memory before dumping them. If you don’t stream your results out to a file, the default is to return them as a Hash making it much easier to develop custom output handlers, or integrate the results into your existing ruby code.

Creating your own web crawler

To begin with, we will start with the tried and true google/ruby example that has served so well with previous scRUBYt! releases. So we start by defining a new Skimr extractor:

@extractor = Skimr::Extractor.new(:agent => :standard) do
end

You’ll notice here we pass in an agent type. It’s an optional parameter, and if you leave it out it will default to :standard which means a combination of mechanize/hpricot to parse your results. Others will become available in future releases to allow you to scrape AJAX heavy sites again. Next, to tell it what page we want to start at:

@extractor = Skimr::Extractor.new(:agent => :standard) do
  fetch "http://www.google.com/ncr"
end

Now we run over to Google (the ncr bit tells Google not to redirect me to a country specific site), because we’re about to start a search:

@extractor = Skimr::Extractor.new(:agent => :standard) do
  fetch "http://www.google.com/ncr"
  fill_textfield "q", "ruby"
  submit
end

Hopefully the above is fairly obvious, we’ve entered the term “ruby” into the field named “q”, and hit submit. Play along in your browser so you can see what we are playing with.

Extracting data from the website

Okay, so we’ve got the navigation part covered. Now we want to pull out a list of all the results, it’s quite simple:

@extractor = Skimr::Extractor.new(:agent => :standard) do
  fetch "http://www.google.com/ncr"
  fill_textfield "q", "ruby"
  submit
  page_title "//h3[@class='r']"
end

Just provide an XPath to the element on the page that you want, and scRUBYt! will extract all elements that fit that definition. As the results are now available as a Hash object we could simply do the following:

>> @extractor.results.first
=> {:page_title=>"Ruby Programming Language"}
>> @extractor.results.last
=> {:page_title=>"Welcome! [Ruby-Doc.org: Documenting the Ruby  Language]"}

I’ll follow this up in a couple of days with some examples of scraping deeper pages, merging multiple result sets, some of the new features we’ve included, and talk about some of the future enhancements that are coming soon.

Custom Starting Templates for Rails Apps

Mon, 08 Dec 2008 12:33:23 GMT

Okay, so I’ve been a little slow on the uptake on this one. But for anybody else that missed it, you can now define your own standard setup for a rails app. (And kudos to Peter Szinek, who I work with on the leading ruby web spider/data scraping library, for the heads up on this last week)

Reducing the time taken to start coding

That’s essentially what this change does. Most of us probably have out preferred setup, especially if we’ve deployed a few sites. Typically I create the rails directory structure, then pull in HAML, rspec, rspec on rails, my form helper, restful auth, etc. It’s not terribly difficult, but it is laborious. Well someone else I once worked with, Jeremy McAnally (woo!, look at me name drop ;), created a great little tool called rg which would allow you to define your own templates for when you create a new application. Now just like engines has been pulled into the latest edge core to give rails the same ability as merb-slices, so too has rg now been pulled in to match the functionality offered by creating your own custom merb-stack.

How to create a custom rails template

So if like me you have your usual grab bag you want to setup when you start a new app, you can define your own template.rb file like this:

run "rm public/index.html"
git :init

plugin "rspec", :git => "git://github.com/dchelimsky/rspec.git", :submodule => true
plugin "rspec-rails", :git => "git://github.com/dchelimsky/rspec-rails.git", :submodule => true
plugin "exception-notifier", :git => "git://github.com/rails/exception_notification.git", :submodule => true
plugin "restful-authentication", :git => "git://github.com/technoweenie/restful-authentication.git", :submodule => true
plugin "semantic-form-builder", :git => "git://github.com/rubypond/semantic_form_builder.git", :submodule => true
plugin "paperclipped", :git => "git://github.com/kbingman/paperclipped.git", :submodule => tru

gem "will-paginate", :git => "git://github.com/mislav/will_paginage.git"
gem "haml", :git => "git://github.com/nex3/haml.git"
rake "gems:install", :sudo => true
run "haml --rails ."

route "map.login '/login', :controller => 'sessions', :action => 'new'"

rakefile("cruise_controle.rake") do
  <<-TASK
    desc "Run all the tests, including API and acceptance tests"
    task :cruise do
      Rake::Task['db:migrate'].invoke
      Rake::Task['spec'].invoke
      Rake::Task['spec:stories'].invoke
      Rake::Task['metrics:all'].invoke
      Rake::Task['flogger:record'].invoke
    end
  TASK
end

initializer 'form_builder.rb', <<-CODE
  ActionView::Base.default_form_builder = SemanticFormBuilder
CODE

generate("authenticated", "user session")
generate("rspec")

rake("db:migrate")

git :add => "."
git :commit => "-a -m 'Initial commit'"

There is quite a lot going on there, I’ll quickly run through what I’ve done. Firstly, I’ve deleted index.html as I never use it. Next we initialise a local git repository and pull in all the plug-ins I want (as git submodules). Next we have a couple of gems I need, so we define them and install (using sudo) if they’re not already on the system. I setup a route to the login page, create a rake task to run cruise, an initializer to change the default form builder to be my own. And then finally I create the user/session models for restful-authentication, migrate the database, and commit all the changes.

Running your custom template

Now to create a new rails app using this template do the following:

rails my_new_app -m ~/template.rb

Or alternatively if you work across various machines you can post the template up to a server (git, your own box, etc) and specify a full HTTP to the remote file:

rails my_new_app -m http://gist.github.com/33443.txt

Ruby on Rails Hosting Round-up

Thu, 04 Dec 2008 23:18:27 GMT

UPDATE: I’ve extended this post based on the feedback in some of the comments. So I haven’t used all of the rails hosts listed here, but I’ll make it clear which those are.

Working for various clients and along side various colleagues over the years, I’ve been witness to a whole number of hosting providers in the ruby/rails space. Some are great, others acceptable, and a few were just down right terrible. So how do you go about choosing which host is the right one for you? Read on…

One of the early things I came to appreciate when working with rails was how it made me get more involved with the rest of the eco-system on the server. Previously I’d just copy an ASP/.Net app across a share, maybe restart IIS. Or I’d upload some PHP and maybe setup a virtual host entry for Apache and be done with it. Now it’s not uncommon to have to create a deploy user, install some gems, maybe download a few dependencies (imagemagick?) and compile them on the server. Not to mention setting up monit or god to monitor various processes. With all that in mind, I’ve given up on paying the extra money to get a “managed service”. More often than not it all goes out the window because I’m using a package I compiled myself rather than the dated version in the package management system, or they just don’t have enough of a clue on the ruby/rails front to be useful. So to make this a somewhat level playing field, keep in mind that I’m working under that basis that anything software related you need to be able to resolve yourself with these providers (if they give you access). We will leave hardware up to them.

Unmanaged Rails Hosting

So without further ado, here’s my list of personal experiences if you’re willing to go the unmanaged route.

Serverbeach

This is the unmanaged brand of Peer1, and they’re who I currently host all of my own production and development sites with. I really can’t endorse them highly enough. You can have a low-end dedicated server for development purposes from as little as $75/month, and they’ll often run specials so watch their page. I grabbed a quad-core dual xeon with 2GB of RAM, and a 2000GB transfer allowance for $179/month (they currently have RAID capable dual-core Xeons for $139). A choice of server locations if that is important to you, as well as hardware firewall, backup, and all the various offerings you come to expect. They also send a survey around each year and customers get to vote on what new offerings they should provide. Cloud computing was on the list this year, lets see if it turns out to be as competitive as EC2 and the like.

Rimuhosting

Another excellent provider. I’ve used these guys as my main provider for quite a while until our needs got a little too much for a fully worked VPS, and their dedicated plans just didn’t quite match the value Serverbeach offered. That being said, unless you’ve got particularly high transfer or memory requirements I doubt you’ll have any problems at all with what they offer. I still have one of our sites running over there, and a number of client sites, and I have no intention of moving.

RailsMachine

I’ve worked on these quite a bit for various clients, and I feel a little ho hum. There’s not as though there is anything exactly wrong with the product they offer, I just think for the price point ($250/month for a dedicated CPU) it’s a little on the expensive side for what you get. It’s almost a cross between an unmanaged and fully managed setup. You pay the price of the latter but set a lot of it up yourself by running a series of scripts and installing local gems (as detailed on their wiki).

A2Hosting

Now I need to put a disclaimer on this one at the preface; I haven’t used these guys in almost three years, and hopefully things have changed since then. However, at the time their rails support was dreadful. I only ever ran one site there which was a rather simple blog, and uploading images and resizing with rmagick (ala attachment_fu/paperclip) resulted in me bringing down what was obviously a seriously overloaded server. And from what I was informed it wasn’t just my account which was affected, it would halt any VPS running on the same physical machine. After being unable to place any kind of CPU quota on my account to stop other from being affected, or having a box more suitable for running rails as advertised I had to take things elsewhere. Suffice to say, I’ve not been back.

Slicehost

I’ve not used them personally, but have worked closely with quite a few people who host their sites on them. For the money ($20/month), they’re hard to beat. And if all you need is a VPS to run your blog, I’d think you’d be hard pressed beating them for value.

Linode

Another one I’ve not used personally, but it got a few mentions within the comments of the article when I first posted it so I thought it was worthy of pointing out. They have their own management panel and run your setup using Xen. Prices start from US$19.95/month and include 200GB data transfer.

Managed Services

At the other end of the spectrum is the fully managed services. Take all the worry out of actually building your box and only concern yourself with deployment. If that’s the kind of dream you think you’re willing to pay for, then your options are:

Engine Yard

The big daddy in the ruby/rails hosting world, if you’ve not heard of them you must obviously have been living under a rock. At $399+/month you’re comparatively paying through the nose when measured against the unmangaged offers, but when the the employee lists consists of the likes of Ezra Zygumnowitcz, Yehuda Katz, Evan Phoenix, etc. then I don’t think you’re going to find a better crack squad of ruby experts to manage things for you.

Rackspace

They’ve been in the game for years, long before rails was on the scene, and they are seriously excellent for most hosting needs. I’ve only used them a few times, and only have good things to say. However, feedback in the comments seems to indicate that for rails based apps their support is lacking. They don’t advertise their price on their site, but expect it to be somewhere above $400/month. At that price, it’s hard for them to be a serious consideration against EngineYard.

Peer1

ServerBeach back again with their managed offering. I can’t really say much more other than their hardware is exceptional and their support when I’ve needed it likewise. You’ll need to contact them for details on price.

RailsMachine

Another one back again with a managed offering, this time for $349/month. On top of your hosting monthly hosting fees it puts them above most the others in terms of cost.

Conclusion

There are a heap to choose from, and far too many for me to be able to give an experienced and qualified view on. Personally though, I’m quite happy with my unmanaged dedicated servers and having a separate management contract for the ones that I need to maintain five 9 uptime with. It gives me the business continuity I want and the comfort of having a much closer personal relationship with a handful of people who actually know and understand my business, at a price that is significantly lower than most of the packaged deals.

I’d love to hear from others out there that have had experiences with other hosting providers though so please leave your thoughts, good or bad, in the comments section.

Oh, and if you’re after someone to help manage your server I use these guys Atalanta Systems and Bitfield Consulting

When (ruby) floating just isn't good enough

Fri, 28 Nov 2008 15:05:23 GMT

Hopefully we all know that when talking about numbers “float” means “floating point” and as such the number isn’t entirely precise, it’s just an approximation. Most of the time it’s a good enough approximation to serve our needs, but when it comes to dealing with money it’s best to play it safe and have the absolute precision we want. Otherwise all those .000001 of a cents can add up to a large sum.

Working with fixed point arithmetic

To get around the short-comings of using floating points, a fairly common and trivial task is simply multiply it to the number of significant digits we care about and then treat it as an integer. So for dealing with money, we would multiply the value $120.19 by 100 to end up with 12019 cents. Any arithmetic we do, we do on the expanded cent value and we store that in the database. If we want to display a nicely format dollar value to the end user, we do that after we pull it back out from the database.

Losing money with Ruby

As I said, all this is rather trivial and many of you have probably had to do it in the past. So here is an excercise for the reader to follow along with at home. What do you think the following would output?

"0.29".to_f

If you answered 0.29 pat yourself on the back. Let’s take it a step further:

"0.29".to_f * 100

Who answered 29.0? Congratulations. Okay, and now to take home the full showcase and all the cash:

("0.29".to_f * 100).to_i

Do we hear a 29 out there? Hooray! You’re not alone, but you’re wrong. Fire up your nearest ruby console and give it a shot, you’ll actually get _28_ as the output. If you’ve made the same assumption yourself in the past, it’d be worth going back and checking your code to make sure it works. Likewise if you’re using a 3rd party library it’s worth double-checking they’ve done the correct implementation.

Doing ruby floating points and money properly

So how do you get around this problem? Well from reading the ruby docs on float I’d still expect .toi_ to work, but clearly it’s not actually truncating the value as documented. Instead, you have to call round():

("0.29".to_f * 100).round

Alternatively, you can use the money gem as it does it properly.

Easily Generating Content For Your Site

Mon, 24 Nov 2008 08:46:00 GMT

Apologies for the lack of action here late, I’ve got a long list of stuff I’ve meant to be writing about but just not quite finding the time to fit it in around client work. I think I’m going to just have to start allocating a time each fortnight into my schedule, but don’t hold me to that. But while we are talking about content ideas, how about some help generating some for your own site?

Part of the problem many people have, be they bloggers or just running their own small online business, is generating content ideas to help drive traffic. A quick and easy solution comes courtesy of the guys I’m contracting to a few days a week at the moment, Wordtracker. The new Keyword Questions tool lets you enter in a seed word, and fires back the questions that people have asked on that topic from their database. Don’t get too caught up in the numbers here, as it’s only a (hopefully representative) sample of the total number of internet users. The most useful part of it is the actual questions that are being asked.

Try doing a search for business on there. If you were running a site trying to target small businesses, it’d be worthwhile considering putting up a series of posts dealing with creating a business plan, how to get money to start a business, and how to create an online business, etc. The same can be done with whatever niche/sector you are operating. And better yet, it gives you the exact keywords people are using to help with your SEO efforts.

This tool came out on the back of just one of the content creation ideas in Wordtracker’s 50 Kick-Ass Keyword Strategies. If you’re looking for new simple ways to improve your visibility search engines it’s well worth the investment. Given my experience working with them I’m hoping I can feed some of the great SEO knowledge into various blog posts over time too.

Merbcamp Videos

Fri, 17 Oct 2008 12:36:01 GMT

Update: They are all now available on the merbcamp site.

Unfortunately I only managed to capture the stream from day 2. But here are all the videos. I’ll be posting them in downloadable formats later this evening. If you want to review my notes I took while watching the presentations, check out notes from merbcamp day 1 and notes from merbcamp day 2

Merbcamp - Day 2

Sun, 12 Oct 2008 17:29:17 GMT

In case you missed the first installment, click to read day one at merbcamp. This is what happened on day 2:

Deploying a Merb app

“Your number one priority when building a merb app should be deployment”. It’s going to make scaling much easier later. As a result, there are some decisions you need to consider very early on in the process.

Freezing

You’ve got two ways to freeze merb and dependencies into your application folder:

thor merb:stable
thor merb:edge

But should you freeze or not? And if so, which option should you take? The 3 options you’ve got are:

Frozen gems

Stable release
Everything is contained within your app (in merb_root/gems)
Start app with merb_root/bin/merb
Maximum deployability

Frozen git

Edge release
Everything is contained within your app (in merb_root/src)
Start app with merb_root/bin/merb
Maximum deployability

“Thawed”

Stable or Edge
Everything as gems
Start app with wherever merb has been installed
Least deployability, least recommended

Managing your app dependencies with thor

Once you’ve frozen everything within your app directory, you have two ways to upgrade your dependencies. Updating all of your frozen gems within your application from rubyforge or wherever:

thor merb:gems:refresh

Or rebuilding all of your native extensions:

thor merb:gems:redploy

Web servers

The main options on the front end are apache or nginx, pushing requests through to mongrel, thin, or ebb. Nothing really that different to rails here.

The other option is passenger, again nothing different to rails here.

Starting and stopping merb

You’ve got 2 options here:

merb -k
merb -K

the latter starts merb gracefully, the latter forcefully. Graceful will clean up pid files. Forceful will allow the service to start regardless. New in the latest code base is the following enhancement:

kill -USR1 pid_here

Which will output the current config of the server (running on the PID you specify) as YAML.

You’ve also got merb-manage, which you pass in start/stop/restart/reload. Reload does a fast restart which means very very little downtime between starts because it keeps the config and gem dependencies but just reloads the app files.

merb –help is your friend to find out what other options you’ve got

monitoring merb

collectd is a lightweight statistics collection daemon. Plugin based and network aware. A configuration definition looks like a fairly lightweight XML-style file. Includes a basic PERL CGI script which extracts data out and allows it to be viewed on a graph. Allows you to map things like the number of TCP connections over time, ruby process CPU time, memory usage, etc.

collectd-nagios is a plugin that allows nagios to interrogate any statistic collected by collectd. Point it at a statistic and flag up the appropriate alert should it not be within acceptable bounds.

Configuration management

An intro to puppet for those that haven’t used/seen it before. Allows you to describe how you want a various machine, machine type, etc. to be configured. What services need to be installed, running, and the required configurations files. Your manifests/configurations are managed via a version control system (svn, git, whatever you want). You check in your config changes, pull it out onto your puppet master and all your nodes will be updated with the required

Exception Handling

Use the merb_exceptions plugin and configure with the appropriate settings like so:

:exceptions:
  :app_name: Skippy
  :email_from: exceptions@skippy.com
  :email_addresses:
  - me@skippy.com
  :environments:
    - production

Make sure you’ve got merb-mailer setup, and set your delivery method to sendmail to reduce the potential problems:

Merb::Mailer.delivery_method = :sendmail

Named Instances

Within your configuration file you can give your instance a name, and it will mean you can easily identify processes when doing a process listing. The syntax is:

Merb::Config.use do |c|
  c[:name] = 'skippy'
end

Real time web apps with Orbited and Merb

What is Comet? It’s real-time web apps and it’s easy. Facebook should have known better, they didn’t need to write a custom stack on top of erlang. The wheel doesn’t need to be re-invented, we have the socket and protocol definitions that we need.

In HTML5 we’ve access to WebSocket, and you can create a new one like this:

new WebSocket("ws://host.com")

And we get 3 callbacks:

onread
onopen
onclose

Orbited is a python server that provides a WebSocket API within all browsers. The Orbited daemon acts as a security control between the browser and TCP, tunneling over HTTP. This means that you can implement push based services using protocols like XMPP to provide data to clients rather than the current AJAX polling approach. The list of protocols being implemented is much longer than I’ll list here, but it’s unlikely you’ll be left wanting.

Ultimately it means you can use HTTP where it’s appropriate, and then more appropriate protocols when they are better suited to the task (i.e., messaging).

It sounds like Orbited is a bit of an interim technology to back-port the WebSocket definition from HTML5 into today’s browsers. Once HTML5 is implemented you’ll be able to have native WebSocket support.

Writing a Merb plugin 101

Merb::Slices

merb-slices are reusable and over-writable mini-apps. They are much better than generated code because every time you want to change your code, you don’t need to re-generate the code you just pull in the update. If you’ve got your own overrides there is no problem. Some of the most popular slices are:

Login form
Password reminder
Blog
Forum
Event Registration
Admin Interface
Tune Up

When should you use them? When you want to target application content.

Merb::Plugins

Firstly, they are not DataMapper plugins (like the actsas_* plugins). They are used when you want to extend the framework, but don’t use aliasmethod_chain. This was touched on briefly, and it seems that in the merb world it’s about as evil as evil gets. Basically it’s a hack that lets people write their own methods as a decorator around the underlying framework API to get their own implementation in where they need it. Merb has taken the approach of providing hooks at the relevant places you might need them.

Merb Hooks

Merb::BootLoader

Is called when merb first starts up. You have access to at least two events (at the moment) within the boot up process:

Merb::BootLoader.before_app_loads
Merb::BootLoader.after_app_loads

Using the Merb API

There are essentially 3 types of methods within the API. Private, public, and plugin methods. Don’t ever touch the private ones, there is no guarantee they won’t be changed in core without warning. If you don’t have the hook you need, ask them for it. You can specify a version dependency within your plugin, and so long as you stick to the rules you’re right to say that your plugin will work <= 2.0.

Testing Plugins

Make sure you test your plugins through the full stack, you want to make sure that the plugin you’re using will work with the version of merb you are using. If you’re mocking everything out, you’ve got no guarantee of that.

Learning from Django

The django equivalent of merb-slices has proven to be awesome, so embrace them. On a project you’ll likely end up with a heap of them but don’t worry. The benefits from reusability are worth it, and they embrace the unix philosophy of do one thing, do it well, and work well together. Even simple django apps typically end up with a dozen or so slices.

So move towards slice-centric development where the slice becomes the core way to develop. The end result is something like the project pinax in django. The merb project gluttonberg is heading in that direction.

The merb routing kicks ass, hopefully django looks like that soon.

Something that is missing in merb that django does have is automatic admin interface generation. It’s not like rails scaffolding so it won’t break when you implement model changes. It’s tightly bound to the ORM though, and given merb’s ORM-agnostic approach it could be difficult to implement on this side of the fence.

Django also has generic controller generation, which sounds a lot like the make_resourceful type implementations in rails. You also get feed syndication almost for free.

It also looks like django.contrib has some awesome re-usable components. An example given in the presentation was importing logic to handle capturing US social security numbers. You import the library, and you get for free the appropriate form and all the validation.

The django docs have historically also kicked ass, and getting good docs makes adoption so much easier. Rdocs really don’t cut it.

Driven Merb: A user story

A pretty quick presentation on the benefits of using user stories in development, and BDD. The high-level full stack tests fits nicely with Yehuda’s “test what you care about” points from yesterday. We got some quick examples of running stories in both webrat and selenium. If you’re new to stories, check out David Chemlimsky’s stuff and rspec and the new story runner (called cucumber), they’ll do it much more justice than my paraphrasing here.

From Rails to Merb (Merb team panel Q&A)

Q: What is thor, and how do we use it?

A: Rake was sort of a hack that became the lingua franca for making scripts in ruby, but it didn’t support command line arguments so people provided a multitude of ways to get options out. Thor allows you to specify a command and provide a series of options, that was originally intended to run your own binaries. It’s as easy to use as rake, but provides more options, and allows tasks to be system wide and not just application specific.

Q: What is the core team doing to make merb easy to install and get up and running?

A: You don’t need git to run merb, you only need that if you want to run edge. You can install everything you need via gem. You will be able to do “gem install merb” and then “merb-gen app” to get up and running.

Q: What do you see as the biggest barrier to growth of merb usage?

A: The biggest barrier has been how fast it’s been developing and how quickly things have been changing. Going 1.0 will be the biggest barrier knocker-downer on that front, as people can be happy that the API is solid and stable and won’t be changing every week.

Q: How do you guys find time to organise yourself on various open source projects?

A: No sleep, and general addictions to OSS.

Q: What is the Engine Yard/Core team view on alternative ruby implementations such as Rubinius and JRuby, and does merb run on JRuby?

A: Yes it runs on JRuby. And merb will run on anything that rails runs on, Engine Yard’s position is ruby is good and make ruby win. So support them all.

Q: Short of improving stability and small evolutionary changes is there anything cool/big/different you’d like to to introduce to merb in plugins/slice/etc?

A: Wait for the keynote

Q: I have an app deployed with users, authentication, etc. how do I chunk it down to migrating with merb?

A: merb-auth isn’t going to dictate how your authentication is going to work, so you should just be able to port over your user model and the logic within. If you’re using restfulauthentication by Rick Olson then there is a mixin you can use._

Q: How does core see Rubinius support within merb?

A: As soon as it supports rails, we’ll support it. There’s lots of cool things in Rubinius that you can’t do in MRI, so as soon as it’s closer to production ready they’ll be onto it. We won’t limit merb to Rubinius once it’s out though, we’d look to support MagLev and co.

Q: Just like RJS covers up javascript, is merb stack going to cover merb to the point where it would be painful to get under the hood?

A: Absolutely not, if it does then it’s a bug. Merb stack is designed to generate the code you’d have to do yourself, not hide you from merb core. You’ll also be able to make your own stacks if you need to have your own standard setup.

Q: One of the things I dislike about rails is how the mindset of the rails core developers has changed as Rails has gained popularity. Do we have to worry about that here?

A: Absolutely not. If any of the core tenants of merb are violated Ezra will kill people :) Because the merb project isn’t tied into specific applications that they’re trying to build (unlike the 37signals apps that rails was extracted from), they’re always thinking about the benefit to the community. That’s not to say rails isn’t thinking about the community, but the merb approach is less opinionated because they are dealing with a diversity of opinions and not a single approach.

Q: How do we support internationalization in merb? Did you abandon the rails internationalization solution?

A: No we didn’t abandon internationalization and we still want to do it, but we want to do it in the merb way. Matt A personally worked on the rails API, and the problem with the way it was done in rails wont work with merb. Merb’s approach is all about performance and speed, and the original approach was slowing merb down. We will be working on it, but the only real solution is to do it in C. It’s about half way done.

Q: With ruby 1.9 coming around in December, has merb come to support that?

A: merb should run fine on it, but it hasn’t been tested for quite a while. That being said, we still wonder if the API really is locked down for 1.9 and that there won’t be any changes before December. The main problem is with support in other gems your app is dependent on. At this point, there is no clear guidance from the ruby core team so we have to wait until people are using 1.9

Q: You guys on the panel are all ruby and merb hackers, but you need to help us get there. What help can you provide?

A: Go read interesting code and try and grok it and understand it, then hack and make sure you understand it. Try and fix bugs in frameworks. There are a bunch of good books out there, read blog, read code on github, check out your local user groups and IRC. Make sure things you’re doing are fun and cool to keep your motivation levels high. Code things you’re interested in. There are 2 merb books coming out, “The Merb Way” and “Merb in Action”.

Q: Can you each talk about your workflow on a typical day with regard to coding?

A: The general consensus seems to be to spike when working on something experimental. Then detailing a list of things to implement, that can be either some specs, an actual implementation against the still-to-be-developed API, a list of work, etc. then go make it work. Go speak to someone else about the problem to get another perspective, especially from someone who doesn’t understand the problem you’re working on.

Q: If we see something that interests in merb and we want to contribute, what process do we follow?

A: Get in touch (mailing list, irc, etc.), check out lighthouse, or submit a patch. Come talk to them before you submit a patch though, you don’t want to spend hours working on a patch and having it rejected because it doesn’t fit the general approach guidelines. The process is usually fork the the repo, create a branch for each bug you fix so it can be cherry picked, fix it and don’t send a pull request but create a patch and attach it to a lighthouse ticket or provide the branch to pull from (and keep it up to date with master so it’s easier to apply when they get around to it).

Q: What’s your favourite deployment strategy?

A: Capistrano or Vlad, deploy to staging and if it works deploy to live. But always do it from a tag. And use ruby enterprise edition, it fixes garbage collection and allows you to efficiently share memory across child processes. When you deploy your app, you need to make sure that you use the thor task to rebuild the native gems so they’re recompiled for the deployed platform

Expanding our idea of the background web with Merb

HTML makes things look pretty, but all the interesting stuff happens in the background behind what you see in your web browser.

Running Background Tasks in Merb

You have a few ways to run tasks in the background. If you want to use cron, you can just setup some rake tasks. Within merb itself you have the renderthencall and run_later methods. You can call them like:

render_then_call do
  Node.inactive.purge
end

run_later do
  Node.scan(Node.first)
end
render

From the presentation I couldn’t glean what the effective difference is between the two calls.

WebServices in Merb

Merb handles RESTful web services with the provides, display, and render methods. An example definition that makes it possible is:

class Posts < Application
  provides :json
  def show
    @post = Post.get(params[:id])
    display
  end
end

It will automatically assume you want to provide HTML as well, if you want only JSON then you use onlyprovides_ instead of provides. Merb’s equivalent of the rails respondto_ statement is a case statement:

case content_type
when :html
  render
when :json
  display @post.to_json
end

The rails respond_to block means a proc needs to be created and evaluated when you won’t need to evaluate at least half of it (i.e., you only return one format). It’s about 10x slower than the merb approach. You actually don’t need the case statement normally though you should just call display and it will work out the format switching.

Keynote

Yehuda is talking about some of the bad press about “ruby is slow” and how the community response is usually “Ok!”, and that’s just embarrassing. He want looking for some benchmarks and found that rails generally outperforms CakePHP, CodeIgniter and similar frameworks the people traditionally think are much faster than ruby. They got in the region of 3-4 requests/sec. So how does merb stack up?

Merb Performance Benchmarks

A simple “hello world” scenario was used to show that ruby isn’t super slow, that if you go through the stack you still get reasonable performance. Returning a rack response directly from the router handled 3,500 req/sec. Doing a string render returned 2,000 req/sec. Rendering a template comes in at 1,500 req/sec.

Compare that to Apache serving static HTML which got 4,000 req/sec. Comparing it to the PHP equivalent of “hello world” showed almost identical performance to the merb controller rendering string “hello world”. Things really get interesting when comparing it to rendering a full template vs. using PHP CodeIgniter which is meant to be the fastest PHP engine. CodeIgniter gets 300 req/sec, that’s against the 1,500 req/sec you get from the standard merb setup. CakePHP was even worse with under 100 req/sec.

The ruby leaders have incorrectly conceded the performance ground just because ruby is slow at fibonacci, but PHP is slow in other areas. It has to load entire frameworks up at runtime. Our web servers (rack/thin) are far more optimized and based in C.

Merb vs Rails vs Sinatra

Next was comparing merb against other ruby options. Returning directly from the merb router kills everything, but there aren’t many scenarios you could use that. Sinatra is just slightly faster than returning from the merb controller, which both outperform Rails by a 3-4x. Rails itself still kills CakePHP.

The Future - Merb 2.0

It’s going to stay modular, but it’s also going to be integrated (back to that unix philosophy). Looking at django is an important part of moving forward because they’ve done lots of things well, but it won’t be so monolithic and a tangled web of dependencies. But the out of the box user experience should be just as pleasant. A consistent merb ORM so that you interface in a single way, and the ORM plugins will work out what underlying methods would be called in DataMapper, Sequel, or whatever you are using. These means the admin type interfaces django has would be possible.

Mailer will move towards a messenger approach, mail isn’t the only way you want to send notifications these days.

Moving towards Merb 2.0

The 1.0 release is a stake in the ground, the 1.x releases will be entirely backward compatible but they’re just stable experiments and an incremental step towards 2.0. Don’t consider any new changes between 1.0 and any 1.x to always be there, the 1.x releases are not the same stake in the ground. Bug fixes and additions to the stable release will be applied as 1.0.x.

How can you help?

Contribute to the central slice repository, but make sure they rigidly adhere to the public API and don’t use any plugins or slices that aren’t. Report missing functionality. Work on the wiki, you can’t beat community driven documentation. Help test RC1. Report any missing documentation. List your site on Merbunity.com.

Merbcamp - notes from the edge

Sat, 11 Oct 2008 18:21:14 GMT

A recap on the presentations from the first day of the very first Merbcamp. If you want to read about day to, check this page Merbcamp - Day 2.

Ezra’s Keynote

Ezra gave a great speech talking about merb, some of the design decisions that are directing it, but the two most interesting points for me were:

Minigems

A really useful drop-in replacement for rubygems. Instead of loading the full gem list into RAM (and into every mongrel instance of your app as a result) minigems loads them as required. The reported saving is around 10-20MB of RAM, depending on how many gems you have installed on your system. And using it is simple, instead of:

require 'rubygems'

you do:

begin 
  require 'minigems'
rescue LoadError 
  require 'rubygems'
end

Thanks to Fabien for all his hard work, go grab it from his github repo.

Nanite

A “self-assembling fabric of daemons” that Ezra has written. It’s based on RabbitMQ, and essentially you create agents and mappers. Agents do the work, mappers keep track of who does what. And to keep running with what is written on the tin, agents are transient, persistent, and transactional.

Essentially the agents ping the mappers on a regular basis with information on who they are, what they can do, and their current/average load. If the mapper doesn’t hear from the agent within the timeout period, it’s removed from the pool of workers. When new work comes in and is passed to the mapper, it can decide who to pass the work to based on any arbitrary rules. It can go to the agent with the lowest load, it can be broadcast to all agents, a random one, or any rule you decide. This is awesome if you’re wanting to push work off into the cloud. Just dial up a new agent, it informs the mapper of who it is and what it can do, and there is zero configuration requirement… just keep dialing them up as required. Additionally, you can just turn them off and the mapper will handle it gracefully. Ezra is looking to add in support for the merb rack interface so the same can be done for the actual merb process, just dial up merb web servers as nanite agents and scale your site on demand at the click of a button.

Go check it out

The slides and a video of the presentation is now also available.

Merb plumbing: The router

This one ran way too quick for me to take any detailed technical notes on, but suffice to say that the routing in merb is absolutely amazing. Handling both domains and subdomains is a breeze, with code like the following:

match(:subdomain => /(.*)/).
to(:account => ":subdomain[1]") do
  #application routes here...
end

r.match(%r[^/movies/(\d+)-(\d+)-(\d+)$]).
to(:controller => "movies", :movie_id => "[1][2][3]", :action => "show")

As you can see, you can do pattern matching within the route definition, and then use the captures as parameters. You can also add in requirements on the parameters much like rails:

match("/artices(/:year)", :year => "[\d]*").

Or specify a route that has optional components:

match("/artices(/:year(/:month(/:day)))/:title").

It may be too subtle to notice here so I’ll point it out; unlike rails the optional parts of the route don’t have to be at the end. For this rule to be met, the requested path must start with /artcies and have a :title at the end, :year, :month, and :day are all optional.

Particularly impressive was Yehuda mentioning in the Q&A session how routes can initiate DB calls, and then pass the found objects directly into the controller. One call for matching the route, and then re-using the same object in the controller. Nifty!! Some more detailed usage examples have been put up in this pastie.

Merb Slices

They’re kinda like engines in rails, without as much voodoo because of the way merb has opened up the necessary hooks to support them natively within core. Install commonly used pieces of functionality in your apps as a slice, and away you go

What if you want to customize?

You have a few options, and it depends on what it is you want to customize but your options essentially are:

monkey patch it (using overrides)
use the “patch” rake task
update settings within the config hash

On that last one, each merb slice has it’s own configuration has available at

Merb::Slices::config[:blog_slice]

Over-riding behaviour

Much like engines, you can use slices to create custom controller actions, custom models, or custom views. What happens if you don’t like the way the behaviour in the slice works? Well go change it. If you want to natively support an action or format that the slice didn’t come with, just go create the template. Your own monkey-patches exist in the /slices directory within your app. Just add in the new models, views, controllers definitions, etc. They’ll take precedence over any conflicting definitions within the slice.

Initializer

When a slice initializes, it gets quite a number of hooks available to determine what it should load and add in its own behaviour. The main load points are:

Declaring dependencies/boot loader
Defining routes
Event hooks (initializing engine, engine loaded, etc.)

Handling URLs

The URL helper method is handled by slice_url for your slices, so you use it like this:

slice_url(:controller => ...., :action => ...)
slice_url(:merb_auth_slice_password, :login)

The latter would give you the URL to the login route, as defined within the merb_auth slice

Being a good citizen

A few other points to keep in mind, you should namespace everything to prevent conflicts with the actual application that is including your slice. This includes models. You just namespace models like so:

class MySlice::Post
end

Also, package up any assets you depend on. The end-user can include them in their app with a rake task:

rake slices:myslice:copy_assets

And now the assets will be available in /myslice/images/…

download the presentation here

Yehuda’s Testing Philosophy

Basically the gist of this talk was that TDD (at least how most people do it) is useless as a means of regression testing. If you’ve have to go back and change tests because you’ve changed code then you no longer have any proof that your code still works. There is no existence of a test that tests your original intention. That being said, TDD is still good and Yehuda still uses it for development, but his approach is somewhat different.

Yehuda’s approach to testing rests on 3 fundamental principles:

If you have broken application something should fail
If you have a working application all tests should pass
Test what you care about

The latter point needs some clarification to prevent the “Well, duh!” response. You don’t care about which methods are called, you care about the end result. Do you really care that you’ve called find on your model? Or that the output you’re expecting has been displayed as intended to the end user?

If you change views, helpers, etc. then usually tests fail because you’ve changed code. The tests are incorrectly claiming that things are broken, but they’re not. The site still works. Refactor tests into such a way that they don’t break so easily, and you still have proof that everything still works.

One point that came up in the Q&A: He thinks you should never mock your own code. If you’ve got your own code, just test directly against it. Mocks should be used for web services or a 3rd party API. 3rd party providers should provide a mock framework for testing against.

All very interesting, and I need to give it some more thought. I can see the underlying benefits from the number of times a minor change has taken a few hours to implement because of the impact on testing, not because of the complexity of the change. I do wonder what impact it has on tests also serving as documentation of your implementation though.

MerbAuth: Darwinian authentication

MerbAuto was originally the first plugin with multi-ORM support, and was a direct port of Restful Authentication from rails but it was awfully complex. Then slices came along, and merb-auth was born.

Authenticating the session

In the interest of reducing some of this complexity, merb-auth was re-written with a pretty major implementation change. Don’t authenticate controller, we don’t really care about it. Authenticate the session. It’s the at the session level that we can determine who a user is, not on an individual controller. Thankfully, merb routing makes session available within the router. Not only does that mean that we can authenticate the session, but we can do it in the routing!!

Design Benefits

Merb routing sets response codes correctly from raises, so rather than a 302 redirect on failure you can raise an unauthorized response. Authentication is not user management, and your underlying user requirements could be vast so this is just an authentication framework and you’re left to implement user admin/creation yourself. And it’s just that, a framework and not a plugin. It allows you to build your own authentication for your application.

Implementation options

merb-auth includes a password slice which allows you to have both form base authentication or regular HTTPbasic authentication. It also offers you two different ways to protect your application; either in the router or within a controller. To protect at the router level you just do the following

protect do
  resources :admin
end

Or in a controller:

before :ensure_authenticated

Why protect within a route?

The main benefit is that it is earlier in the request/response cycle, and so processing can stop right in the dispatcher and send the response straight back to rack. Within a controller you’ll need to create a controller instance, possibly run some filters, etc.

Authentication strategies

A stratergy contains logic for authentication. You need to implement a run method, and then within it any logic you feel pertinent to authenticating a user. You can also declare many different strategies for various login types, say one for teachers and one for students.

More details and actual code implementations are included in the presentation

Damage control: A primer on Merb

This is an intro to merb and how to get everything installed and get up and running quickly. The usual things on how merb isn’t opinionated and is very modular. If you want to get straight into it though, you can get opinionated and get a setup with the following franmeworks:

helpers
datamapper
rspec
merb-auth
sqlite3
jquery

To get up and running quickly these days Yehuda has written a new gem called thor. It’s basically a cross between rake and sake, except that you can write the tasks in syntax that looks more like ruby as opposed to it’s own custom DSL. Once you’ve installed thor, you need the merb tasks:

gem install thor
thor install http://merbivore.com/merb.thor

and now to install merb and it’s dependencies you use thor

thor merb:edge --install

And a useful piece of advice for those that like to drink the kool-aid a little too much, if you’ve already got a perfectly good working app in rails there is really no point porting it to merb.

Keep a hot cache with merb-cache

The video stream died at this point so I’ve nothing to talk about, I’ll come back and update it retrospectively once I’ve caught up. The same goes for the following topics. If anybody can kindly fill me in in the interim while I try and track down some streams…

Using multiple databases in Merb: with special guest CouchDB

DataMapper: the Ruby ORM that kicks butt and takes names

Merb and Sequel

jQuery on Merb

Yay! Someone publically pointing out why using the built in rails ajax/javascript helpers is generally a bad idea. Especially remoteformfor. Javascript is part of the web stack, and you’ll eventually get to a point where you actually need to know how to program in it so hiding behind the rails helpers can only help you for so long. Plus all the nasty violations you get re separating presentation from logic.

Testing with Javascript

For testing javascript there is ScrewUnit (check out this post for more detail) which is a javascript port of rspec with before, after, and describe blocks. There is also a mocking framework for handling XHR so you can appropriately test your ajax requests.

Testing with selenium is good for the full coverage and acceptance testing that a person would normally be doing. It can be a difficult to test everything you care about though.

Why can we use ruby to generate HTML and SQL, but not Javascript?

SQL is a mismatch for what we’re trying to do, hasmany, belongsto, and all the various helpers represent logic which isn’t well defined in the SQL layer. And HAML is only a very very thin abstraction layer from HTML. Javascript is an entire language though, which isn’t running on our environment it’s running on the client PC and has it’s own state. Most Javascript code generators don’t work that well, and the point at which you need to actually know the language is hit much earlier in the development of a project than it is for something like SQL. That being said, it’s still useful to have someone who actually knows SQL on your team.

DataMapper Migrations

Sun, 05 Oct 2008 21:20:20 GMT

I’ve been working on some merb related things of late, and I’m planning on writing a more detailed article on creating a merb application from scratch as so many of the tutorials out there kind of leave you hanging. In the interim though, I’m just going to touch on a little nugget of information that isn’t very well publicised regarding DataMapper, one of the ORMs that gets used with merb.

ActiveRecord Style Migrations with DataMapper

That’s right, you aren’t left having to wrestle with the command line and ALTER TABLE statements just because you’ve left ActiveRecord behind. All the migration joy you’ve come to love with ruby on rails can come and live with you happily with merb and DataMapper.

First things first, you’ll want to generate a new migration:

merb-gen migration my_migration

We should now have a file named 001_my_migration.rb and if we look at it, it won’t be too foreign based on our ActiveRecord experiences:

migration 2, :my_migration do
  up do
  end

  down do
  end
end

To pad it out with some code that is actually useful, add in the definitions for what you want done:

migration 2, :my_migration do
  up do
    create_table(:posts) do
      column(:id, Integer, :serial => true)
      column(:title, String, :size => 255)
      column(:description, Text)
      column(:published, Boolean)
      column(:created_at, DateTime)
      column(:updated_at, DateTime)
    end
  end

  down do
    drop_table(:posts)
  end
end

Running the DataMapper Migration

And now that we’ve got the migration we are happy with, applying it to our underlying database is just a matter of the following:

rake dm:db:migrate

Introducing Map.ly, collaborative mapping

Tue, 12 Aug 2008 09:00:00 GMT

From the necessity is the mother of all invention category, I introduce a new product I've hastily cobbled together over the past week or so. Map.ly is a means of 'sharing directions, locations, and maps' with other people.

It is still very much in development, I'm loathe to call it beta as it's probably closer to alpha but still; So what is it's purpose? My girlfriend is forever having to plan meetings and trips for the people in her team, and trying to herd these sheep ahem I meant orientate her colleagues in foreign cities can be tricky. She needed something to accompany their printed itinerary showing all the various locations they'd need to be during their stay, and where they were in relation to each other. And print out a google map for each location seemed needlessly difficult and not that helpful, we wanted to plot everything on the one map if possible.

Enter Map.ly. Double click where you want to place a marker, enter in the name and some details, and save. You'll get a nice short URL back to the map that you can pass around and you can all add your own markers to it. I think support in IE might currently be a little bit flaky, I'll check later today. And there are a number of features still in the pipeline (getting directions between points, entering in addresses, etc.) but I'm hoping it will prove to be something useful. So go give someone some directions to somewhere

And most useful to the geeks out there, is that it's forced me to resurrect a long since dead gem I once wrote for mapping as I felt YM4R and the other Map API gems just weren't ruby enough. Once the feature set is locked in, I'll release an updated version allowing you to do all the cool things that get implemented on Map.ly.

Nginx Intermediary SSL Certificates and Passphrases

Mon, 11 Aug 2008 09:00:00 GMT

I've had to do this 3 or 4 times in the past 12 months, and each time I waste a good chunk of time trying to re-discover how to do it. So this post is as much for my own documentation sake as it is sharing the love. If you've ever bought an SSL certificate from GoDaddy or a similar provider that provides and intermediary certificate to include, you might have run into a bit of a problem trying to work out how to include it in your nginx config. Well it's actually pretty straight-forward, provided you remember the steps.

Merging the site and intermediary SSL certificates

First thing you'll need to do is create a new certificate, which is just the one for your site and the intermediary merged together. In Apache you'd specify them both individually. Nginx only allows you to specify one, so lets put them in the same file:


cat rubypond.com.crt >> rubypond.com.crt.merged
cat intermediate_bundle.crt >> rubypond.com.crt.merged

Removing SSL Passphrase

The next step, is to remove the passphrase off the key for the site. I'd love to keep it on, but it means nginx can't be restarted without an operator at the console to enter in the passphrase every time which is definitely less than ideal. So to remove the passphrase from your key, I'll first back it up and then take it off with openssl:


mv rubypond.key rubypond.key.passphrased
openssl rsa -in rubypond.key.passphrased -out rubypond.key

Configuring Nginx to use your new SSL certificate

This config is taken straight from my earlier article on setting up nginx, ssl, and virtual hosts, but I'll include it here just for good measure too:


server {
  listen 443;
  server_name myserver.com;  

  ssl on;
  ssl_certificate /etc/nginx/certs/rubypond.com.crt.merged; 
  ssl_certificate_key /etc/nginx/certs/rubypond.key; 

  # put the rest of your server configuration here.

  location / {
     proxy_set_header X-FORWARDED_PROTO https;

     # put your config here
  }
}

Just be sure to point the certificate entry to the new merged certificate, the key at the now un-passphrased key, and restart nginx. Done.

The complete guide to setting up Starling

Thu, 17 Jul 2008 09:00:00 GMT

We wanted to push some long running tasks off to the background so that we didn't tie up a mongrel needlessly. I've wrestled (and won!) with backgroundrb in the past, but it just seemed like a chore. And they've since changed the API enough to mean it would be back to the drawing board, so we may as well assess some of the other options. So without too much further ado, the complete guide to installing, using, and monitoring starling.

For those that haven't heard Starling is the back-end queuing system that Twitter use. And despite the stability issues Twitter have, if initial impressions are anything to go by I don't think Starling is the reason. It's fairly light, sits on top of MemCache, and is very quick. The initial lure to Starling though was via the Workling plug-in. We could get a prototype out the door using just the Spawn workling, and then when a queue became important switch over.

Installing Workling

We're going to take the same approach I did, and get a prototype background process happening with Spawn first. So let's install Workling and Spawn:


script/plugin install http://svn.playtype.net/plugins/workling/
script/plugin install http://spawn.rubyforge.org/svn/spawn/

Now in your environment (environment.rb? an initializer? you decide) add the following:


Workling::Remote.dispatcher = Workling::Remote::Runners::SpawnRunner.new

Done!

Creating A Workling Background Worker

Now it's time to create a worker, basically to define what it is exactly that should be executed in the background. For the same of illustration, I'm just going to loop and create some records. I create the follow app/workers/example_worker.rb file:

class ExampleWorker < Workling::Base

  def create_new_records
    1000.times do
      Item.create :some_name => options[:some_name]
    end
  end 

end

Calling Your New Workling Background Worker

Now in whichever controller would normally need to start this task, you can place a call to the workling worker:

ExampleWorker.async_create_new_records(:some_name => "This Person")

Workling will create a new method for us to call, by prefixing out method name with async_ to allow us to call it asynchronously. Pass in a hash of values you want to use into the method, workling will add a key named :uid containing the unique identifier assigned to this task.

And now we're done here too! This will use Spawn to fork the job, take it out of the mongrel process, and let your rails stack continue and not hold up your other users.

Installing Starling

Install Starling by grabbing the gem and the MemCache Client (which strangely has a dependency on ZenTest which I need to investigated):

sudo gem install memcache-client starling

Now the dependencies are installed, we need to start up the required services. You're going to need the Starling daemon running, as well as the workling client. I'd rather not have Starling run as root so first I create a user, and then give it somewhere to create it's pid and spool files (you can probably skip this and run as root on your local dev box):

sudo /usr/sbin/adduser -s /sbin/nologin starling
sudo mkdir -p /var/spool/starling
sudo mkdir -p /var/run/starling
sudo chown starling:starling /var/spool/starling
sudo chown starling:starling /var/run/starling

And now time to start up the services:

sudo -u starling starling -P /var/run/starling/starling.pid -d
script/workling_starling_client start

A note of particular importance here. The workling_starling_client will run in whatever environment RAILS_ENV is set to, and is not passed in via a command line option. It also does some fruity things with loading up your workers at run time, so if you're running development and test on the one box you'll need to stop the workling client and restart it each time you want to change environments (or best yet, set your rake:spec/rake:test task to do it for you).

Switching Over To Starling For Background Processing

Switching between background processing systems is pretty simple, we just update the environment setting we made earlier to:

Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new

And that's it, business as usual!

Monitoring the sucker

It is all fine and dandy having it up and running now, but it'd be nice to keep it that way. That's where god comes in and some tomfoolery with our app.god config. If you've not already done so, I suggest you read my previous article on monitoring rails with god which talks about creating a generic config file in your rails app so you don't need to manage god configs for each instance.

I'm going to change the example app.god in that article to look like:

class Rubypond
  attr_reader :ports, :server_names, :workling_client

  def initialize
    @ports = [5000, 5001]
    @server_names = "www.rubypond.com rubypond.com" 
    @workling_client = true
  end
end

@apps << Rubypond.new

And within the god.conf on the server I'll add the following methods:

def configure_workling_client(w, rails_root,  app_name)
  w.uid = "mongrel" 
  w.gid = "webserver" 
  w.name = "#{app_name}-workling" 
  w.group = "#{app_name}" 
  w.interval = 30.seconds
  w.start = "RAILS_ENV=production #{rails_root}/script/workling_starling_client start" 
  w.stop = "#{rails_root}/script/workling_starling_client stop" 
  w.restart = "#{rails_root}/script/workling_starling_client restart" 
  w.pid_file = "#{rails_root}/log/workling.pid" 

  w.behavior(:clean_pid_file)
end

def monitor_app(w)
  start_if_not_running(w)
  restart_if_resource_hog(w)
  monitor_lifecycle(w)
end

Okay, so monitor_app is actually just renaming monitor_rails_app to be more generic. Make sure you change references everywhere though. Elsewhere in the god.conf file we had a block similar to the following:

@apps.each do |app|
  app_name = app.class.name.downcase
  rails_root = "/var/www/apps/#{app_name}/current" 

  app.ports.each do |port|
    God.watch do |w|
      configure_mongrel(w, rails_root, app_name, port)
      monitor_rails_app(w)
    end
  end

  God.watch do |w|
    configure_workling_client(w, rails_root, app_name)
    monitor_app(w)
  end if app.respond_to? :workling_client
end

The last God.watch there is where the magic happens. It will look at the app we've defined in app.god, and if we've provided an attribute called :workling_client then it will automatically monitor a workling client for this app. We're not quite done yet though. We also need Starling to be running, but that's not on a per app basis we just have one for the server. So I'm just going to insert this directly into god.conf:

God.watch do |w|
  w.name = "starling" 
  w.group = "starlings" 
  w.uid = "starling" 
  w.gid = "starling" 
  w.interval = 30.seconds # default
  w.start = "/usr/bin/starling -P /var/run/starling/starling.pid -d" 
  w.start_grace = 20.seconds
  w.restart_grace = 20.seconds
  w.pid_file = "/var/run/starling/starling.pid" 
  w.behavior(:clean_pid_file)

  monitor_app(w)
end

Now restart god to take the new config, deploy you changes to your rails app, and cross your fingers ;)

Sexy Forms in Rails

Wed, 16 Jul 2008 09:00:00 GMT

I've been meaning to release this for quite a while now, I've finally got around to packaging it up for some form of public consumption. Basically, I got sick of having to manually create label tags for each of my form inputs. They should be there by default so that users with visual or other impairments have enough additional information to use the site. I'm also lazy, and would have a tendency to forget to put them in otherwise. I also wanted a consistent way to display any additional contextual information, so here it is.

I'll go through the code in detail here, but I'll keep it updated with any changes and available for download at the semantic form builder git repo. So now on to the code.

It started with a custom FormBuilder.

class SemanticFormBuilder < ActionView::Helpers::FormBuilder
  include SemanticFormHelper

  def field_settings(method, options = {}, tag_value = nil)
    field_name = "#{@object_name}_#{method.to_s}" 
    default_label = tag_value.nil? ? "#{method.to_s.gsub(/\_/, " ")}" : "#{tag_value.to_s.gsub(/\_/, " ")}" 
    label = options[:label] ? options.delete(:label) : default_label
    options[:class] ||= "" 
    options[:class] += options[:required] ? " required" : "" 
    label += "<strong><sup>*</sup></strong>" if options[:required]
    [field_name, label, options]
  end

  def text_field(method, options = {})
    field_name, label, options = field_settings(method, options)
    wrapping("text", field_name, label, super, options)
  end

  def file_field(method, options = {})
    field_name, label, options = field_settings(method, options)
    wrapping("file", field_name, label, super, options)
  end

  def datetime_select(method, options = {})
    field_name, label, options = field_settings(method, options)
    wrapping("datetime", field_name, label, super, options)
  end

  def date_select(method, options = {})
    field_name, label, options = field_settings(method, options)
    wrapping("date", field_name, label, super, options)
  end

  def radio_button(method, tag_value, options = {})
    field_name, label, options = field_settings(method, options)
    wrapping("radio", field_name, label, super, options)
  end

  def check_box(method, options = {}, checked_value = "1", unchecked_value = "0")
    field_name, label, options = field_settings(method, options)
    wrapping("check-box", field_name, label, super, options)
  end

  def select(method, choices, options = {}, html_options = {})
    field_name, label, options = field_settings(method, options)
    wrapping("select", field_name, label, super, options)
  end

  def time_zone_select(method, choices, options = {}, html_options = {})
    field_name, label, options = field_settings(method, options)
    # wrapping("time-zone-select", field_name, label, super, options)
    select_box = this_check_box = @template.select(@object_name, method, choices, options.merge(:object => @object), html_options)
    wrapping("time-zone-select", field_name, label, select_box, options)    
  end

  def password_field(method, options = {})
    field_name, label, options = field_settings(method, options)
    wrapping("password", field_name, label, super, options)
  end

  def text_area(method, options = {})
    field_name, label, options = field_settings(method, options)
    wrapping("textarea", field_name, label, super, options)
  end

  def submit(method, options = {})
    field_name, label, options = field_settings(method, options.merge( :label => "&nbsp;"))
    wrapping("submit", field_name, label, super, options)
  end

  def submit_and_cancel(submit_name, cancel_name, options = {})
    submit_button = @template.submit_tag(submit_name, options)
    cancel_button = @template.submit_tag(cancel_name, options)
    wrapping("submit", nil, "", submit_button+cancel_button, options)
  end

  def radio_button_group(method, values, options = {})
    selections = []
    values.each do |value|
      if value.is_a?(Hash)
        tag_value = value[:value]
        label = value[:label]
        help = value.delete(:help)
      else
        tag_value = value
        value_text = value
      end
      radio_button = @template.radio_button(@object_name, method, tag_value, options.merge(:object => @object, :help => help))
      selections << boolean_field_wrapper(
                        radio_button, "#{@object_name}_#{method.to_s}",
                        tag_value, value_text)
    end
    selections    
    field_name, label, options = field_settings(method, options)
    semantic_group("radio", field_name, label, selections, options)    
  end

  def check_box_group(method, values, options = {})
    selections = []
    values.each do |value|
      if value.is_a?(Hash)
        checked_value = value[:checked_value]
        unchecked_value = value[:unchecked_value]
        value_text = value[:label]
        help = value.delete(:help)
      else
        checked_value = 1
        unchecked_value = 0
        value_text = value
      end
      check_box = @template.check_box(@object_name, method, options.merge(:object => @object), checked_value, unchecked_value)
      selections << boolean_field_wrapper(
                        check_box, "#{@object_name}_#{method.to_s}",
                        checked_value, value_text)
    end
    field_name, label, options = field_settings(method, options)
    semantic_group("check-box", field_name, label, selections, options)    
  end
end

Personally I think they are a terribly underused aspect of the Rails framework. Anyway, stepping through what we've got here. We inherit from ActionView::Helpers::FormBuilder and then begin overriding the various tags that will be called within a form_for. All we do for most methods is call field_settings followed by wrapping. In the method field_settings we set the name of the field to the standard rails convention like person_first_name (@object is created for us automatically by form_for so we don't need to worry about it here). Next we set the label text to display for this form input, it will default to something sensible (generally the name of the field you are generating the input for) or you can override it per input by passing :label in as an option. And finally, we check if you have passed :required => true in and if so we set a required class so we can style it differently. All the relevant info gets returned, so we can pass it into the wrapping method.

So where exactly is wrapping? I've moved it out into a helper, as I wanted to use this functionality in regular form_tag calls as well as form_for. So take a look in semantic_form_helper:

module SemanticFormHelper

  def wrapping(type, field_name, label, field, options = {})
    help = %Q{<span class="help">#{options[:help]}</span>} if options[:help]
    to_return = []
    to_return << %Q{ <div class="#{type}-field #{options[:class]}">}
    to_return << %Q{<label for="#{field_name}">#{label}#{help}</label>} unless ["radio","check", "submit"].include?(type)
    to_return << %Q{<div class="input">}
    to_return << field
    to_return << %Q{<label for="#{field_name}">#{label}</label>} if ["radio","check"].include?(type)    
    to_return << %Q{</div></div>}
  end

  def semantic_group(type, field_name, label, fields, options = {})
    help = %Q{<span class="help">#{options[:help]}</span>} if options[:help]
    to_return = []
    to_return << %Q{<div class="#{type}-fields #{options[:class]}">}
    to_return << %Q{<label for="#{field_name}">#{label}#{help}</label>}
    to_return << %Q{<div class="input">}    
    to_return << fields.join
    to_return << %Q{</div></div>}
  end

  def boolean_field_wrapper(input, name, value, text, help = nil)
    field = []
    field << %Q{<label>#{input} #{text}</label>}
    field << %Q{<div class="help">#{help}</div>} if help
    field
  end

  def check_box_tag_group(name, values, options = {})
    selections = []
    values.each do |item|
      if item.is_a?(Hash)
        value = item[:value]
        text = item[:label]
        help = item.delete(:help)
      else
        value = item
        text = item
      end
      box = check_box_tag(name, value)
      selections << boolean_field_wrapper(box, name, value, text)
    end
    label = options[:label]
    semantic_group("check-box", name, label, selections, options)    
  end      

end

There are 3 cases to look at initially. Wrapping a normal input, wrapping a boolean input (like a check box), and wrapping a group of check boxes. For wrapping a normal input, we create a label on the left and insert the field into a div on the right. I've got the boolean one there for the 'remember me?' type of scenario on a login page, where it looked to make more sense to have the label to the right of the checkbox rather than the left. And finally, wrapping a group of elements which isn't handled at all well in the current Rails setup. Overwhelmed yet? Let's just look at how you use it then:

<% form_for @document do |f|
  field_set_tag "Form Details" do %>
    <%= f.date_select :date, :required => true, :help => "date the something happened" %>
    <%= f.text_field :number, :required => true, :help => "the reference number for this thing" %>
    <%= f.select :external_id, [["Choose an option...",""]] + @externals.map{|c| [c.name, c.id]}, :required => true, :label => "options", :help => "select something from the list" %>
    <%= check_box_tag_group "document[other_items][]", @others.map{|u| { :value => u.id, :label => u.description }}, :label => "including these?", :help => "tick the whatever boxes are appropriate for this&nbsp;thing" %>
    <%= f.text_field :name, :help => "what was Willis talkin' about?" %>
    <%= f.check_box :list, :label => "mailing list", :help => "can we send you a bunch of spam?" %>
    <%= f.submit_and_cancel("save", "cancel") %>
<% end %>

You'll see we can pass in :required => true to visually identify required fields to the user. If we don't like the form name (like the generic 'list' attribute, then we can override with :label =>. And if there is additional contextual help to provide, we've got :help =>. We've also got a pretty way of showing a selection of check boxes with check_box_tag_group, but you'll have to specify the field name yourself. I've also put a convenience method in called submit_and_cancel that takes the text for a submit, and cancel button respectively. Catching the cancel action is up to you though.

And voila! The finished product. Any problems with it, please let me know or fork and fix/pull request on github. I've tested it on Safari 3, Firefox 2/3, and IE7. No idea what it looks like on IE6 just yet.

UPDATE: It's now a plugin, sexy and semantic forms are now available as a plugin

Useful Flash Messages in Rails

Fri, 11 Jul 2008 09:00:00 GMT

Inevitably somewhere in your rails app you display flash messages to your users to inform them that an action has (or hasn't) taken place. Sometimes they need to provide more info, maybe you've just created some new information for them. Why make them sit around, or wonder where they need to go next when you can take them straight to it in the message itself?

I've ripped this code out of a project I've been working on and made it a little more generic. Initially I was going to just insert a link in the flash message, but as I was rightly pointed out by my pair and elegant coder extraordinaire, that would mean I'd be polluting my controller with logic and content that should be handled by the view. How do we solve a problem like pollution?

First, we updated the controller so that the flash message now looked something like this:

flash[:error] = "Username and password do not match. If you have 
                   forgotten your password you can %s"

The %s is there to use a string substitution later. If you wanted to insert your link somewhere else, move the %s. Next, we created a new key in flash to hold our additional information (the link path and text):

flash[:error_item] = ["reset it&nbsp;here", forgot_path]

Quite simply, it's an array with the first item being the text and the second the path. Now for handling it within our view, here's what I've got in application_helper.rb:

FLASH_NOTICE_KEYS = [:error, :notice, :warning]

  def flash_messages
    return unless messages = flash.keys.select{|k| FLASH_NOTICE_KEYS.include?(k)}
    formatted_messages = messages.map do |type|      
      content_tag :div, :class => type.to_s do
        message_for_item(flash[type], flash["#{type}_item".to_sym])
      end
    end
    formatted_messages.join
  end

  def message_for_item(message, item = nil)
    if item.is_a?(Array)
      message % link_to(*item)
    else
      message % item
    end
  end

The reason for the constant is that I don't use just :error and :notice, I've also got :warning for times when something needs action but not because of something the user has done wrong (I don't want to make them feel bad when they've not had the opportunity to be good!). In flash messages I exit out on the first line if we don't have content to process for the flashes I expect to output (:error, :notice, and :warning) otherwise I loop over each and create a div to contain the message. The class on the div will be either error, notice or warning so you can style it properly. We defer working out the content to message_for_item

In message_for_item I check if the input is an Array, and if so I use the build in string substitution method to insert the link in the relevant place. As you may notice, I've called splat on item by putting an asterisk in front of it. That means your array can essentially just be the parameters you'd pass in to link_to, allowing you to add additional classes or options if you wish. If it's not an array, I skip the link to and just to the substitution. This will handle traditional flash messages as well as some that might want substitution of a string or other object. Wherever you'd previously inserted your flash messages, now you call:

<%= flash_messages%>

and the end result is something like this:

So what else could you do with this? Well we extended it to be a little more intelligent based on the object that was passed in. You could also use it as part of a before filter if you've got flash messages that are relatively similar across actions on controllers. Just put %s in the bit to substitute, and then at the relevant part of your action you can insert the a string into flash[:error_item] or the appropriate key, and have it displayed. Like this:

before_filter :set_message, :only => [:create, :update]

  def create
    @order = Order.create!(params[:order])
    flash[:notice_item] = "#{@order.items} items" 
  end

  def update
    @order = Order.find(params[:id])
    @order.update_attributes(params[:order])
    flash[:notice_item] = "#{@order.items} items" 
  end

  private
    def set_message

      flash[:notice] = "Thank you, we have updated your order with %s" 
    end

I hope it helps. Coming up next, my sexy form builder!

Make your own IM bot in Ruby, and interface it with your Rails app

Thu, 26 Jun 2008 09:00:00 GMT

In a super secret project I'm currently working on, I've been looking for ways of making it easier for people to interact with the system without the need to log in to the website. That's included the obvious things like having a RESTful API so they can put their own services and interfaces on top, but that only works for developers or 3rd party application providers. What about your average Joe on the street, how come they get forced into having to log in to the site to update their data, etc.? Why not provide a means of carrying out simple tasks in an application they probably already have? And so an Instant Messaging interface was designed to send a instruction to a robot, and it's relatively simple to get setup.

The best thing about it is that we help give more control and time back to the user, hopefully making the system more appealing to use. No more breaking your train of thought while you go log into a site to check something, just a quick message to a bot and you can forget about it. Or conversely, you wont defer the task because the overhead in writing a reminder is possibly more than the time it would take to just do it. And for those fans of the Getting Things Done (GTD) approach to task management, it means a whole range of tasks can be suddenly just 'get done' in the process phase

So how do we give such amazing power and flexibility to our users? First you're going to need to install the xmpp4r gem which at the time of writing doesn't seem to play nice with the new version of rubygems (I think it's to do with the way new rubygems preserves permissions). The problem appears to be fixed in version 0.3.2.99, but only 0.3.2 is available as a gem. So in the interim, we are going to have to download it directly from the git repository and build it ourselves:

git clone git://github.com/ln/xmpp4r.git xmpp4r
cd xmpp4r
rake gem:install
sudo gem install xmpp4r-simple

And now some ruby code. Connecting is straight-forward:

require 'rubygems'  
require 'xmpp4r-simple'
messenger = Jabber::Simple.new('my-bot@gmail.com', "bot-password")

As is sending a message:

messenger.deliver("a-real-person@gmail.com", "Why hello there Mr. Person, your bot is here now!")

If you wish to have your service sit around and respond to messages, then something you'll need something like the following:

while true
  messenger.received_messages do |msg|  
    puts msg.body  
    messenger.deliver("a-real-person@gmail.com", "Got your message, thanks!")  
  end  
  sleep 2  
end

Now just to point out for those that aren't paying close attention, the above code does the following:

Creates a loop that we will continually go over
Call received_messages and pass any messages into a block, iterating over each one at a time
Stop processing for 2 seconds so the system can take a break and we don't use up all of the resources

Of particular importance once again, I've created an infinite loop here to listen to incoming messages. DO NOT put this code in a rails app, this is the kind of process that needs to be handled outside of rails as a supporting but background task. If you try and kick it off via rails it will process indefinitely and tie up one of your mongrel instances until it dies and likely degrade the performance of your site in the process.

That's all well and good, but wasn't the whole point of this to give our users a more convenient way to interface with our site and their data? Let us have a look a a slightly more useful implementation:

require 'rubygems'  
require 'xmpp4r-simple'
messenger = Jabber::Simple.new('my-bot@gmail.com', "bot-password")
while true
  messenger.received_messages do |msg|
    user = User.find_by_im_name(msg.from)
    if user
      case msg.body
      when /^help /i
        messenger.deliver(msg.from, "Valid commands are......")  
      when /^status /i
        user.status = msg.body.sub(/^status /i)
        user.save
        messenger.deliver(msg.from, "Thanks #{user.first_name}, your status has been updated")  
      when /^balance\?/i
        messenger.deliver(msg.from, "#{user.first_name}, your current remaining balance is #{user.remaining_balance}")  
      else
        messenger.deliver(msg.from, "Sorry #{user.first_name}, I didn't understand that. Message me with 'help' for a list of commands")  
      end
    else
      messenger.deliver(msg.from, "Sorry, but we've not got this account registered on our system. Sign-up or update your details at http://www.mysite.com/")  
    end    
  end  
  sleep 2  
end

The easiest way to have this run now is to call it via script/runner so that we can get access to our ActiveRecord models and interface with the underlying data. All you need to do is kick this off with an init script, setup something like god for process monitoring to ensure it stays alive and keeps within some same memory and CPU constraints.

Ruby and Microformats

Thu, 19 Jun 2008 09:00:00 GMT

Heard about microformats ? Want to know how to consume them within your rails app? It's a suprisingly simple task. But why would you bother? Well lets take a somewhat practical example. What if on your site you wanted to include a list of upcoming events that you were either attending or might be of use to your users? Say for example, events in the UK for the next 30 days

The first thing you need to do, is go grab mofo

sudo gem install mofo

And now include it in your ruby/ruby on rails application:

require 'mofo'

Next, find the URL of the page you want to consume that is exposing it's data as a microformat (read the microformats website for various formats that are available and details on how they are implemented). And within our controller method, we will go and fetch the appropriate information from Upcoming

@events = hCalendar.find 'http://upcoming.yahoo.com/search/?quick_date=next_7_days&search_placeid=DevLebebApj4RVbtaQ&rt=0'

And within our view we could do something like the following (ignore the lack of partials and other useful things, this is just for demonstration purposes):

<ol>
<% @events.each do |event| %>
  <li>
    <h1><a href="<%event.url%>"><%=event.summary%></a></h1>
    <p><%=event.location%></p>
    <p><%=event.dtstart.strftime("%d %B")%></p>
  </li>
<% end %>
</ol>

Of course, being good little net citizens we should probably also make the list into a microformat too just in case someone else wants to try and consume it from our site, so the addition of a few class attributes and a title tag or two will sort that out:

<ol class="vcalendar">
<% @events.each do |event| %>
  <li class="vevent">
    <h1><a href="<%event.url%>" class="url summary"><%=event.summary%></a></h1>
    <p class="location"><%=event.location%></p>
    <p class="dtstart" title="<%=event.dtstart.strftime("%Y-%m-%d")%>"><%=event.dtstart.strftime("%d %B")%></p>
  </li>
<% end %>
</ol>

Keep in mind, the current implementation will send off a request to upcoming.yahoo.com every time someone loads your page and could seriously harm performance, I'd recommend caching the results either to file or directly into a model/table and then having a process that regularly checks and updates the data at a certain interval.

The possibilities here and endless, and you don't have to consume and re-display but you could create your own mashup for your personal use. Many shopping sites now include review information as a microformat, TripIt makes all of my itineraries available the same way. My recent interest has been spurred on by a site I'm working on where most of the data will be available as a microformat, hopefully letting 3rd parties develop uses for the site that I'd never imagined.

Bulk insertion of data with ActiveRecord

Wed, 18 Jun 2008 09:00:00 GMT

Apologies to all for the extended hiatus, I was in the US for a month on holiday; and then I just got lazy. Anyway, I'm back and plugging someone else's extension. I came across ActiveRecord Extension last night, and was quite frankly dumbfounded that 1) I hadn't come across it before, and 2) More people weren't using it. In short, it will let you use ActiveRecord to batch insert multiple rows in a single query, rather than looping over the Model#create method and getting all the overhead that comes with it (and the hundred insert statements).

Zach Dennis has written a great little plugin called ActiveRecord::Extensions which makes the bulk inserts almost as painless as your regular create.

First, install the gem:

sudo gem install ar-extensions

Include it in your app (you could put it in environment.rb if you're going to use it a lot, for now I'll place it in my model definition in user.rb:

require 'ar-extensions'
class User
end

Then to use it, it really couldn't be more straightforward. Use the following in your controller or rake task to import your data:

fields = [:first_name, :last_name, :email]
data = [["glenn", "gillen", "foo@bar.com"],
        ["john", "jones", "jim@bar.com"],
        ["steve", "smith", "bar@foo.com"]]

User.import fields, data

And we're done, 3 new rows have been inserted into the users table, with just the single query. Any more questions, the RDocs are here

Monitoring Backgroundrb with God

Wed, 09 Apr 2008 09:00:00 GMT

I mentioned yesterday that I'd elaborate on how to extend our app.god config so that god process monitoring automatically becomes aware of our changes. Well for those that can't wait, Graham who I pair with on the majority of my work beat me to the punch with with his post on Monitoring BackgroundRB with God. And at the end of the day English is his native language, so it's probably an easier read (I speak Australian).

Letting nginx automatically detect new rails apps

Tue, 08 Apr 2008 09:00:00 GMT

Following on from my post yesterday explaining how to let god automatically detect any new rails apps is few capistrano and config tweaks. This time the changes will let nginx use the same config file we created to so that setting up a new upstream and rails config is as simple as a call to deploy:setup in capistrano.

Make sure you've setup app.god

These changes have a dependency on the changes from my previous post, so be sure to follow the steps on creating app.god to dynamically configure god for rails

Create a new nginx task for capistrano

Within your deploy.rb you want to insert this snippet:

namespace :deploy do

  after "deploy:setup", "deploy:nginx:setup" 

  namespace :nginx do
    desc 'Setup nginx upstream config'
    task :setup, :roles=>:app do
      @apps = []
      eval(open("./config/app.god").read)
      nginx_upstream_config = []
      @apps.each do |app|
        upstream_name = app.class.to_s.downcase.gsub(/.*::/,"")
        nginx_upstream_config << "upstream #{upstream_name} {\n" 
        app.ports.each do |port|
          nginx_upstream_config << "\tserver 127.0.0.1:#{port};\n" 
        end
        nginx_upstream_config << "}\n" 

        nginx_upstream_config << <<-EOF
        server {
          listen 80;
          client_max_body_size 50M;
          server_name #{app.server_names};
          root #{deploy_to}/current/public;
          access_log  /var/log/nginx.#{application}.access.log  main;

          if (-f $document_root/system/maintenance.html) {
            rewrite  ^(.*)$  /system/maintenance.html last;
            break;
          }

          location / {
            proxy_set_header  X-Real-IP  $remote_addr;
            proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Host $http_host;
            proxy_redirect false;
            proxy_max_temp_file_size 0;

            if (-f $request_filename) { 
              break; 
            }

            if (-f $request_filename/index.html) {
              rewrite (.*) $1/index.html break;
            }

            if (-f $request_filename.html) {
              rewrite (.*) $1.html break;
            }

            if (!-f $request_filename) {
              proxy_pass http://#{upstream_name};
              break;
            }
          }

          error_page   500 502 503 504  /500.html;
          location = /500.html {
            root   #{deploy_to}/current/public;
          }
        }        
        EOF

      end

      put nginx_upstream_config, "#{shared_path}/config/nginx.conf" 
      sudo "ln -sf #{shared_path}/config/nginx.conf #{deploy_to}/current/nginx.conf" 
      sudo "mkdir -p /etc/nginx" 
      sudo "ln -sf #{deploy_to}/current/nginx.conf /etc/nginx/#{application}.conf" 
      sudo "/etc/init.d/nginx restart" 
    end
  end

end

I'll run through what is happening here. Within deploy:nginx:setup we first instantiate the @apps variable, so that the app.god we created previously has an instance to insert itself into. We then read in and execute app.god, which means @apps will be populated with our application now.

Now for each application (why would we have more than one? I'll explain at the end) we loop over and create an nginx upstream definition, with each of the ports specified within app.god. Next is to create the standard nginx/rails config (feel free to change to suit your own needs), push it up to the server, and then link it into /etc/nginx/application_name.conf.

Required changes in nginx

The only thing you need to do within your nginx.conf file is add the following line at the end of your http service definition:

include /etc/nginx/*.conf;

And now, when you run:

cap deploy:setup

Capistrano will not just setup the rails directory structures, but will also update the nginx config to make it aware of your new application. It also means that your application specific nginx config now resides in your rails directory structure. Should you decide to take the application off the server, the nginx config disappears when the app does.

So why do you use @apps to specify the application?

Well with the god config from the previous post, god is dealing with every rails application on the server and stores the list of them within @apps. However, with this in most cases we are only dealing with one (the app we are deploying) so it's somewhat redundant. I say in most cases because there have been times when we've deployed the same application twice on the same server, running on different ports and accessible from different URLs. This meant we could do the following in app.god:

class Rubypond

  def initialize
    @ports = [8061, 8062, 8063]
  end
end

@apps << Rubypond.new

class RubypondLabs
  attr_reader :name, :ports, :prefix

  def initialize
    @name = "jetstream" 
    @ports = [8071, 8072, 8073, 8074]
    @prefix = "/labs" 
  end
end

@apps << RubypondLabs.new

Next up, how to extend what we've done so far to manage backgroundrb for us too.

Rails god config

Mon, 07 Apr 2008 09:00:00 GMT

If you're anything like me, you're not a big one for shelling out needless money to host the various development sites your working on. That inevitably means you'll end up with a whole host of small applications running on the one server, each needing a mongrel or two, and of course you want to make sure they are up and running. Thanks to the wonder that is ruby (and god) we can easily set it up to automatically become aware of any new apps we deploy via capistrano.

Enter app.god

Don't stress if you've never heard of app.god, it's a convention we've taken on where I'm currently working so chances are it doesn't exist outside my own little world. Here's what it looks like

class Rubypond
  attr_reader :ports, :server_names

  def initialize
    @ports = [5000, 5001]
    @server_names = "www.rubypond.com rubypond.com" 
  end
end

@apps << Rubypond.new

What we do is define a new class named after whatever we want to call our application, and then create a new instance off it in an array names @apps. Next step, making god aware of these config files.

Setting up god.conf

If we are going to be running multiple rails apps on the one server, there is going to be a lot of duplication with the config files. Life will be much easy if we extract the common bits into some methods that we can call at the appropriate time to create our god watch:

def configure_mongrel(w, rails_root,  app_name, port)
  w.uid = "mongrel" 
  w.gid = "webserver" 
  w.name = "#{app_name}-#{port}" 
  w.group = "#{app_name}" 
  w.interval = 30.seconds
  w.start = "mongrel_rails start -c #{rails_root} -p #{port} -P #{rails_root}/log/mongrel.#{port}.pid -e production -d" 
  w.stop = "mongrel_rails stop -P #{rails_root}/log/mongrel.#{port}.pid" 
  w.restart = "mongrel_rails restart -P #{rails_root}/log/mongrel.#{port}.pid" 
  w.pid_file = "#{rails_root}/log/mongrel.#{port}.pid" 

  w.behavior(:clean_pid_file)
end

def start_if_not_running(w)
  w.start_if do |start|
    start.condition(:process_running) do |c|
      c.interval = 5.seconds
      c.running = false
    end
  end
end

def restart_if_resource_hog(w)
  w.restart_if do |restart|
    restart.condition(:memory_usage) do |c|
      c.notify = "glenn" 
      c.above = 150.megabytes
      c.times = [3,5]
    end

    restart.condition(:cpu_usage) do |c|
      c.notify = "glenn" 
      c.above = 50.percent
      c.times = 5
    end
  end
end

def monitor_lifecycle(w)
  w.lifecycle do |on|
    on.condition(:flapping) do |c|
      c.notify = "glenn" 
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 5.minutes
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
    end
  end
end

def monitor_rails_app(w)
  start_if_not_running(w)
  restart_if_resource_hog(w)
  monitor_lifecycle(w)
end

Letting god know about our rails apps

Next thing we want god to know about all our rails apps, we can use a standard ruby trick to traverse the appropriate directory tree for find our app.god configs, and then load them in using the ruby load command:

@apps = []
Dir["/var/www/apps/*/current/config/*.god"].each { |config| load config}

And because of the way we've instantiated our classes in each config, @apps will now be an array of applications we want to configure. So we can iterate over each, and plug in the appropriate values:

@apps.each do |app|
  app_name = app.class.name.downcase
  rails_root = "/var/www/apps/#{app_name}/current" 

  app.ports.each do |port|
    God.watch do |w|
      configure_mongrel(w, rails_root, app_name, port)
      monitor_rails_app(w)
    end
  end
end

Now all you need to do when you deploy a new application is make sure you have the config/app.god file and restart your god instance, and it will automatically monitor your new application. Sweet! Next up I'll run through a capistrano task to complete the deal by using the same config to setup nginx. So, we may as well seal the deal by ensuring that happens in capistrano (in your deploy.rb file):

after "deploy:setup", "deploy:god:restart" 

namespace :deploy do

  namespace :god do

    task :restart, :roles=>:app do
      sudo "/etc/init.d/god restart" 
    end

  end
end

Using rspec have_tag

Mon, 31 Mar 2008 09:00:00 GMT

Not everyone realises that that rspec havetag assertion is simply a wrapper for the Test::Unit assertselect. And even if you do, do you really know how powerful a tool it is? I’ve seen many tests and specs where people jump through a whole heap of hoops to check that the desired element exists, well no more! Fresh for today, a quick run down on some of the most useful and powerful selectors you can use with havetag (and by default, assertselect too).

Basic rspec have_tag usage

Alright, so the most basic of tests is to see if an element exists:

response.should have_tag("li")

Here we aren’t fussy, so long as any <li> element exists the assertion will pass. We can be a little more specific by also requiring that <li> tag to contain certain text:

response.should have_tag("li", :text => "I'm a list item")

Now it will match <li>I’m a list item</li>. And taking baby steps again, we can ensure that the list item has been given a specific class, ID, or both with one of the following:

response.should have_tag("li.my_class", :text => "I'm a list item")
response.should have_tag("li#my_element_id", :text => "I'm a list item")
response.should have_tag("li#my_element_id.my_class", :text => "I'm a list item")

You can essentially reference the elements exactly as you would in a CSS file.

Matching element attributes with have_tag

What if you want to ensure that something other than a class or ID exists on the element? Well again just like in CSS have_tag in rspec will let you target elements containing attributes:

response.should have_tag("img[alt=My accessible text]")

This will check that a tag like <img alt=”My accessible text”> exists somewhere in the result rendered by you view. You can extend this further with wildcard and pattern matchers on the attributes

response.should have_tag("img[alt~=readable accessible usable]") # Match any of these words
response.should have_tag("img[alt^=My]") # Match attribute beginning with "My"
response.should have_tag("img[alt$=text]") # Match attribute ending with "text"
response.should have_tag("img[alt*=essibl]") # Match "essibl" anywhere in attribute
response.should have_tag("img[alt]") # Match any img element with alt attribute

Advanced rspec have_tag CSS selectors

Taking it up a notch are some of the more advanced CSS selectors. In reality, I rarely (if ever) need to use these, but they’re handy to have for those tricky situations where you’d otherwise have to construct a loop to go through all the element manually. Assume we have a list of elements like this:

<ul>
    <li>I'm element 1</li>
    <li>I'm element 2</li>
    <li>I'm element 3</li>
    <li>I'm element 4</li>
    <li>I'm element 5</li>
  </ul>

We could use the following assertions to get the respective elements based on their order:

response.should have_tag("li:nth-child(2)", :text => "I'm element 2")
response.should have_tag("li:nth-last-child(2)", :text => "I'm element 4")
response.should have_tag("ul:first-child", :text => "I'm element 1")
response.should have_tag("ul:last-child", :text => "I'm element 5")

You can take a similar approach to retrieve elements based on their type. Consider the following chunk of HTML:

<div>
    <h1>My big heading!</h1>
    <p>I've got some paragraph text</p>
    <h2>My smaller heading...</h2>
    <p>I'm going to describe the list below</p>
    <ul>
      <li>I'm list 1</li>
      <li>I'm list 2</li>
      <li>I'm list 3</li>
    </ul>
  </div>

and then the have_tag selectors to get those elements:

response.should have_tag("p:nth-of-type(2)", :text => "I'm going to describe the list below")
response.should have_tag("li:nth-last-of-type(2)", :text => "I'm list 2")
response.should have_tag("p:first-of-type", :text => "I've got some paragraph text")
response.should have_tag("p:last-of-type", :text => "I'm going to describe the list below")

Using formulas in nth- specified selectors

As you see, we can target an element that is a specific child in the order of the DOM hierarchy. What about testing the following:

<ul>
    <li class="highlight">I'm element 1</li>
    <li>I'm element 2</li>
    <li>I'm element 3</li>
    <li class="highlight">I'm element 4</li>
    <li>I'm element 5</li>
    <li>I'm element 6</li>
    <li class="highlight">I'm element 7</li>
    <li>I'm element 8</li>
  </ul>

How would we test that all the appropriate elements have had their additional class applied? Well the easiest way would be to provide a formula to the nth- selector types.

response.should have_tag("li:nth-of-type(3n+1).highlight", :count => 3)

That will ensure that every third <li> element, starting at number 1, will have the highlight class applied and that we return exactly 3 matches.

Asserting the expected number of matches with rspec have_tag ##

As you saw in the previous example, it’s quite simple to specify exactly how many matches should be returned. But what if you don’t know? (note: I can’t think of a valid reason why you wouldn’t if you are mocking out your tests correctly, but I digress) Then you can specify an upper or lower limit or a range:

response.should have_tag("li:nth-of-type(3n+1).highlight", :maximum => 4)
response.should have_tag("li:nth-of-type(3n+1).highlight", :minimum => 1)
response.should have_tag("li:nth-of-type(3n+1).highlight", 1..4)

Iterating over child elements in a block ##

Now imagine you have a page with more than one form on it. One for posting comments to a blog, one for logging in maybe? We want to test that the login form exists (and is going to the correct path), and that there are input elements for username and password within that form. Check this out:

response.should have_tag "form[action=/sessions]" do
  with_tag "input[type=text][name=username]"
  with_tag "input[type=password][name=password]"
end

Neat eh?

Moving over to Git

Fri, 28 Mar 2008 09:00:00 GMT

So you’ve seen that Rails/Capistrano now natively supports Git? You’ve bearing hearing all the hype about it but don’t really know what it is, how to use it, or how to install it. But like most, you know it was invented by that Linus guy that invented Linux, so it can’t be all bad. Well then come jump on board!

What is Git?

Git is a distributed version control system (you can skip this bit if you’ve used mercurial). If you can imagine SVN, but without the need to check back to a central server you’re probably about half way there. Without the concept of a central server, it means I can check out code and develop until my heart is content on my laptop, while in transit, without an internet connection. While SVN wouldn’t stop me from doing this it did prevent me from committing any check-ins during this period. That’s problematic if I’ve a particularly long trip (on a plane) as I then end up checking in a large chunk of functionality in one single big commit and I wont be able to undo just a portion of it. It’s particularly good with a distributed team (which often happens these days with coders all around the world) as I can check out, check in, and not fear that I’m occasionally breaking someone else’s code until it comes time to bundle up all my work and submit it to the core code base. I get all the benefits of tiny atomic commits so that I can easily undo just a small change, and none of the failings of running your own SVN repository as a silo.

That’s because, while there is no dependency on a central server, you can commit and checkout to and from a central Git server whenever you like. Even better, you can commit and checkout directly with a colleague or any other user who has pulled a copy of the code and bypass the server altogether (e.g., two remote developers working on the same code who don’t want to disrupt the central repo just yet).

Installing Git on linux

If you’re on a linux distro you may very well find you already have Git installed. If you don’t, it should be available as a package:

sudo yum install git-core

Installing Git on Mac OS X (or installing Git from source)

For the Mac OS X users out there it will probably be easiest to install it from source. The following commands should get you there:

wget http://kernel.org/pub/software/scm/git/git-1.5.4.tar.bz2
wget http://www.kernel.org/pub/software/scm/git/git-manpages-1.5.4.tar.bz2
tar xjvf git-1.5.4.tar.bz2
cd git-1.5.4
./configure
make
sudo make install
cd ..
sudo tar xjvf git-manpages-1.5.4.tar.bz2 -C /usr/local/share/man

If you’re so inclined, now would probably be a good time to change to default settings to get coloured output and a whole host of nifty shortcuts and improvements. Check out this git quickstart guide for a list of them

Using Git

If you’ve used mercurial before this should all seem like second nature. If you’re used to SVN, then there is only a minor learning curve. The thing you need to keep in mind is that your local working copy of the checkout, is also a server. So when you do a commit, you are really only saving it to your local machine. Likewise, if you did a checkout, you’re doing it on the local machine. If you want to update your local copy from a remote server like you would in SVN you need to pull, to send your updates somewhere else you need to push. Let’s quickly go through how you’d checkout a working copy, and make some changes to it. Firstly, you need to create a local clone of the repository:

git clone git://domain.com/path/to/git/repository my_local_dir

This will create a new directory called mylocaldir and pull in a copy of the the repository from the remote server. Next you make some changes locally to the files and try to check them in:

cd my_local_dir
git commit -a

This will commit all local changes, to your local server. Now it’s time to push those changes back upstream:

git push

But what if we want to check if there are any changes from the remote server? Well we just pull them in:

git pull

Creating a remote Git repository

That’s all well and good if you’ve got a repository already setup to check out from. But what if you want to create your own. Well you can go with a hosted option from providers like GitHub or Gitorious or if you are the DIY type, host your own. Go to the remote server location where you want to have your repo and:

mkdir myrepo.git
cd myrepo.git
git --bare init

Now obviously we’ve created a directory and initialised a new Git repository within it here. What the bare option does is inform Git that this won’t be a working copy. We don’t need to waste disk with the actual files here, we just need to version control information and the changes (delta) between files.

You’ve probably got a local codebase that you now want to check into your brand new remote repo so you can code away, check in, remotely back it up, and have others contribute. So locally:

cd myapp
git init
git add .
git commit -m "Initial check in"

What we’ve done here is initialise the local Git repository in our myapp directory, add all the files in, and then commit the new files. All that is left to do is let this repository know about a remote host, and then push our changes there:

git remote add origin ssh://remote.myserver.com/path/to/repo/myapp.git
git push origin master

This lets the local repository know to treat the remote server as the origin of the source, followed by a command to push our local master copy up to the origin server. From now on, you can send any changes to the server with a simple:

git push

And that’s it! Following shortly, how to port your existing SVN projects over and/or peacefully let Git and SVN projects co-exist.

17 Rails plug-ins you should be using

Mon, 18 Feb 2008 09:00:00 GMT

So you're pretty happy with your rails-fu, but are you using all the plug-ins you could be to make life easier for you and your users? The following is a list of plugins I use on almost all, or if not the vast majority of, projects I work on these days. As soon as I've setup my rails directory tree, I install these bad boys too.

acts_as_commentable

The quickest and easiest way to be able to add comments to an ActiveRecord model. Don't think just comments in the 'blog' sense though, I've used it in numerous places to act as a means of users providing feedback and notes on specific items.

acts_as_ferret

Ferret is a ruby port of the Apache/Java Lucene project which enables quick full-text searching. acts_as_ferret is simply a way of integrating that into rails easily, allowing you to do a full text scan across one or multiple ActiveRecord models in just a few lines of code.

acts_as_taggable_on_steroids

Taggable on steroids is a big step up from the original taggable plugin from DHH. It lets you do tagging like you see on most social networking sites simply, but will also calculate the cloud size of a specific term for you so you can highlight the most/least popular terms (like I do on this site).

annotate_models

There are quite a few times where I'm needing to debug something on a server, or need to explain to someone else what an underlying table for an ActiveRecord model looks like and opening up a SQL session just seems overkill. Step in annotate_models, a handy rake task that will put the schema in a comment block at the top of your model. No more voodoo about where attributes are coming from, or confusion over what the underlying structure is.

attachment_fu

Rick Olson's upgrade to the original acts_as_attachment is a must have. It will handle image uploads and cropping/re-sizing, storage in a database/file system/AmazonS3 oh, and it works on more than just images! Upload those PDF and other attachments until your heart is content.

autotest

I can't imagine life without autotest hooked up with growl notifications on my macbook. You define your specs and follow your TDD/BDD approach to development and autotest will automatically keep track of what files you are editing and what tests need to be re-run. No need to wait for the whole test stack to execute to see the impact of one change, as soon as you hit save autotest will run just the required tests. Neat!

backgroundrb

There is nothing worse than tying up a mongrel instance with a long running process, especially one you don't necessarily need an instant user response on (maybe resizing an image? generating a report?). So push the task off the backgroundrb and let it take it's sweet time without affecting the rest of the site.

exception_logger

Exception logger will trap any exceptions that are raised and store them to a database, and then provides a web interface for you to manage them. Much easier than trawling log files for problems.

haml

Hampton has made HTML fun again, and helped make it more structured and semantically correct in the process. HAML is a killer templating system to replace (or run along side if you wish) ERB. It really is worth a look, although it's benefit didn't fully smack me in the face until I ported an old project across to it.

lowpro

Dan Webb wrote lowpro as a way of unobtrusively adding javascript into your rails applications. It wasn't until reading through his blog I realised how terrible (in terms of backward compatibility and separation of presentation and code) the standard rails way of doing things was.

make_resourceful

Hampton strikes again with a plug-in that could cut your RESTful controllers down to as little as 3 lines in some instances. I posted about how to simplify RESTful controllers last year

minus r

Dan Webb's back with more javascript help. This time, making RJS templates even easier by allowing you to enter straight javascript into them and not a bunch of ruby that has to be parsed.

query_analyzer

So how exactly to you know what to do to improve the performance of your rails up? query_analyzer certainly puts you on the right track for anything ActiveRecord related by outputting the sql EXPLAIN results into your log file. Then you can view the query execution path and work out what would be best served by having indexes added to them.

query_trace

Knowing how to improve your queries is one thing, but how about you stop them being called altogether? Sounds like a useful task, until you try and identify exactly where that 'select * from users' is being called. Enter query_trace, to tell you where the problem is.

restful authentication

The quickest and easiest way to add in users, signup/login capabilities, and notification/confirmation emails. If you've not heard of this and used it you must have been living under a rock for 18 months.

rspec and rspec_on_rails

The new fashionable kid on the block from a testing perspective. I love the way the BDD approach generates tests that my non-IT counterparts can read, understand, and agree that it meets their requirements.

will_paginate

My new favourite pagination plugin out of the many trying to replace the one that has been removed from Rails 2.0. It's just so simple to both implement and test.

Improved timezones in Rails 2.1

Fri, 08 Feb 2008 09:00:00 GMT

The latest release of rails has made the handling of timezones in rails even easier. No longer do you need to install a couple of plugins, although you still need the tzinfo gem.

With the new baked in support for timezones in rails you can ignore the previous article on how to support timezones in rails 1.2.x. Now all you need to do is set the desired timezone you want to be working in:

Time.zone = "Pacific Time (US & Canada)"

Behind the scenes the date will be saved as UTC, but the appropriate timezone conversion will happen for you whenever you output a date or datetime.

> Time.zone = "Pacific Time (US &amp; Canada)" 
> @post = Post.find :first
> @post.created_at
=> Fri, 08 Feb 2008 09:00:00 PST -08:00
> @post.created_at = Time.utc 2008, 1, 1
> @post.created_at
=> Mon, 31 Dec 2007 16:00:00 PST -08:00

It also makes it a breeze to display timezones that are customised to your end-users locality with a before filter:

class ApplicationsController < ApplicationController
  before_filter :set_timezone

  private
    def set_timezone
      Time.zone = current_user.time_zone || "Pacific Time (US & Canada)" 
    end
end

Alternatively you can specify a default timezone in your environment.rb:

Rails::Initializer.run do |config|
  config.time_zone = "Pacific Time (US & Canada)" 
end

Oh and of course Time.now will still return the time according to the server, if you want the current time in the user's local timezone then:

Time.zone.now

Improving rails app and mongrel performance with Thin

Wed, 06 Feb 2008 09:00:00 GMT

Thin is a replacement webserver for mongrel* that further improves the already lightening performance of mongrel. Installation is a breeze, performance is fantastic, but it's still in alpha so use with caution.

As you can see in the chart above (taken from the Thin website, I've not had a chance to verify independently), the improvement while handling 10 concurrent connections isn't exactly an order of magnitude larger. But you'll see that as concurrency increases, performance holds steady unlike using mongrel. This is because it's using EventMachine network I/O library to handle incoming requests much quicker than mongrel can. But mongrel is super quick at parsing HTML and Marc recognised that, so Thin still uses mongrel to handle that aspect (and uses Rack to talk to it).

Installation

It's a gem, so it's easy:

sudo gem install thin

Configuration

If you want to just get straight into it, you can start up a thin server with the following:

cd /to/my_rails_app
thin start

If however you want to setup a cluster similar to you have with your mongrel cluster, and then run those instances so it will continue to work with your upstream config in nginx do the following:

thin config -C config/thin.yml --servers 3 --port 5000 --chdir
thin start -C config/thin.yml

That will create a config file with 3 instances starting at port 5000.

Improving Thin performance with unix sockets

But we can do one better than that. Thin also supports a feature not often seen in the ruby webserver world, but one you've probably seen with MySQL. It can enable access to the webserver via a unix socket. What's the point? Well it means that communication doesn't have to got back through the internal ethernet interface with the associated TCP overhead. And thankfully, nginx has support for accessing upstreams via a socket. You can start a server as sockets with the following command:

thin start --server 3 --socket /tmp/thin.sock

That will create 3 sockets in /tmp/, named thin.0.sock, thin.1.sock, and thin.2.sock. All that is left to do is update your nginx.conf to use these as your upstream:

upstream  my_server {
  server   unix:/tmp/thin.0.sock;
  server   unix:/tmp/thin.1.sock;
  server   unix:/tmp/thin.2.sock;
}

Setting up nginx, ssl, and virtual hosts

Fri, 01 Feb 2008 09:00:00 GMT

installing nginx

To install nginx you have a few options depending on the flavour of operating system you are running. You can either install it via a package manager, or download the source and compile it yourself. Using the package manager will mean you will get the latest known stable version for your OS, and have to worry less about working out dependencies or conflicts. However, you may be a version or two behind the official release.

installing nginx via yum/apt-get/rpm

If you’ve got a preferred package manager on your system then it should be as easy as one of the following (choose whichever is appropriate): sudo yum install nginx sudo apt-get install nginx

Alternatively you can download the rpm and install using that:

wget ftp://ftp.univie.ac.at/systems/linux/fedora/updates/9/i386.newkey/nginx-0.6.32-1.fc9.i386.rpm
rpm -Uvh nginx-0.6.32-1.fc9.i386.rpm

installing nginx via source

If none of the above is suitable, then you’re off to compile it on your own. The following should do the trick and include the libraries/packages we need:

wget http://sysoev.ru/nginx/nginx-0.7.2.tar.gz
tar zxvf nginx-0.7.2.tar.gz
cd nginx-0.7.2
./configure --prefix=/usr/local/nginx --sbin-path=/usr/local/sbin --with-debug --with-http_ssl_module
make
sudo make install
sudo nginx

configuring an nginx virtual host

Now that it’s up and running, time to configure nginx to know about a virtual host (or two) on your system. For now, lets just serve some static content so we get a feel for the config files:

sudo nano /usr/local/nginx/conf/nginx.conf

You’ll see the default nginx config file, in recent versions it’s fairly well commented to help you understand what is going on. Scroll down and find the definition that looks like this:

server {
      listen       80;
      server_name  _;

      #charset koi8-r;

      #access_log  logs/host.access.log  main;

      location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
      }

      error_page  404              /404.html;
      location = /404.html {
        root   /usr/share/nginx/html;
      }
    }

I wont go into too much detail here, the important thing to note is where the root location is. Here’s is specified as /usr/share/nginx/html, which means a request to http://yoursite.com/directory/file.html is going to try and serve the file on your server from /usr/share/nginx/html/directory/file.html. To create a new virtual host lets change this definition to look more like this:

server {
      listen       80;
      server_name  www.yoursite.com;
      root   /var/www/yoursite.com/html;

      location / {                
        index  index.html index.htm;
      }
    }

Now all requests coming in to http://www.yoursite.com will be handled by this definition, but only requests for www.yoursite.com. If you want to handle another domain within this definition you can just put an additional server name in like this:

server_name  www.yoursite.com www.othersite.com;

Alternatively, if you want to have www.othersite.com serve up content from a different location you would just add another server definition:

server {
      listen       80;
      server_name  www.yoursite.com;
      root   /var/www/yoursite.com/html;

      location / {                
        index  index.html index.htm;
      }
    }

    server {
      listen       80;
      server_name  www.othersite.com;
      root   /var/www/othersite.com/html;

      location / {                
        index  index.html index.htm;
      }
    }

configuring nginx for your rails app

The hosts are setup, now we can have one of them start serving some rails apps. Lets change the virtual host definition we had above for www.yoursite.com to look like the following:

server {
      listen 80;
      server_name  www.yoursite.com;
      root   /var/www/apps/yoursite/current/public;

      # Set the max size for file uploads to 50Mb
      client_max_body_size 50M;

      # this rewrites all the requests to the maintenance.html
      # page if it exists in the doc root. This is for capistrano's
      # disable web task
      if (-f $document_root/system/maintenance.html) {
        rewrite  ^(.*)$  /system/maintenance.html last;
        break;
      }

      location / {
        # needed to forward user's IP address to rails
        proxy_set_header  X-Real-IP  $remote_addr;

        # needed for HTTPS
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect false;
        proxy_max_temp_file_size 0;

        # If the file exists as a static file serve it directly without
        # running all the other rewrite tests on it
        if (-f $request_filename) { 
          break; 
        }

        # check for index.html for directory index
        # if its there on the filesystem then rewite 
        # the url to add /index.html to the end of it
        # and then break to send it to the next config rules.
        if (-f $request_filename/index.html) {
          rewrite (.*) $1/index.html break;
        }

        # this is the meat of the rails page caching config
        # it adds .html to the end of the url and then checks
        # the filesystem for that file. If it exists, then we
        # rewite the url to have explicit .html on the end 
        # and then send it on its way to the next config rule.
        # if there is no file on the fs then it sets all the 
        # necessary headers and proxies to our upstream mongrels
        if (-f $request_filename.html) {
          rewrite (.*) $1.html break;
        }

        if (!-f $request_filename) {
          proxy_pass http://railsapp;
          break;
        }
      }
    }

What we have here is a series of rules that match the requested file against some regular expressions and underlying filesystem files. To tell nginx to serve any content that it finds matching the full requested path, starting in the directory defined as root we do the following:

if (-f $request_filename) { 
      break; 
    }
    if (-f $request_filename/index.html) {
      rewrite (.*) $1/index.html break;
    }
    if (-f $request_filename.html) {
      rewrite (.*) $1.html break;
    }

If that doesn’t work, we assume it need to be served by the rails app so we send it through using this:

if (!-f $request_filename) {
      proxy_pass http://railsapp;
      break;
    }

But what does http://railsapp mean? No such server exists! Within nginx we need to define an upstream proxy. This will allow us to wrap all our mongrels up within a single definition, and nginx can take care of sending it through to the instances in the back end. To define the railsapp upstream mongrels we add the following outside our server definition:

upstream railsapp {
      server 127.0.0.1:3000;
      server 127.0.0.1:3001;
      server 127.0.0.1:3002;
    }

setting up your nginx ssl server

What if we want to serve out site over HTTPS for security reasons? No problem! We just add a server definition that listens on the correct port (that’s 443 for those of you playing along at home). So our definition would look like this:

server {
      listen 443;
      server_name  www.yoursite.com;
      root   /var/www/apps/yoursite/current/public;

      # Rest of config here...

redirecting all traffic to your ssl server

To be extra secure, we’ve decided that we want all traffic to be handled over SSL. Well we can use the ssl_requirement plugin in our rails app to completely prevent non-HTTPS access, just incase. We can also add this server definition to our nginx config to re-direct anybody who mistakenly links to the plain old HTTP address:

server {
      listen 80;
      server_name www.yoursite.com;
      rewrite ^/(.*) https://yoursite.com/$1 permanent;
    }

setting nginx error pages

Nginx will display some standard error messages should there be a problem, they’re about as attractive as the standard ones that come with rails though. If you want to jazz them up and make the whole experience less jarring for your users then update these lines:

error_page   404 500 502 503 504  /http_error.html;
      location = /http_error.html {
      root   /var/www/errors;
    }

And point it to the file you want to come up for all your errors. If you want to differentiate between various errors then configure each one individually:

error_page   500 502 503 504  /50x.html;
      location = /50x.html {
      root   /var/www/errors;
    }
    error_page   404   /404.html;
      location = /404.html {
      root   /var/www/errors;
    }

directing nginx log output

If you’re running multiple virtual hosts, it will probably pay to give each a log file of it’s own rather than having a single jumbled mess to worry about. Basically any configuration you provide in the http definition you can re-define on a per-server basis within each server definition. So simply add these lines into the relevant virtual host/server definition:

server {
      listen 80;
      server_name  www.yoursite.com;
      root   /var/www/apps/yoursite/current/public;

      access_log  /var/log/nginx/yoursite.access.log  main;
      error_log  /var/log/nginx/yoursite.error.log  debug;

      # Rest of config...
    }

make nginx serve cache content from a custom location

Some people may cache their rail output into a custom directory (and why wouldn’t you, it makes clearing the cache so much easier). If that’s the case, the usual nginx/rails configuration won’t work as expected. Your site will still work, you’ll just not be making the most of letting nginx serve your static content. To resolve it, add these lines in before the section that passes the request to your upstream proxy:

if (-f /cache$request_filename) { 
      rewrite (.*) /cache$1 break;
      break; 
    }

    if (-f /cache$request_filename.html) { 
      rewrite (.*) /cache$1.html break;
      break; 
    }

Where /public/cache is where you are storing all of your cached content. Obviously, alter to fit accordingly.

Free ATOM feed loving

Wed, 23 Jan 2008 09:00:00 GMT

Chris over at Err the Blog have a great article about how you can very quickly and easily create ATOM syndication of your content with a gem called mofo.

What are microformats?

I'll avoid any misrepresentation by taking the definition straight from the microformats site

Designed for humans first and machines second, microformats are a set of simple, open data formats built upon existing and widely adopted standards Basically what it means in practice is that if you've written well structured and semantically correct (X)HTML code, then with the sprinkling of a few additional class definitions or attributes you can extend what you've already got to support microformats.

What are some example microformats?

There are already formats defined for people, places, and business cards, events and calendaring, voting, reviews, friends and connections, and tags to name but just a select few.

Okay, so how do we turn this into a ATOM feed?

Well here is what the entry for this article in the main page listing would currently look like:

<div class="entry">
  <span class="header">
    <h3><a href="/articles/2008/01/23/free-atom-feed-loving/">Free ATOM feed loving</a></h3>
    <p class="info">Posted by glenn gillen on Jan 23, 2008</p>
    <p class="info tag_line">Posted in <a href="/tag/microformats">microformats</a>, <a href="/tag/atom">atom</a></p>
  </span>
  <a href="/articles/2008/01/23/free-atom-feed-loving/"><img alt="" size="preview" src=""/></a>
  <br/>
  <p>Chris over at Err the Blog have a great article about how you can very quickly and easily
     <a href="http://errtheblog.com/posts/78-feeds-for-free">with a gem called mofo.</a></p>
  <a href="/articles/2008/01/23/free-atom-feed-loving/#extended">More...</a>
</div>

And all it would take to turn it into a microformat hEntry is to make it the following:

<div class="entry hentry">
  <span class="header">
    <h3><a href="/articles/2008/01/23/free-atom-feed-loving/" class="entry-title">Free ATOM feed loving</a></h3>
    <p class="info">Posted by <span class="author vcard fn">glenn gillen</span> on <abbr class="updated" title="2008-01-23T09:00:00Z">Jan 23, 2008</abbr></p>
    <p class="info tag_line">Posted in <a href="/tag/microformats">microformats</a>, <a href="/tag/atom">atom</a></p>
  </span>
  <a href="/articles/2008/01/23/free-atom-feed-loving/"><img alt="" size="preview" src=""/></a>
  <br/>
  <p class="entry-content">Chris over at Err the Blog have a great article about how you can very quickly and easily
     <a href="http://errtheblog.com/posts/78-feeds-for-free">with a gem called mofo.</a></p>
  <a href="/articles/2008/01/23/free-atom-feed-loving/#extended">More...</a>
</div>

Just to highlight it incase you missed it. There is a new hentry class on the container div, a new entry-title on the link to the entry, a span defining a vcard around my name, and the readable date has an abbr tag to translate it back into something the computer prefers. And finally, the p containing the content has had an entry-content class applied.

So that takes care of outputting it in as a microformat, but what use is that unless you've got something to consume it? How about we use mofo to render our view, extract the microformats, and then publish them as a feed for inclusion in your favourite reader? Add something like this to your controller:

def index
  @posts = Post.find(:all)
  respond_to do |format|
    format.html 
    format.rss do
      target = url_for(:action => :index, :format => "html")
      render :xml => hEntry.find(target).to_atom(:title => 'whatever')
    end
  end
end

Great! So tell me how to install it already!

Literally a 1-2-3:

sudo gem install mofo
cd rails_app/vendor/plugins
gem unpack mofo

Playing with Merb

Thu, 17 Jan 2008 09:00:00 GMT

So there has been quite a lot of talk of late about merb, the new up and comer vying for the rails crown. It was initially developed by Ezra of EngineYard fame, and it's generally considered that 'They know their stuff'(tm). So what is merb? Is it a viable alternative to rails? How easy/hard is the migration from one to the other?

I figured it was long overdue that I take a look.

What is Merb? How is it different to Ruby on Rails?

Merb is a Rails-inspired MVC framework for web application development. It's major point of difference with Rails is that it's not so prescriptive on what external libraries you have to use. Don't like prototype/scriptaculous? Swap it out for JQuery or Moo. Don't like ActiveRecord? Try DataMapper or Sequel instead. One of the implications of this is that it means that no longer do you have to fire up a cluster of mongrels to compensate for the fact that aspects of rails aren't thread safe.

Getting started with Merb

Installing it is simple as Ezra and the crew have made it available as a gem.

sudo gem install merb

Then to create a new application skeleton and get started it's just:

merb my_app

So far it should seem fairly familiar! But how do we start playing with our database?

Connecting to a database

Well because merb doesn't impose any decisions on what ORM to use, you need to install your own. You can stick with ActiveRecord if you like, but let's try something new.

sudo gem install datamapper merb_datamapper do_mysql

We've just installed the DataMapper ORM, the merb interface, and the mysql drivers. Next thing to do to test connecting to a database is setup the connection, and create a model. Update the config/dependencies.rb file and uncomment the following line:

### Uncomment for DataMapper ORM
use_orm :datamapper

Next we need to create a config/database.yml file which should be nothing shocking to look at:

---
# This is a sample database file for the DataMapper ORM
:development: &defaults
  :adapter: mysql
  :database: myapp_development
  :username: root
  :password:
  :host: localhost

:test:
  <<: *defaults
  :database: myapp_test

:production:
  <<: *defaults
  :database: myapp_production

and now the joy of creating a model:

script/generate model person --skip-migration

Skip migrations? Are you mad? Do I have to go back creating tables myself? Bear with me here, I'll touch on migrations shortly. Next I'll define some properties on my model in apps/models/person.rb:

class Person < DataMapper::Base
  property :name, :string
  property :dob, :date
end

Migrations with DataMapper

So we've planned out our model, added in the properties we need. Now if only we could add/retrieve records to/from a table? Creating the required schema in the database is one line:

rake dm:db:automigrate

That's right, DataMapper has automated migrations. It will infer from your model definitions what fields need to exist and create them as appropriate. It'll also take care of primary and foreign keys too! You need to be careful though, because remove a column from the model definition and the next time you automigrate you'll lose that column in the DB too.

So what about the views? What templating systems do I have?

Okay, so creating models isn't too difficult or different from rails. Thankfully the same is true of views too. You can use erubius if you like, but to keep things as close to my version of 'normal' as possible I'm sticking with HAML. So it's really business as usual on this front. Create your views as you normally would, and it should 'just work'

Watch this space for a deeper end-to-end tutorial or screen cast where I'll have a full basic app. Feel free to shoot through any ideas you have.

Testing web services with rspec

Mon, 07 Jan 2008 09:00:00 GMT

So you've come over to testing with rspec and it first the bill on most occasions. There are still a few areas though, like decent integration testing, where you still fall back to Test::Unit. So what about testing web services? Sure, ActionWebService is no longer part of rails core but a number of my clients have legacy systems we need to talk to, or 3rd party applications that have been built around SOAP. And at the end of the day, they're still a damn sight better than many of the other alternatives even if it's not part of the new church of REST.

So now it's time to combine the best of the old-school with the best of the new, SOAP meets RSpec

Start with a normal rspec controller spec, with two minor changes. Here's an example from a recent project:

require File.dirname(__FILE__) + '/../spec_helper'
require 'action_web_service/test_invoke'

describe MySOAPController do
   
  before do
    @account_xml = <<-EOF
      
        ... some elements here ...
      
    EOF
    
  end
    
  it "should parse account xml and return true if successful" do
    User.should_receive(:create)
    result = invoke :AddAccount, @account_xml
    result.should eql(true)
  end
  
  it "should parse account xml and return false if data is invalid" do
    User.should_not_receive(:create)
    result = invoke :AddAccount, "no xml here!"
    result.should eql(false)
  end
  
end

The only noticeable differences here are that we've got an additional include reference:

require 'action_web_service/test_invoke'

and that rather than raising a get/post/put/delete to retrieve the data, we call invoke against the API and return the result to a local variable to later call assertions on, like so:

result = invoke :AddAccount, @account_xml

The syntax is invoke followed by the method name you wish to call, and then whatever parameters the method is expecting. In this instance it receives an XML string as the only parameter, and is expected to return a boolean response

Unobtrusive javascript with lowpro

Thu, 03 Jan 2008 09:00:00 GMT

Dan Webb's lowpro library has been around for quite a while now, but if the sites I frequent are anything to go by it's adoption is far from widespread. Why? Frankly, I don't know. It's a brilliant library and if you're already using things like prototype and scriptaculous but you're not all over lowpro then there is a good chance you need to be seriously smacked around a bit.

Rails comes with a heap of built-in helpers for doing useful things like autocomplete boxes and inline editing. But at the end of the day, while the implementation from the developers perspective is relatively clean the actual code they generate sucks major ass. If you view source on the output, you'll find that there is lots of javascript now embedded within the HTML document. Not only is the HTML no longer just describing the structure of the page, but in many cases you've created a dependency on the end user to have javascript to be able to use your site. It also means to test your site properly, you have to be using something like selenium as the standard rails test framework(s) won't execute your javascript.

How does lowpro help?

It means you focus on developing a site that works for everyone to begin with. No stinking javascript, no silly business or aesthetic logic being inserted into your HTML. Instead, you have a traditional HTML (and CSS) only site that works for all browsers, all users, all of the time. Then you go about having lowpro add in the logic you require at run-time to improve the user experience for those that support it.

Installing and setting up lowpro

Firstly grab the 0.2.0 release from here and unzip the files into your project. Now this library is dependent on prototype, so if you're using jquery or something else that'll be the topic of a later discussion. Next you have to include prototype and lowpro in your HTML. None of this is rails specific, and none of it is going to involve ruby. It's back to old-skool javascript skills. Here's what you need to include:

<script src="/js/prototype.js" type="text/javascipt"></script>
<script src="/js/lowpro.js" type="text/javascipt"></script>

Using lowpro

So this is obviously the most important bit. Forget everything you used to know about onload, onmouseover, onclick, etc. Now you create a javascript file to contain all the logic you need and who needs to respond to what. Replace <body onload="alert('hello!')"> with:

Event.onReady(function() {
  alert('hello!');
});

The brilliant things about attaching the event in this fashion is that rather than executing after the entire document has finished loading (which can include images and everything), it will fire as soon as the browser has parsed the HTML and knows the document layout (or once the DOM had loaded for those up with the lingo). That means you can have the actions fire as soon as the document has loaded, but before the user sees anything.

But what about those mouse over and mouse click events? You've got a button called 'foo' that you want to say boo when you click on it, likewise you've got all of links with the class of 'bar' that you want to highlight whenever the mouse passes over them. It's as simple as:

Event.observe($('foo'), 'click', function(e) {
    alert('boo!');
});

Event.addBehavior({
    'a.bar:mouseover' : function(e) {
      this.visualEffect('highlight');
    }
});

And there we go, #foo now has an onclick event applied to it. Meanwhile every link with a class of bar has had a mouseover event applied. You can of course nest and stack these events until your heart is content. Kudos once again to Dan for such an awesome library

Touched by god (process monitoring)

Fri, 28 Dec 2007 09:00:00 GMT

The tag ling from the god website simply states 'Like monit, only awesome', and having played with it for a couple of days over the break now I have to agree. Monit was very handy at the time, but I found myself growing increasingly frustrated with it when things wouldn't restart properly (stuck sockets with backgroundrb being an example), and it's lack of logging only compounded the situation.

And then I found god.

All blasphemy aside, this tool really is awesome. The scripts are in ruby so you can be as creative as you want, it has in built support to add in your own behaviour hooks (like a clean_socket action on trying to restart backgroundrb) and best of all it provides incredibly in-depth logging, request-able on a per process basis.

Installing god

It's a gem, so installation couldn't be easier:


sudo gem install god

Configuring god

Next step was to setup a config file. I called mine /etc/god.conf and set it up to dynamically include any thing in /etc/god.d/ like so:


God.load "/etc/god.d/*.god"

For another client who is running a server with multiple applications, we set it up so that the god config could actually be deployed with the rails applications, so the /etc/god.conf file looked like:


God.load "/var/www/apps/*/current/config/*.god"

But now onto the nitty gritty of what is actually in those application specific configs. Here's an example from this site:

app_root = "/var/www/apps/rubypond/current" 

%w{10000 10001}.each do |port|
  God.watch do |w|
    w.name = "rubypond-mongrel-#{port}" 
    w.group ="rubypond" 
    w.uid = "mongrel" 
    w.interval = 30.seconds # default
    w.start = "mongrel_rails start -c #{app_root} -p #{port} \
      -P #{app_root}/log/mongrel.#{port}.pid  -d -e production" 
    w.stop = "mongrel_rails stop -P #{app_root}/log/mongrel.#{port}.pid" 
    w.restart = "mongrel_rails restart -P #{app_root}/log/mongrel.#{port}.pid" 
    w.start_grace = 10.seconds
    w.restart_grace = 10.seconds
    w.pid_file = File.join(app_root, "log/mongrel.#{port}.pid")

    w.behavior(:clean_pid_file)

    w.start_if do |start|
      start.condition(:process_running) do |c|
        c.interval = 5.seconds
        c.running = false
      end
    end

    w.restart_if do |restart|
      restart.condition(:memory_usage) do |c|
        c.above = 100.megabytes
        c.times = [3, 5] # 3 out of 5 intervals
      end

      restart.condition(:cpu_usage) do |c|
        c.above = 50.percent
        c.times = 5
      end
    end

    # lifecycle
    w.lifecycle do |on|
      on.condition(:flapping) do |c|
        c.to_state = [:start, :restart]
        c.times = 5
        c.within = 5.minute
        c.transition = :unmonitored
        c.retry_in = 10.minutes
        c.retry_times = 5
        c.retry_within = 2.hours
      end
    end
  end
end

Hopefully the intention of the config is relatively clear. For the given ports (10000 and 10001) create a new god watch that has these properties and these SLAs. The thing that may need clarifying is what exactly is flapping? Well it's the state god goes into when it tries to start your service, and it instantly terminates for whatever reason. In the example, if these occurs 5 times within 5 minutes then the process will be unmonitored for a further 10 minutes, before trying again. This will go on for 2 hours before finally giving up. Another cool thing is that you don't always need to explicitly provide a restart command, god is intelligent enough to assume that if one doesn't exist it should use start and then stop instead.

Monitoring MySQL with god

But hey, why constrain it just to rails apps? Here's the config to get MySQL into the Garden of Eden:

God.watch do |w|
  w.name = 'mysql'
  w.interval = 30.seconds # default
  w.start = "cd /etc/init.d && ./mysqld start" 
  w.stop = "cd /etc/init.d && ./mysqld start" 
  w.restart = "cd /etc/init.d && ./mysqld restart" 
  w.start_grace = 10.seconds
  w.restart_grace = 10.seconds
  w.pid_file = '/var/run/mysqld/mysqld.pid'
  w.behavior(:clean_pid_file)

  w.start_if do |start|
    start.condition(:process_running) do |c|
      c.interval = 5.seconds
      c.running = false
    end
  end

  # lifecycle
  w.lifecycle do |on|
    on.condition(:flapping) do |c|
      c.to_state = [:start, :restart]
      c.times = 5
      c.within = 5.minute
      c.transition = :unmonitored
      c.retry_in = 10.minutes
      c.retry_times = 5
      c.retry_within = 2.hours
    end
  end
end

Email and notifications

It's all well and good having god keep the system up and accessibly, but it's still nice to know when things have gone wrong so you can try and prevent it from occurring again. You'll need to add some email settings to the top of /etc/god.conf, be sure to change them to something appropriate for your server:

God::Contacts::Email.message_settings = {
  :from => 'god@rubypond-example.com'
}

God::Contacts::Email.server_settings = {
  :address => "localhost",
  :port => 25,
  :domain => "rubypond.com",
  :authentication => :plain,
  :user_name => "glenn",
  :password => "password" 
}

God.contact(:email) do |c|
  c.name = 'glenn'
  c.email = 'glenn@example.com'
end

Then you need to add a notification to the transition state you want to be alerted of:

w.transition(:up, :start) do |on|
  on.condition(:process_exits) do |c|
    c.notify = 'tom'
  end
end

Tada! And there you have it, a whole bunch of god-like goodness. I'll follow up with another article with some more app specific configs, and how to extend with your own behaviours.

A handy ruby utility (belt)

Mon, 17 Dec 2007 09:00:00 GMT

Giles Bowkett has created a very handy gem called utility belt. For those that find themselves forever delving into script/console or IRB, this can make life seem oh-so-fantastic.

I won't bore you with the syntax and full feature list here, as Giles has already extensively done that. But a quick short list of the ones I've already found useful:

Automated indentation
Syntax highlighting
Command line AmazonS3 access
Pastie integration

Quick, go grab it now while stocks last!

New Error Handling in Rails 2.0

Thu, 13 Dec 2007 09:00:00 GMT

Rails 2.0 has added in a very cool way to trap errors without a bunch of messy nested rescue blocks or case statements.

The ticket that added all the magic is here. And it couldn't come too soon as I immediately had a project that needed it. Check out what I was looking at doing:

class AccountsController < ActionController::Base

  def rescue_action_in_public(exception)
    case exception
      when Account::Overdrawn
        flash[:error] = "Sorry the account this transaction would make this account overdrawn" 
        redirect_to account_path(params[:account])
      when Account::Suspended
        flash[:error] = "Sorry the account you are trying to use has been suspended, please contact customer services" 
        redirect_to account_path(params[:account])
      when ActiveRecord::RecordNotFound
        flash[:error] = "No such account could be found" 
        redirect_to user_path(current_user)
      else
        super
    end
  end
end

and now with the new exception handling:

class AccountsController < ActionController::Base
  rescue_from Account::Overdrawn, :with => :overdrawn
  rescue_from Account::Suspended, :with => :suspended
  rescue_from ActiveRecord::RecordNotFound, :with => :not_found

  protected
    def overdrawn
      flash[:error] = "Sorry the account this transaction would make this account overdrawn" 
      redirect_to account_path(params[:account])
    end

    def suspended
      flash[:error] = "Sorry the account you are trying to use has been suspended, please contact customer services" 
      redirect_to account_path(params[:account])
    end

    def not_found
      flash[:error] = "No such account could be found" 
      redirect_to user_path(current_user)
    end
end

woohoo!! Much cleaner and more descriptive of my intentions

Testing XML output in rspec and rails

Mon, 10 Dec 2007 09:00:00 GMT

Have you ever tried testing that you get the expected XML output from your rails respond_to block? Did you take the lazy way like me and think it would be as simple as comparing the body of the response with a string matching the expectant XML? Well there is a problem, if you've got multiple attributes on any elements you can't always guarantee the order that the attributes will be output. So lets do it properly, and check it is valid XML and has the attributes we want. I was working on a project needed a graph of election results. The flash graphing object I was using required XML output, which was easy with an appropriate rails respond_to block. However testing it wasn't as straight forward, or so I though initially. After banging my head for an hour or so I realised that the built in have_tag method and support for css selectors would give me everything I needed:

require File.dirname(__FILE__) + '/../../spec_helper'

describe "When viewing an election category as XML" do

  include ModelHelpers

  before(:each) do
    @election = mock_model(Election, setup_election_helper)
    assigns[:election] = @election
    render 'election/show.xml.builder'
  end

  it "should export valid XML for inclusing in flash charts" do
    response.should have_tag("chart[caption='Test Election'][xAxisName='Nominee'][yAxisName='Votes']")
    response.should have_tag("chart set[value=100][label='nominee 1']")
    response.should have_tag("chart set[value=200][label='nominee 2']")
    response.should have_tag("chart set[value=200][label='nominee 3']")
    response.should have_tag("chart set[value=200][label='nominee 4']")
    response.should have_tag("chart set[value=200][label='nominee 5']")            
  end

end

I'd dabbled with using hpricot to parse the output and traverse through the tree to ensure the elements are all where they should be, but this meets the requirements perfectly. However, it may be worth wrapping the hpricot methods in customer rspec matchers in the long term to make the assertions more readable. For the specific situation though (generating XML to meet the spec of a 3rd party app) I'm happy enough with the result.

Correctly handling multiple time zones in ruby on rails

Thu, 06 Dec 2007 09:00:00 GMT

A recent client I was working for had the unfortunate situation where we'd both independently assumed we understood the requirements, without realising a missing piece of complexity. It was around time zones. The system was implemented with a cut-off date being US EST, but most of their clientele are in LA so it needed to be US PST. What is a developer to do?

Well it turns out that timezone_fu will come to the rescue.

Installing time zone support into your rails app

Getting timezone_fu working with an existing rails app is a cinch, the only dependency is on the TZInfo package, so best you grab that now:


sudo gem install tzinfo

And the install the plugin in your rails app:


./script/plugin install http://www.thrivesmart.com/open_source/timezone_fu

Now before we do anything, make sure that we are defaulting to UTC time in environment.rb:

ENV['TZ'] = 'UTC'

Migrating your models so they know about time zones

Next, take your model that you want to apply your dates to. In my scenario it was an election which had a start and end date for when voting could occur. First thing I needed to do was to add a field to record the time zone that the dates were to be handled in:

class AddTimezoneToElections < ActiveRecord::Migration
  def self.up
    add_column :elections, :timezone, :string
  end

  def self.down
    remove_column :elections, :timezone
  end
end

And then amend my model definition in election.rb to add in the time zone helper:

class Election < ActiveRecord::Base
  has_timezone :fields => [:start_at, :end_at], 
               :default_timezone => "America/Los_Angeles",
               :time_format => "%I:%M %p on %A %d %B %Y" 
end

Storing and retrieving time zones in your rails app

The wonderful thing about this setup is that dates saved or retrieved from the model using the normal attribute accessors will displayed in the local time (as determined by whatever you save in the timezone field). I've also used all of the attributes to show their usage, but you can safely leave efault_timezone and ime_format off if you desire. The latter is a strftime string format outlining how to format the string, so we can get something like:

>> @election.display_end_at
=> "08:10 PM on Wednesday 20 February 2008"

Unfortunately though the strftime method used by the display_attribute helpers still thinks that the timezone is in UTC format (I'll hopefully get around to a patch shortly), so within my views to display the date with the timezone it is actually more like:

<%= "#{@election.display_end_at} #{TZInfo::Timezone.get_proxy(@election.timezone).strftime("%Z")}" %>

Simplifying rails controllers with make_resourceful

Tue, 27 Nov 2007 09:00:00 GMT

See the content of the picture above? Wouldn't it be wonderful if you could reduce all of your controllers down to 3 lines? Well you can't do all of them, but if you are adhering the a RESTful approach then chances are most of the controllers can be condensed considerably using make_resourceful.

Firstly I have to thank Jeremy McAnally for putting me onto this wonderful new plugin for rails. He used it in a project we were working on together and I've used it almost everywhere since. It's from the Hampton Catlin who is also known for the excellent haml plugin/templating engine.So what is make_resourceful?

Basically it's an extension of rails to take 'convention over configuration' and 'don't repeat yourself' even further. If you're following a standard RESTful approach then make_resourceful takes care of almost all of the heavy lifting around interfacing with your models and loading them into instances.

What do you mean by 'standard RESTful approach'?

If your controllers are broken up into the index/show/update/create/edit/destroy methods and you are defining the route to them with map.resources then you are in the right track. Chances are for you user controller you've got index doing something like:

def index
  @users = User.find(:all)
end

and show doing something like:

def index
  @user = User.find(:id)
end

Hampton was smart enough to realise many of us are doing exactly that on a lot of our controllers now, so why not just assume that is the case?

So what is the improvement in code?

Very quickly we can reduce the controller definition to be:

make_resourceful do
    actions :index, :show   
  end

If we've also got a method to create a new user, we can turn that on by simply:

make_resourceful do
    actions :index, :show, :new, :create
  end

What about if we add in edit and update too? We actually want to support every action except for destroy. Well that is exactly what we specify then:

make_resourceful do
    actions :all, :except => :destroy
  end

Can I handle nested routes?

Of course! And the wonderful thing is it's just as simple as what you've seen above:

class PostsController < ApplicationController
  make_resourceful do
    actions :index
    belongs_to :user
  end
end

This will effectively do the following:

@user = User.find(params[:user_id])
@posts = User.posts

What if I want to provide additional data to my find? Or instantiate additional objects for my views? Change the name of my instance?

There are a range of helper methods that make_resourceful sets up and uses that can be overridden on a per controller basis. The key ones to look at are listed in the RDocs for Default:Accessors. You also still have access to all the same before and after filters that you normally have in your controllers.

I'm sold, how do I install it?

As it's a standard rails plugin, installation is the usual way:

ruby script/plugin install http://svn.hamptoncatlin.com/make_resourceful/trunk
mv vendor/plugins/trunk vendor/plugins/make_resourceful

Make sure you run the last step, otherwise you'll have make resourceful exported into vendor/plugins/trunk rather than vendor/plugins/trunk.

Be careful! It is no silver bullet

I've quickly come to realise that make resourceful is great for getting very simple controllers out the door quickly. What it is not however is a complete replacement for ActionController. There will be times where the number of before filters and hacks you have to accommodate to get your controller working means you're better off refactoring the code back into a more traditional approach. However, even in these scenarios it's a great way to quickly define the specs and get you to a point where you can put the application through usability testing while cleaning up the mess afterwards.

Top 5 usability and design principles in web site development

Fri, 23 Nov 2007 09:00:00 GMT

Many clients I work with today have come from a wealth of experience in other media formats, be it the big screen or traditional print. They often bring with them a pre-conceived notion of how they want things to look and work on the site. Quite often it is a direct translation of their existing media strategies and they want to keep the content and format consistent for the sake of 'branding'. The web however is a fundamentally different beast than most mediums due to both the way the user interacts with it and the immensely broad reach. So a slightly different approach is needed. Pay attention to these things, and you'll ultimately end up with a happier audience and happier clients.

Users want instant gratification

There is nothing worse than clicking a link and waiting 10 seconds for a response. Back in the early days this was the norm, but now that content is delivered near instantly an expectation is set. Don't make them endure a long download just to show them a doorway flash page into your site, or some new animated navigation bar. Likewise with operations you now may take a while to process such as a search or uploading and processing of data. Where possible, push long running operations into a background process if the user doesn't need to immediately see the results. Where they do, provide the illusion of a partial instant response with a progress indicator

3 it's the magic (click) number

(... with apologies to schoolhouse rock). An old one from back in the day that still holds up in some regards today. And even for those naysayers that believe things are different, I've not heard a valid reason as to why should strive to meet this objective so long as it doesn't conflict with any others listed here. That is, make sure that whatever the user wants to do or find is never more than 3 clicks away. Beyond that point you run the risk of the user becoming frustrated and using that little Google search box in the top of their browser to find somewhere other than your site to get what they want. And think about it, if you're using a traditional desktop application how often do you need more than 3 clicks to do anything within it? Menu bar > Option > Tool panel. File > Open > Select File. And online? Go to Amazon > Find Book > 1-click order. It won't always be possible with absolutely every site, but you should try and make most adhere to it.

Make sure your elements Fitts

Fitts's Law is from the wikipedia article: "a model of human movement which predicts the time required to rapidly move to a target area, as a function of the distance to the target and the size of the target'. Basically what that means is that the bigger and more obvious you can make the areas you want someone to click, the easier it will be. Seems obvious right? Take a read through this quiz over at AskTog for some insight into how what otherwise seems like common-sense, isn't that sensible at all. Suffice to say, you really need to pay more attention to when you place a 'buffer zone' around your objects, and when you flush things hard against the screen.

Make it as simple as possible, and no simpler

Psychological studies have raised a theory that the human brain is usually only capable of handling around 7 pieces of stimulus at a time. If you are wanting to maximise reader engagement, or improve your click through rates, the best thing to do is reduce the number of options they have to deal with at any given point in time. That goes for navigation, filling in forms, etc. Start pushing past 10 items unless it's absolutely needed or expected and you starting walking the fine line to sensory overload.

Summarise your content

This fits in nicely with the expectation of instant gratification. Meaningful headlines and a first paragraph that acts almost like a conclusion or executive summary, giving the outcome of the following text allows the user to quickly determine if the content is appropriate. At the end of the day, you don't want to falsely lure users into sections they won't find relevant.

Engine Yard Bounty Pays Off

Wed, 21 Nov 2007 09:00:00 GMT

Engine Yard, the current big daddy of rails hosting, decided to put a bounty out to solve a problem all the newly converted rails/nginx users would have to some degree but have previously been ignoring. The load balancer in nginx works in a round robin fashion meaning that any long running request has the potential to cause another user to queue behind it, rather than getting served by one of the free and waiting mongrels. Well wait no more!

Take a look at Ezra's blog for the most definitive run down on how to get it working for you, but the digest version goes a little something like this.

Download the optional module tarball from nginx fair proxy balancer and then do the following:

cd nginx.git
./configureâwith-http_ssl_module
make
sudo make install

Now that you've compiled with the new module all that is left to update your upstream nginx.conf to include the fair directive:

upstream mongrel {
fair;
server 127.0.0.1:3000;
server 127.0.0.1:3001;
server 127.0.0.1:3002;
}

Reject junk email addresses with email veracity

Tue, 20 Nov 2007 09:00:00 GMT

Carsten Nielsen has created an amazing plugin for rails called email veracity. It takes all of the heavy lifting out of what can often be a problematic part of data validation, and takes it a step further than most.

Not only does email veracity remove the need for a convoluted regexp to match against the email string, but it will also check the domain records to ensure that the MX entry is currently accepting mail, wow!

So how do I check the format of an email address?

Add this to the code on your model definition:

validates_email_veracity_of :email

that single line does the domain check too! You can turn it off, apply a timeout, and/or a custom error message if you like too:

validates_email_veracity_of :email, :domain_check => false, :timeout => 5, :message => "is not valid"

Installation

It's available as a plugin so use piston, or the following command:

script/plugin install http://svn.savvica.com/public/plugins/validates_email_veracity_of

Trap rails from sending mail via SMTP

Fri, 16 Nov 2007 09:00:00 GMT

Trap rails from sending mail via SMTP Matt Mower has released a handy little tool called Mailtrap. Basically what it does as act as a very simple SMTP server that accepts requests from ActionMailer, but writes them to file instead of emailing them on.

Installation is very straight forward:

sudo gem install -y mailtrap

And then start it up:

mailtrap start

Which will start the mailtrap app to listen on port 2525. Now all that is left to do is change the config in your rails app to connect to localhost:2525 in either /config/environment.rb or /config/environment/development.rb:

ActionMailer::Base.perform_deliveries = true
ActionMailer::Base.raise_delivery_errors = true
ActionMailer::Base.delivery_method = :smtp
ActionMailer::Base.smtp_settings = {
  :domain => "mydomain.net",
  :address => "localhost",
  :port => 2525,
}

Ebay launches rails app

Mon, 12 Nov 2007 09:00:00 GMT

Not that there haven't already been high profile players talking about ruby and rails, but Ebay have just recently launched own rails app called MicroPlace. It's a micro-donation site like Kiva where you can invest directly into those in the 3rd world and help them escape the poverty cycle.

PrototypeJS adds better OO support

Thu, 08 Nov 2007 09:00:00 GMT

The folks over at prototype have had their latest 1.6 release in beta for a while, and I was using it quite extensively with much joy at my last client. One of the major enhancements to the entire system has been the reworking of the Element and Object classes, and the underlying Class object.

Previously there was all kinds of nastiness when trying to define a custom class. Here's an example taken from the docs:

var Person = Class.create();
Person.prototype = {
  initialize: function(name) {
    this.name = name;
  },
  say: function(message) {
    return this.name + ': ' + message;
  }
};

var guy = new Person('Miro');
guy.say('hi');
// -> "Miro: hi" 

var Pirate = Class.create();
// inherit from Person class:
Pirate.prototype = Object.extend(new Person(), {
  // redefine the speak method
  say: function(message) {
    return this.name + ': ' + message + ', yarr!';
  }
});

var john = new Pirate('Long John');
john.say('ahoy matey');
// -> "Long John: ahoy matey, yarr!"

But now with the improved Class constructor much of that heavy lifting can be put straight into the Class.create(). Then Pirate can be defined with something that looks more akin to proper object inheritance as opposed to extending the object. The end result is cleaner and more readable code:

// properties are directly passed to `create` method
var Person = Class.create({
  initialize: function(name) {
    this.name = name;
  },
  say: function(message) {
    return this.name + ': ' + message;
  }
});

// when subclassing, specify the class you want to inherit from
var Pirate = Class.create(Person, {
  // redefine the speak method
  say: function($super, message) {
    return $super(message) + ', yarr!';
  }
});

var john = new Pirate('Long John');
john.say('ahoy matey');
// -> "Long John: ahoy matey, yarr!"

This is just one of the many improvements in the API, some of the others are equally outstanding. I can't recommend enough the 1.6 release and I've had no problems with it, so checkout the release notes and get cracking with it today!

google, social networking

Fri, 02 Nov 2007 09:00:00 GMT

Sorry about the lull in posts of late all. I've been busy traveling and catching up on much lost quality time with my girlfriend who has returned from Australia. Big news today though, with Google looking to taken on the Facebook Application juggernaut with their own API.

With what sounds like it could be a master stroke of Google can pull it off correctly, they are development a social network agnostic API for applications and then trying to get all the various sites on board to support it. So no longer would the likes of Slide have to develop their apps for Myspace, and then again for Facebook. It'd be once for OpenSocial and the deploy anywhere.

I can see the obvious parallels between this and Java with the 'write once, deploy everywhere' philosophy but I wonder how successful it will ultimately be. As appealing as it is for myself and other developer folk, it won't be us developers that are deciding where the critical mass of eyeballs decide to sign up. If Facebook maintain their own platform and it ends up a two horse race between them and another player, then it makes this kind of platform pointless as I'd still have to write the application twice.

Hopefully this is one of those situations where the size and credibility of Google helps pull it altogether. Go take a look once the site goes live.

Securing SSH access

Tue, 04 Sep 2007 09:00:00 GMT

So you've got a shiny new server, it's connected to the net, and you (or your hosting company) has done the right thing by turning telnet off and forcing the much more secure option of using SSH to connect. But it doesn't end there, there are quite a few little things you can do to further tighten access and keep the boogey men at bay.

Setting up the server

First things first, lets move it off the default port 22. So many scripted and brute force attacks will try and blast at the defaults, and your memory isn't that bad that you have to stick with them. So pick a number, and edit/update the /etc/ssh/sshd_config file to have the following line:

Port 1022

Next, remove the legacy support for the less secure SSH1 protocol in favor of the newer more robust SSH2 protocol:

Protocol 2

You've already created a user and added yourself to sudoers, so there is no need for the root user to log in directly. Time to disable that. While we are at it, lets stop brute force attacks by reducing the number of incorrect attempts a user gets:

PermitRootLogin no
MaxAuthTries 2

Now onto the client

We can remove the need to enter our server password all the time by pushing our public key onto the server and using that to authenticate. If you already have a public/private key combination you can ignore this next step.

Creating your public/private keys is very straight forward on linux and mac, on windows you'll need to get putty. From the command line:

ssh-keygen -t dsa

This will prompt you for some questions and a passphrase, the default options will be fine. The keys will be saved to ~/.ssh/, so now we will move the contents of the public key up to the server.

cat ~/.ssh/id_dsa.pub | ssh username@server "cat >> .ssh/authorized_keys" 
ssh username@server chmod 600 .ssh/authorized_keys

Be sure to replace username and server with the appropriate values for the server we are setting up. Now you should be free to SSH onto the server, without a password!

Back to the server

Now that you don't need a password to get into the server, we can prevent people from trying to use one. We can also restrict the users that are permitted to access while we are at it. So back into /etc/ssh/sshd_config and update/edit the following lines:

PasswordAuthentication no
ChallengeResponseAuthentication no
AllowUsers username

Again being sure to change username to your username. Additional users can be entered, separated with a space. All done and dusted? Restart the SSH daemon:

sudo /etc/init.d/sshd reload

Installing Memcached

Mon, 03 Sep 2007 09:00:00 GMT

Memcached is an alternative to database storage that is often useful for session storage and caching. It was developed by the guys at live journal to help them maintain their unwieldy sessions. Since then it's gained a lot of popularity among the PHP and Rails folk because it removes some of the burden on the typical database bottle neck, and allows things like session to be maintained across a cluster of servers.

First things first, you're going to need to download and make the source for libevent, which memcached has a dependency on:

wget http://www.monkey.org/~provos/libevent-1.3d.tar.gz
tar zxf libevent-1.3d.tar.gz
cd libevent-1.3d
./configure --prefix=/usr/local
make; sudo make install; make clean;
cd ..

Once that is done, you're ready to get onto the setup and install of memcached itself:

wget http://www.danga.com/memcached/dist/memcached-1.2.2.tar.gz
tar zxf memcached-1.2.2.tar.gz
cd memcached-1.2.2
./configure --with-libevent=/usr/local/lib --prefix=/usr/local
make; sudo make install; make clean;
sudo /sbin/ldconfig /usr/local/lib
cd ..

And then finally, the ruby client interface:

sudo gem install memcache-client -y

Once that is installed, moving over your session store to use memcached is a breeze, simply change the setting in your environment.rb file to:

config.action_controller.session_store = :mem_cache_store

Installing Subversion and MySQL

Thu, 30 Aug 2007 09:00:00 GMT

I first got forced into using source control management (scm) systems when I first began working as part of a team of developers and there was times we'd need to edit the same parts of the system. It wasn't too long until those good habits trickled into the stuff I worked on by myself, and for good reason. It's nice to always have an undo option for those time when you go "Whoops!". Subversion (SVN to some) is the version control tool du jour for many, so lets keep it simple to begin with and start with that. I've included installing MySQL and Postgres as well, because well it's version control and databases finish off what you need to have your app deployed; and because they are all pretty straight-forward

Installing Subversion

So first subversion. We will go get the subversion source, the dependencies package (zlib compression support, support for http(s):// repositories, etc), and then configure and install:

wget http://subversion.tigris.org/downloads/subversion-1.4.5.tar.gz
wget http://subversion.tigris.org/downloads/subversion-deps-1.4.5.tar.gz
tar zxf subversion-1.4.5.tar.gz
tax zxf subversion-deps-1.4.5.tar.gz
cd subversion-1.4.5
./autogen.sh
./configure --prefix=/usr/local --with-openssl --with-ssl=openssl
make; sudo make install; make clean;
cd ..

Installing MySQL

Now it's time for MySQL. You'll need to ensure you've got the g/c++ libraries installed for this one:

wget http://dev.mysql.com/get/Downloads/MySQL-5.0/mysql-5.0.45.tar.gz/from/http://www.mirrorservice.org/sites/ftp.mysql.com/
tar zxf mysql-5.0.45.tar.gz
cd mysql-5.0.45
./configure --prefix=/usr/local
make; sudo make install; make clean;
./scripts/mysql_install_db
sudo cp support-files/mysql.server /etc/init.d/mysql
sudo chmod 755 /etc/init.d/mysql
sudo /sbin/chkconfig --level 345 mysql on
sudo /usr/sbin/adduser mysql -r
sudo chown -R root:mysql /usr/local/mysql
sudo chown -R mysql:mysql /usr/local/mysql/var
sudo cp support-files/my-medium.cnf /etc/my.cnf
cd ..

MySQL has just been installed, the standard tables populated, permissions, users, and configs have been setup and it is set to start itself on start-up/reboot. Now you'll need to issue a secure root password to the new server, so run the following command:

/usr/local/bin/mysqladmin -u root password 'new-password'

Replacing new-password with whatever you'd like your secure password to be. Once that is all done, start up the service manually as the mysql user:

sudo /usr/local/mysql/bin/safe_mysqld --user=mysql &

Installing Ruby, Rails, and Rubygems

Tue, 28 Aug 2007 09:00:00 GMT

Getting your first ruby on rails stack up and running can be a bit of a chore, here’s a quick guide on how to do it on a linux box but the instructions should be equally good for OS X users.

Installing Ruby

Now depending on your OS, you’ve got a few options on how to get ruby up and running. The easiest way is to use the built-in package management system. This will give you the latest known stable version of ruby for your environment, and hopefully reduce or remove any potential conflicts with other installed software. The downside is that quite often the packaged version are a few releases behind the currently advertised stable version just due to the time it takes for the linux distribution manages to bundle them up.

Installing Ruby from yum/apt-get

If you’ve got a package manager, just enter whichever of the following commands is the most appropriate:

sudo yum install ruby irb rdoc ruby-devel
sudo apt-get ruby irb rdoc ruby-devel

Installing Ruby from Source

If you’ve decided to take the from source option, then you’ll need to do the following:

wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p72.tar.gz
tar zxvf ruby-1.8.7-p72.tar.gz
cd ruby-1.8.7-p72
./configure
make
sudo make install

Installing ruby gems

Gem is the package system for ruby, and it’s the easiest way to install most of your ruby dependencies (including rails). So grab it from sourceforge:

wget http://rubyforge.org/frs/download.php/43985/rubygems-1.3.0.tgz
tar zxvf rubygems-1.3.0.tgz
cd rubygems-1.3.0
sudo ruby ./setup.rb

Installing Ruby on Rails

Now getting rails installed is a breeze, all you need to do is:

sudo gem install rails

You may need to install your native database driver bindings too, if so:

sudo gem install mysql
sudo gem install postgres

Adding yourself to the sudoers file

Mon, 27 Aug 2007 09:00:00 GMT

If you’ve gone at least some way to trying to make your server secure, you wont be running as root so you wont have access to many of the administration commands you’ll need. It’s easily resolved by adding yourself to the sudoers file, here’s a quick guide on how to do it.

Giving a user access to sudo

Unless you’ve already got an account with sudo access, you’re going to need to log in as root one last time to set one up. So do the following:

su -

And enter in the password for root when prompted. Next we’ll edit the sudoers file using visudo:

visudo

This will open up the file in a vi session for editing. If you’ve not got crazy vi chops don’t worry, were don’t need to do anything complicated. Just page down to the bottom of the document and enter the following:

your_username_here ALL=(ALL)       ALL

Hit esc then type:

:wq

and press ENTER. The file will be saved, and you’ve just granted yourself access to run everything as the root user. Now type:

exit

to log out as root and back in as yourself. Now if we want to edit the sudoers file we can do the following:

sudo visudo

this time when you are prompted for a password, you only need to enter your own one. And tada, the sudoers file is open again. Now let’s give some more users access, but we won’t be quite so generous with what they can do.

Giving a user access to a single command

What about on a production machine where you’ve got a user that is a little bit trusted, but shouldn’t be given total access to the system? It quite easy to setup some walled access to the commands you need to open up. In the example below, we will grant access to god for the user deploy so that we can start and stop our services through capistrano.

deploy  ALL= /usr/bin/god

Sudo command line howto

How about running a command as another user, without changing to that user? Simple:

sudo -u otheruser /usr/bin/command

Or running a command and pushing it to the background:

sudo -b /usr/bin/command

Or just finding out what commands you are allowed to run:

sudo -l

Sudo config (more detailed sudoers file options)

The sudoers file offers quite a lot of control over exactly what someone can run, as well as who they can run it as, and from where. Let’s just quickly run through a few of the other options you’ve got in the sudoers file:

Restricting who a user can run commands as

The following snippet allows the user bob to run all commands from anywhere, but only as alice or anne:

RunAs_Alias     HELPDESK = alice, anne
bob ALL=(HELPDESK) ALL

Restricting where a user can run commands from

This config allows bob to run any command as any user, but only from the defined subnet:

Host_Alias     MYNET = 10.1.2.0/255.255.255.0
bob MYNET=(ALL) ALL

Installing Apache 2.2.4

Mon, 27 Aug 2007 09:00:00 GMT

Working for various clients and deploying a full working rails stack on their hardware and environments is not always as straight-forward and easy as it should be. What distribution are they using? Which package manager? Does the package manager even currently have the right versions I need? So over the coming days I’m going to write a few tutorials on getting the latest software, from source, and compiling it. So intro today, with Apache.

The only assumption I make is that gcc is available on the system so the we can compile the source into binaries. If you don’t have it, you’ll need to install it for your system. So the first thing to do is download the latest unix source from a local mirror and save it somewhere convenient. To make life easier, I’m going to download all the files directly to the server using wget. So it’s simply a matter of:


mkdir rails-setup
cd rails-setup
wget http://www.apache.org/dist/httpd/httpd-2.2.4.tar.gz

But before we get too far ahead of ourselves, I know that I’m going to want to add in the ability to compress output I’m sending back to the end user, as well as support for HTTPS/SSL. So I’m going to need those libraries too. Lets go get them:


wget http://switch.dl.sourceforge.net/sourceforge/libpng/zlib-1.2.3.tar.gz
wget http://www.openssl.org/source/openssl-0.9.8e.tar.gz

And tada! We’ve hopefully got all we need. Time to compile.


tar zxf zlib-1.2.3.tar.gz 
cd zlib-1.2.3
./configure --prefix=/usr/local
make; sudo make install; make clean;
cd ..

So what have we just? We’ve grabbed the zlib tarball and extracted and decompressed it into it’s own directory. We then go in there, and run ./configure to ensure it is setup for our environment. Then we run make, install it once it’s done, and then clean up the leftovers.

Time to do the same with OpenSSL.


tar zxf openssl-0.9.8e.tar.gz
cd openssl-0.9.8e
./config --prefix=/usr/local
make; sudo make install; make clean;
cd ..

Okay, now we should be ready for Apache. Here’s the next series of commands I typically run:


tar zxf httpd-2.2.4.tar.gz
cd httpd-2.2.4
./configure --enable-mods-shared=all --enable-proxy --enable-proxy-balancer --enable-proxy-http --enable-rewrite --enable-cache --enable-headers --enable-ssl --enable-deflate --with-included-apr --with-ssl=/usr/local/ssl --prefix=/usr/local/apache2
make; sudo make install; make clean;
cd ..

So what we’ve done is run the configure script to set things up for the current environment, letting make know that we want to enable

Mod proxy (so we can send requests through to mogrel/rails)
Mod proxy balancer (so that we can distribute to various mongrel instances)
Enabled proxy support (need it to pass requests to mod proxy)
Turn on URL re-writing (so we can handle rails routing)
Enable dynamic file caching
Let us take control of the HTTP headers
Turn on support for SSL/TLS (nice and secure)
Turn on support for deflate transfer encoding (helps keep the transfer size down)
The location of the OpenSSH installation
The location where we want to install Apache

It will then run through, compile all the code, install it on the server, and clean up it’s mess it might have left behind.

And there you have it! Apache is setup, and should be working and running on your server. Over the coming days I’ll post further instruction on getting the rest of the stack running. That is going to entail MySQL and/or Postgres, ruby, rails, mongrel, mongrel_cluster, and monit for managing the services and starting/stopping everything. And of course, the various configuration that is required to make it all talk together.

You could also delete the files we downloaded and the directories we installed from because our work here is done. I personally keep them around at least until everything is working together and the system is stable, but it’s up to you.

Welcome To The New Location

Sat, 11 Aug 2007 09:00:00 GMT

Well it has been quite a while in the making, but I’ve finally decided to go it out on my own. After taking a few months off traveling and to head home and visit friends and family I’ve relocated back to the UK and consolidated my various freelance and investment interests across continents into a single company, Ruby Pond. That’s not to say I’ve not been busy working. There are a number of projects which I’ve been working on, for both external clients and personal ventures. Hopefully a steady stream of them will start coming out the virtual doors in the coming weeks, so stay tuned.

Until then, expect this blog to be my geek soap box for anything related to the various facets of of my business and random learnings. For my on-going travel diaries from my newly bedouin lifestyle or to hear what my other half has to say check out the updates (when we actually get around to them) at Em and glenn