A technical post: Ruby interpreter comparison---MRI 1.9.1 vs. REE 1.8.7

I ran a small, controlled "real-world" experiment comparing these two interpreters.  The results show identical time performance.

My interest is Rails application speed

This site, OregonLaws.org, is a Rails 2.3.4 application.  (These blog pages are the sole exception, running on Wordpress.)  I manage the architecture and deployment so that server response time is 200 ms or less --- a goal I borrowed from Google.  I've been interested in moving to Ruby 1.9.1 now that all of the packages I use are compatible with it.  There are still some issues running Rails on 1.9.1 (such as the string encoding problems) and I wanted to see if there's really anything to be gained by spending the time on it vs. using Ruby Enterprise Edition.

Results

Here are the data from my trials (lower is better):

Methods

I ran the OregonLaws.org Rails 2.3.4 app in development mode with WEBrick in an Ubuntu 9.10 VM on my Mac laptop.  I have the latest of each interpreter type installed:  (1) The 1.9.1.243-2 MRI (standard) Ruby interpreter, installed via the Ubuntu package system.  (2) The 1.8.7-2009.10 REE Ruby interpreter installed from the source code. I switch between these by changing my PATH.  The entire environments are identical except for the Ruby interpreters.

After starting the server ("./script/server") I run one trial of each page and discard the results, in order to prime any caches.  I then just clicked through the pages---a mix of static and dynamic content---typing in the "Completed in" time from the development log.

I calculated cumulative average times of 73 ms for MRI and 75 ms for REE, +/- 0.5 ms.  I'm reporting these as whole numbers because the decimals must be dropped according to the rules of significant digits.  I believe that the 2 ms difference between them is too small to be considered a real finding.  (It's been a while since I've calculated statistical significance; but my gut tells me that this difference is just random noise/chance.)

Interpretation

These results show that these two interpreters have essentially identical execution times in a real-world scenario.  Based on this, it makes sense for me to stay with REE, as it is 100% compatible with my Rails application.

Caveats

1. Small sample size
2. App is running development mode

The results might look a little different if done in a production application, and over a longer period of time with more pages requested.  For my purposes, though --- to get a fairly good idea about how these two interpreters compare --- I believe the results are very helpful.