r10k¶ ↑

r10k provides comparative benchmarks for many ruby web application frameworks with 10, 100, 1000, and 10,000 routes. Here are the currently supported frameworks (in order of number of gem downloads):

rails

Ruby on Rails (rubyonrails.org/)

sinatra

Sinatra (www.sinatrarb.com/)

roda

Roda (roda.jeremyevans.net)

hanami

Hanami (hanamirb.org/)

cuba

Cuba (cuba.is)

syro

Syro (soveran.github.io/syro/)

cuba

Camping (www.ruby-camping.com/)

These frameworks are designed to be run with the default settings. There are some variants that can be run with non-default settings:

roda-hash-routes

Roda, using the hash_routes plugins

roda-osm

Roda, using optimized_string_matchers, optimized_segment_matchers, and plain_hash_response_headers plugins

roda-psm

Roda, using placeholder_string_matchers plugin (or for benchmarking Roda < 2.27)

roda-sym

Roda, using symbol matchers

hanami-api

Hanami, using hanami-api gem

The frameworks that are tested by default are those with more than 250,000 gem downloads, except for Sinatra:

rails roda cuba hanami

Sinatra is excluded by default because testing sinatra takes about 6x longer than testing Rails, and testing Rails takes 6-25x longer than testing Roda, Cuba, or Hanami.

Installation¶ ↑

After cloning the repository, install the related gems:

gem install gruff # if you want to create the graphs
gem install rails roda cuba # frameworks to test
gem install hanami-controller hanami-router

A Gemfile is not used, because all web frameworks are tested independently, and with a Gemfile, bundler assumes all gems must be able to work together.

Usage¶ ↑

To use:

rake bench graphs

Note that rake bench can take significant time, especially on slower hardware. If you only want to benchmark specific apps, set the R10K_APPS environment variable:

rake bench graphs R10K_APPS="roda cuba"

The builders directory contains the files that build the applications. If you want to examine the application files used for benchmarking, you can use the build task and then look in the apps directory. The bench task produces CSV files which are stored in the data directory. The graphs task takes those CSV files and turns them into nice looking graphs (using gruff), storing the graphs in the graphs directory.

Benchmarks¶ ↑

r10k produces three benchmark numbers:

• rps: Requests per second

• memory: Memory usage to load the application (before sending requests)

• runtime_with_startup: Total time taken to run benchmark

Graphs are produced for each benchmark. There is an additional log_rps graph produced, which shows the requests per second in a logarithmic scale.

Checking¶ ↑

If you want to add support for a new framework, or check that support for existing frameworks runs correctly, use the check task:

rake check

This ensures the following:

• Incomplete routes return 404

• POST requests return 404, 405, or 501

• All valid routes return expected bodies

• All valid routes return 200 status

• All valid routes include text/html in Content-Type header

• All valid routes return correct Content-Length

Because most Ruby web frameworks tested do not treat trailing slashes in routes correctly, considering a request without a trailing slash the same as a request with a trailing slash, the check task will only warn for incorrect handling of trailing slashes, instead of raising an exception. Of the web frameworks tested, only Roda and Sinatra treat routes with trailing slashes correctly.

To save time, checks are done with only 2 routes per level.

Static Benchmarks¶ ↑

r10k historically used static paths in benchmarks. This was determined to be a poor benchmark, since the fastest implementations basically just used a Ruby hash for route lookup. The dynamic route benchmarks that r10k now uses are more similar to production routes. However, there may still be be value it benchmarking static route performance, so the static route benchmarks are still available:

rake static_bench static_graphs

The data for the static route benchmarks can be found in the static/data directory, and the graphs in the static/graphs directory.

Additional web frameworks are supported for the static benchmarks:

nyny

NYNY (github.com/alisnic/nyny)

rack-app

Rack::App (github.com/rack-app/rack-app)

synfeld

Synfeld (github.com/swerling/synfeld)

watts

Watts (github.com/pete/watts)

Additional variants are supported for the static benchmarks:

static-route

A simple Rack implementation

rails-minimal-middleware

Rails, with most middleware removed

roda-multi-route

Roda, using the multi_route plugin

roda-run

Roda, using separate Roda applications per branch

roda-static-routing

Roda, using the static_routing plugin

sinatra-minimal-middleware

Sinatra, using the minimum middleware

Environment variables:

DIM

Dimensions for the graphs (default: 1280x720)

LEVELS

The number of levels for the routes (default: 4)

LEGEND_FONT_SIZE

Override base font size for legend text in graphs

R10K_APPS

Which applications/configurations to benchmark

R10K_ITERATIONS

The number of iterations to run (default: 2)

R10K_NUM_THREADS

The number of threads to run (default: 1)

R10K_WARMUP_ITERATIONS

The number of warmup iterations to run (default: 0)

ROUTES_PER_LEVEL

The number of routes in each level (default: 10)

MAXIMUM_VALUE_*

Set maximum value for graph (* can be rps, log_rps, memory, runtime_with_startup)

Total number of routes in largest application is ROUTES_PER_LEVEL**LEVELS.

License¶ ↑

MIT

Author¶ ↑

Jeremy Evans <code@jeremyevans.net>