Skip to content

Scripts for benchmarking various PHP implementations when running open source software.

License

Notifications You must be signed in to change notification settings

paulbiss/oss-performance

 
 

Repository files navigation

Overview

The goal is to provide a benchmark suite, testing something representative of real-world situations. This suite also includes some unrealistic microbenchmarks - comparing the results of these is fairly pointless, however they can still be useful to profile, to find optimization opportunities that may carry over to a real site.

This script sets up nginx, siege, and PHP5/PHP7/HHVM over FastCGI, over a TCP socket. Configuration is as close to identical as possible.

The script will run 300 warmup requests, then as many requests as possible in 1 minute. Statistics are only collected for the second set of data.

Results

We don't have anything to share yet - we want to standardize and document how the interpreters are built/installed first.

Usage

As a regular user:

composer.phar install # see https://getcomposer.org/download/
hhvm perf.php --wordpress --php5=/path/to/bin/php-cgi # also works with php7
hhvm perf.php --wordpress --hhvm=/path/to/hhvm

Running with --hhvm gives some additional server-side statistics. It is usual for HHVM to report more requests than siege - some frameworks do call-back requests to the current webserver.

Batch Usage

If you want to run multiple combinations:

composer.phar install # see https://getcomposer.org/download
hhvm batch-run.php < batch-run.json > batch-run-out.json

See batch-run.json.example to get an idea of how to create batch-run.json.

Requirements

  • composer
  • nginx
  • siege
  • unzip
  • A mysql server on 127.0.0.1:3306
  • hhvm

I've been using the current versions available from yum on Centos 6.3. HHVM is required as this is written in Hack.

The Targets

Toys

Unrealistic microbenchmarks. We do not care about these results - they're only here to give a simple, quick target to test that the script works correctly.

'hello, world' is useful for profiling request handling.

Wordpress

  • Data comes from installing the demo-data-creator plugin (included) on a fresh install of Wordpress, and clicking 'generate data' in the admin panel a bunch of times.

  • URLs file is based on traffic to hhvm.com - request ratios are:

    100: even spread over long tail of posts 50: WP front page. This number is an estimate - we get ~ 90 to /, ~ 1 to /blog/. Assuming most wordpress sites don't have our magic front page, so taking a value roughly in the middle. 40: RSS feed 5: Some popular post 5: Some other popular post 3: Some other not quite as popular post

The long tail was generated with:

<?php
  for ($i = 0; $i <= 52; ++$i) {
  printf("http://localhost:__HTTP_PORT__/?p=%d\n", mt_rand(1,52));
}

Ordering of the URLs file is courtesy of the unix 'shuf' command.

Drupal

Aims to be realistic. Demo data is from devel-generate, provided by the devel module. Default values were used, except:

  • Users were spread over the last year, rather than the last week
  • New main menus and navigation menus were created
  • New articles and pages were created, with up to 30 comments per content, spread over the last year instead of the last week

As well as the database dump, the static files generated by the above process (user images, images embedded in content) have also been included.

SugarCRM

The upstream installation script provides an option to create demonstration data - this was used to create the database dump included here.

There are two unrealistic microbenchmarks:

  • just the login page - the page with the username/password form. Added to confirm a reported issue.
  • just the logged-in home page. Added to be a little more realistic than rendering the form, however we have no idea what a realistic request distribution would look like

Laravel

Unrealistic microbenchmark: just the 'You have arrived' page from an empty installation.

Magento

  • Data is official Magento sample data, however because of its original size we have replaced all images with a compressed HHVM logo and removed all mp3 files.
  • After importing sample data we use the Magento console installer to do the installation for us.
  • URLs are a variety of different pages:
    • Homepage
    • Category page
    • CMS page
    • Quicksearch
    • Advanced search
    • Simple product
    • Product with options
    • Product reviews

Mediawiki

The main page is the Barack Obama page from Wikipedia; this is based on the WikiMedia foundation using it as a benchmark, and finding it fairly representative of wikipedia. A few other pages (HHVM, talk, edit) are also loaded to provide a slightly more rounded workload.

Profiling

perf.php can keep the suite running indefinitely:

hhvm perf.php --i-am-not-benchmarking --no-time-limit --wordpress --hhvm=$HHVM_BIN

You can then attach 'perf' or another profiler to the running HHVM or php-cgi process, once the 'benchmarking' phase has started.

Direct support (especially for XHProf) is planned, but not yet implemented.

Contributing

Please see CONTRIBUTING.md for details.

About

Scripts for benchmarking various PHP implementations when running open source software.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Hack 84.9%
  • PHP 15.1%