'ab' is not fast enough

[joanhey]

When benchmarking, It is very important to see the CPU usage, for find the bottlenecks.
With this benchmark using 3s per fw, it is not easy to control the CPU usage.
But changing to 30s per fw, It's possible.

The slowest frameworks are saturating the CPU (~100% usage), and that is correct. But the fastest frameworks only ~70% CPU usage. And the CPU usage increase with the low performant frameworks.
The problem is ab that can't give enough food (requests) to the fastest frameworks. We are benchmarking the ab performance, not the php performance.

Core i7 860 @ 2.80GHz × 8 PHP 5.5.9

ab -c 10 -t 3 http://localhost/hello.php(.html)
wrk -t10 -c10 -d3s --latency http://localhost/hello.php(.html)

ab	requests per second	transfer/second Mb	CPU usage %
hello.php	13,373.38	2,80	40-45
hello.html	14,254.07	3,48	35-40

wrk	requests per second	transfer/second Mb	CPU usage %
hello.php	35,285.86	6.77	~90
hello.html	49,283.88	11.15	~90

But perhaps only happens with fast CPUs, so I tried with:

Core 2 Duo 2.53Ghz PHP 5.6.21

ab -c 10 -t 3 http://localhost/hello.php
wrk -t10 -c10 -d3s --latency http://localhost/hello.php

hello.php	requests per second	transfer/second Mb	CPU usage %
ab	2,060.37	0.51	~40
wrk	5,621.32	1.30	~95

Exactly the same, we calculate the ratio:

CPU model	wrk	/	ab	Ratio
Duo	5,621.32	/	2,060.37	2.72831
i7	35,285.86	/	13,373.38	2.63851

The ratio is very similar, so It's only an ab performance limitation, independent of which CPU.
This ratio decrease proportionally to the low performance of the framework. Where ab is fast enough for the slowest frameworks.

For example, with the core i7, one of the fastest fw go from ~10.000 to ~16.000 req/s (more than hello.html with ab). And one of the slowest fw go only from 603 to 634 req/s.

This actual benchmark, is completely limited by ab performance.

Please use wrk, a modern HTTP benchmarking tool.

https://github.com/wg/wrk

https://github.com/wg/wrk/wiki/Installing-Wrk-on-Linux

[kenjis]

Thank you for your information.
It is very interesting and useful.

[stamster]

Indeed, ab is good for ad-hoc benchmarks but for near real-world you need multi threaded solution.

I like wrk, but also siege is worth mentioning!

[nazar-pc]

I want to add that concurrency 10 is quite low, on my machine increasing concurrency to higher values and time to at least 5 seconds shows significantly different results.
Maybe add some switches to benchmark.sh to control this?

[dejurin]

When benchmarking, It is very important to see the CPU usage, for find the bottlenecks.
With this benchmark using 3s per fw, it is not easy to control the CPU usage.
But changing to 30s per fw, It's possible.

The slowest frameworks are saturating the CPU (~100% usage), and that is correct. But the fastest frameworks only ~70% CPU usage. And the CPU usage increase with the low performant frameworks.
The problem is ab that can't give enough food (requests) to the fastest frameworks. We are benchmarking the ab performance, not the php performance.

Core i7 860 @ 2.80GHz × 8 PHP 5.5.9

ab -c 10 -t 3 http://localhost/hello.php(.html)
wrk -t10 -c10 -d3s --latency http://localhost/hello.php(.html)

ab requests per second transfer/second Mb CPU usage %
hello.php 13,373.38 2,80 40-45
hello.html 14,254.07 3,48 35-40
wrk requests per second transfer/second Mb CPU usage %
hello.php 35,285.86 6.77 ~90
hello.html 49,283.88 11.15 ~90
But perhaps only happens with fast CPUs, so I tried with:

Core 2 Duo 2.53Ghz PHP 5.6.21

ab -c 10 -t 3 http://localhost/hello.php
wrk -t10 -c10 -d3s --latency http://localhost/hello.php

hello.php requests per second transfer/second Mb CPU usage %
ab 2,060.37 0.51 ~40
wrk 5,621.32 1.30 ~95
Exactly the same, we calculate the ratio:

CPU model wrk / ab Ratio
Duo 5,621.32 / 2,060.37 2.72831
i7 35,285.86 / 13,373.38 2.63851
The ratio is very similar, so It's only an ab performance limitation, independent of which CPU.
This ratio decrease proportionally to the low performance of the framework. Where ab is fast enough for the slowest frameworks.

For example, with the core i7, one of the fastest fw go from ~10.000 to ~16.000 req/s (more than hello.html with ab). And one of the slowest fw go only from 603 to 634 req/s.

This actual benchmark, is completely limited by ab performance.

Please use wrk, a modern HTTP benchmarking tool.

https://github.com/wg/wrk

https://github.com/wg/wrk/wiki/Installing-Wrk-on-Linux

https://github.com/dejurin/php-micro-router-framework-benchmark enjoy 👍 )

[joanhey]

A real benchmark using wrk:
https://www.techempower.com/benchmarks/#section=test&runid=58042695-831a-4a35-8c60-2b872a06f799&hw=ph&test=fortune&l=zik073-7&c=4

https://github.com/TechEmpower/FrameworkBenchmarks

@dejurin Enjoy 👍
PD: try to understand why a micro framework it is not so fast in a real app.