Evaluate using more optimizations like LTO, PGO, PLO

[zamazan4ik]

Hi!

I want to discuss several compiler optimizations that could be helpful for the project. Maybe you even already tested them and have some benchmark numbers to share publicly ;)

At first, did you test Link-Time Optimization (LTO) for GQL? If yes, do you have any numbers regarding performance and the resulting binary size? It would be nice to test since enabling LTO is not too time-consuming thing (usually, huh).

I suggest enabling LTO only for the Release builds so as not to sacrifice the developers' experience while working on the project since LTO consumes an additional amount of time to finish the compilation routine. If you think that a regular Release build should not be affected by such a change as well, then I suggest adding an additional release-lto (or dist) profile where additionally to regular release optimizations LTO also will be added. Such a change simplifies life for maintainers and others interested in the project persons who want to build the most performant version of the application. Using ThinLTO also should help).

If we are talking about more advanced things, recently I started evaluating using Profile-Guided Optimization (PGO) for various applications and workloads (including compilers, parsers, databases, and many others) - all the results are available at https://github.com/zamazan4ik/awesome-pgo .

As I have done many times before, I decided to test the Profile-Guided Optimization (PGO) technique to optimize the project performance. For reference, results for other projects are available at https://github.com/zamazan4ik/awesome-pgo . Since PGO helped a lot with many other applications, I decided to apply it on GQL to see if the performance win (or lose) can be achieved. Here are my benchmark results.

Test environment

• Fedora 40
• Linux kernel 6.10.7
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Compiler - Rustc 1.79.0
• GQL version: master branch on commit afd98727ae064ba93fc12462323b89e2acafc633
• Disabled Turbo boost

Benchmark

For benchmark purposes, I use built-in into the project benchmarks. For PGO optimization I use cargo-pgo tool. I got the release bench results with the taskset -c 0 cargo bench command. The PGO training phase is done with taskset -c 0 cargo pgo bench, PGO optimization phase - with taskset -c 0 cargo pgo optimize bench.

taskset -c 0 is used for reducing the OS scheduler's influence on the results. All measurements are done on the same machine, with the same background "noise" (as much as I can guarantee).

Additionally, I also enabled LTO with lto = true added to the [profile.release] profile in the root Cargo.toml file.

Results

I got the following results:

Release:

Tokenizer 100 Char      time:   [2.6253 µs 2.6421 µs 2.6521 µs]

Tokenizer 100K Char     time:   [2.7490 ms 2.7611 ms 2.7761 ms]

Tokenizer 1M Char       time:   [27.405 ms 27.535 ms 27.708 ms]

Tokenizer 10M Char      time:   [325.00 ms 326.28 ms 327.41 ms]

LTO compared to Release:

     Running benches/benchmarks.rs (target/release/deps/benchmarks-ab3f59841022f88d)
Tokenizer 100 Char      time:   [2.1278 µs 2.1290 µs 2.1317 µs]
                        change: [-19.360% -18.982% -18.614%] (p = 0.00 < 0.10)
                        Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild

Tokenizer 100K Char     time:   [2.3260 ms 2.3272 ms 2.3281 ms]
                        change: [-16.050% -15.707% -15.408%] (p = 0.00 < 0.10)
                        Performance has improved.

Tokenizer 1M Char       time:   [22.735 ms 22.755 ms 22.771 ms]
                        change: [-17.644% -17.293% -16.955%] (p = 0.00 < 0.10)
                        Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
  1 (10.00%) low mild
  1 (10.00%) high mild

Tokenizer 10M Char      time:   [281.17 ms 281.71 ms 282.31 ms]
                        change: [-14.010% -13.660% -13.271%] (p = 0.00 < 0.10)
                        Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild

LTO + PGO optimized compared to LTO:

Benchmarking Tokenizer 100 Char
Benchmarking Tokenizer 100 Char: Warming up for 3.0000 s
Benchmarking Tokenizer 100 Char: Collecting 10 samples in estimated 5.0000 s (3.3M iterations)
Benchmarking Tokenizer 100 Char: Analyzing
Tokenizer 100 Char      time:   [1.5251 µs 1.5256 µs 1.5264 µs]
                        change: [-28.495% -28.390% -28.296%] (p = 0.00 < 0.10)
                        Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
  2 (20.00%) high mild

Benchmarking Tokenizer 100K Char
Benchmarking Tokenizer 100K Char: Warming up for 3.0000 s
Benchmarking Tokenizer 100K Char: Collecting 10 samples in estimated 5.0527 s (2970 iterations)
Benchmarking Tokenizer 100K Char: Analyzing
Tokenizer 100K Char     time:   [1.7008 ms 1.7010 ms 1.7013 ms]
                        change: [-26.898% -26.845% -26.788%] (p = 0.00 < 0.10)
                        Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high severe

Benchmarking Tokenizer 1M Char
Benchmarking Tokenizer 1M Char: Warming up for 3.0000 s
Benchmarking Tokenizer 1M Char: Collecting 10 samples in estimated 5.6216 s (330 iterations)
Benchmarking Tokenizer 1M Char: Analyzing
Tokenizer 1M Char       time:   [17.013 ms 17.019 ms 17.024 ms]
                        change: [-25.242% -25.205% -25.167%] (p = 0.00 < 0.10)
                        Performance has improved.

Benchmarking Tokenizer 10M Char
Benchmarking Tokenizer 10M Char: Warming up for 3.0000 s
Benchmarking Tokenizer 10M Char: Collecting 10 samples in estimated 6.7384 s (30 iterations)
Benchmarking Tokenizer 10M Char: Analyzing
Tokenizer 10M Char      time:   [221.47 ms 221.69 ms 221.92 ms]
                        change: [-21.492% -21.305% -21.138%] (p = 0.00 < 0.10)
                        Performance has improved.

(just for reference) LTO + PGO instrumented compared to LTO

Benchmarking Tokenizer 100 Char
Benchmarking Tokenizer 100 Char: Warming up for 3.0000 s
Benchmarking Tokenizer 100 Char: Collecting 10 samples in estimated 5.0000 s (1.4M iterations)
Benchmarking Tokenizer 100 Char: Analyzing
Tokenizer 100 Char      time:   [3.5490 µs 3.5509 µs 3.5535 µs]
                        change: [+66.481% +66.997% +67.734%] (p = 0.00 < 0.10)
                        Performance has regressed.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high severe

Benchmarking Tokenizer 100K Char
Benchmarking Tokenizer 100K Char: Warming up for 3.0000 s
Benchmarking Tokenizer 100K Char: Collecting 10 samples in estimated 5.1725 s (1430 iterations)
Benchmarking Tokenizer 100K Char: Analyzing
Tokenizer 100K Char     time:   [3.5640 ms 3.5780 ms 3.5900 ms]
                        change: [+53.546% +54.133% +54.698%] (p = 0.00 < 0.10)
                        Performance has regressed.

Benchmarking Tokenizer 1M Char
Benchmarking Tokenizer 1M Char: Warming up for 3.0000 s
Benchmarking Tokenizer 1M Char: Collecting 10 samples in estimated 5.9734 s (165 iterations)
Benchmarking Tokenizer 1M Char: Analyzing
Tokenizer 1M Char       time:   [35.796 ms 35.825 ms 35.848 ms]
                        change: [+57.249% +57.379% +57.514%] (p = 0.00 < 0.10)
                        Performance has regressed.

Benchmarking Tokenizer 10M Char
Benchmarking Tokenizer 10M Char: Warming up for 3.0000 s
Benchmarking Tokenizer 10M Char: Collecting 10 samples in estimated 8.3819 s (20 iterations)
Benchmarking Tokenizer 10M Char: Analyzing
Tokenizer 10M Char      time:   [413.53 ms 413.96 ms 414.40 ms]
                        change: [+46.604% +46.947% +47.267%] (p = 0.00 < 0.10)
                        Performance has regressed.

According to the results, LTO and PGO measurably improves the project's performance.

Further steps

Post-Link Optimization (PLO) is similar to PGO technique but uses a slightly different approach. You can take a look at LLVM BOLT for more information. However, I recommend starting playing with PLO only after applying PGO - PGO is a much more stable and time-proven technology compared to PLO tools.

Thank you.

[AmrDeveloper]

Wow @zamazan4ik thats impressive, i know about BOLT and how it optimized programs like linux kernal and want to try it too,

The result from LTO + PGO looks great, can you please open PR with config for them

Thank you,
Amr Hesham

[zamazan4ik]

Yep! Let's start from LTO: #118

PGO will require more time to enable it properly