Evaluate Profile-Guided Optimization (PGO) and LLVM BOLT

[zamazan4ik]

Hi!

Recently I did many Profile-Guided Optimization (PGO) benchmarks on multiple projects - the results are available here. So that's why I think it's worth trying to apply PGO to htmlq. I already performed some benchmarks and want to share my results here.

Test environment

• Fedora 38
• Linux kernel 6.5.5
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Compiler - Rustc 1.73
• htmlq version: the latest for now from the master branch on commit 6e31bc814332b2521f0316d0ed9bf0a1c521b6e6
• Disabled Turbo boost

Benchmark setup

For benchmarking purposes, I use examples from the README file. Release build is done with cargo build --release, PGO optimized build is done with cargo-pgo. PGO profiles are collected from the benchmark workload itself.

All benchmarks are done multiple times, on the same hardware/software setup, with the same background "noise" (as much I can guarantee ofc).

Results

I got the following results: https://gist.github.com/zamazan4ik/4b5a78925532fe25a57534ddf8293696

According to the tests, PGO consistently improves htmlq performance at least in the README examples workload.

Further steps

I can suggest the following things to do:

• Evaluate PGO's applicability to htmlq in more scenarios.
• If PGO helps to achieve better performance - add a note to htmlq's documentation about that (probably somewhere in the README file). In this case, users and maintainers will be aware of another optimization opportunity for htmlq.
• Provide PGO integration into the build scripts. It can help users and maintainers easily apply PGO for their own workloads.

Here are some examples of how PGO is already integrated into other projects' build scripts:

• Rustc: a CI script for the multi-stage build
• GCC:
  • Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
  • A part in a "wonderful" configure script
• Clang: Docs
• Python:
  • CPython: README
  • Pyston: README
• Go: Bash script
• V8: Bazel flag
• ChakraCore: Scripts
• Chromium: Script
• Firefox: Docs
  • Thunderbird has PGO support too
• PHP - Makefile command and old Centminmod scripts
• MySQL: CMake script
• YugabyteDB: GitHub commit
• FoundationDB: Script
• Zstd: Makefile
• Foot: Scripts
• Windows Terminal: GitHub PR
• Pydantic-core: GitHub PR

After PGO, I can suggest evaluating LLVM BOLT as an additional optimization step after PGO.