Benchmark results can vary because presets is not standardized enough on different settings

[6AKU66]

We need more reliable benchmark results for ability more accurate compare different systems.

But problems we have are:

1. Configuration preset /option preset don't fixate all performance settings affected options;
2. You can't pick preset on which you want perform benchmarking from chobby scenario menu.
3. We need to make /option preset Custom save up config if it's was changed to any another preset.

Appendix to point 1.
Options like /option losrange, /option reclaimfieldhighlight, custom player's widgets and etc.

P.S I see multiple purposes for benchmarking:

1. Compare different hardwares;
2. Compare different presets (including custom if someone's need it);
3. Compare different engines;
4. And sometimes use it not as benchmark, but rather stress-test (like i do with synthetic MT collision test).

[sprunk]

ability more accurate compare different systems.

Why is this a useful ability? You primarily need to compare different game or engine versions on the same system to check whether a change is good or if there was a regression. Comparing different systems just gives you results that are already known (newer and more expensive hardware will perform better in its respective area).

[6AKU66]

Why is this a useful ability?

Because people might plan update for them system and they would like to know how much faster new system will be. Like maybe they wanted update system to another one, but they changed them's mind because it's would give only 5-10% speed gain.

newer and more expensive

Not always.

P.S (Not related info, just sharing)
Benchmark most affecting settings for sim speed in descending order:
Particle limit (amount of particles on the battlefield)
Reclaim field hightlight
Lights (gadget explosion_light)
Grass (gadget explosion_light)

[thehobojoe]

I don't think this is a useful issue, performance differences can only be accurately tracked by running the same tests on the same hardware with the same parameters.

[6AKU66]

I think this ticket should stay open:

"by running the same tests on the same hardware"
Some people would like to compare results they get on them's hardware vs another hardware.
"with the same parameters"
This is why i created this ticket, we need to make sure that everyone running game on same graphics settings.

P.S
"performance differences can only be accurately tracked by running the same tests on the same hardware with the same parameters. "
(and i assume same software?)
How even you would get performance differences here?

[Beherith]

This benchmark is mainly for benchmarking our own systems and settings vs itself, not for comparing users to each other in a broad sense.

[Beherith]

There is too much uncontrollable variability otherwise.

[6AKU66]

It's can be both. I can remake this ticket to be more fitting one.