Performance issue with xanmod 6.6.x

[mabod]

I was not able to fid the issue tracker for xanmod. So here I create a 'discussion' for the performance issue I found.
I compared linux-tkg 6.6.32 kernel with xanmod-lts 6.6.32 and found a 11 % performance drop with xanmod.

Both kernels where using 250 Hz and march=x86-64-v3

I have a standardized darktable benchmark where I convert a RAW file into a JPG file. The benchmark does it in two ways. First with opencl enabled and second with opencl disable on CPU only. The results are reproducible. I ran the benchmarks after a fresh boot. Always 6 runs in a row and the calculating the average time darktable spend in its "pixel pipeline".

xanmod :

opencl: average= 2.666 secs +/- 0.015 secs
cpu:    average= 4.258 secs +/- 0.026 secs

linux-tkg:

opencl: average= 2.330 secs +/- 0.011 secs
cpu:    average= 3.871 secs +/- 0.032 secs 

I did not expect such a big difference. Any idea why the performance of xanmod-lts is so much slower?

PS:
CPU= AMD Ryzen 9 5900X with 64 GB RAM
GPU= Radeon RX 6650 XT

[xanmod]

Xanmod has all cpu mitigations enabled by default, linux-tkg too?
You can disable it add mitigations=off in boot kernel parameter.

[mabod]

linux-tkg has also all mitigations turned on.

From linux-tkg config:

CONFIG_CPU_MITIGATIONS=y
CONFIG_MITIGATION_RFDS=y
CONFIG_MITIGATION_SPECTRE_BHI=y
CONFIG_ARCH_CONFIGURES_CPU_MITIGATIONS=y
CONFIG_TULIP_NAPI_HW_MITIGATION=y

[xanmod]

https://github.com/Frogging-Family/linux-tkg/blob/master/linux-tkg-patches/6.9/0013-optimize_harder_O3.patch

Tkg compiles the entire kernel with -O3, which is probably the reason for the performance gain. Today I no longer use this cc flag due to possible stability and latency problems. Torvalds doesn't recommend it either.
Now, just like ClearLinux, I only apply -O3 to the x86 core that has been extensively tested.

[mabod]

Tkg compiles the entire kernel with -O3

I doubt that -O3 is the reason. when I do the benchmark with the arch lts kernel I get similar results like with linux-tkg:

6.6.32-1-lts:

opencl: average= 2.351 secs +/- 0.017 secs
cpu:    average= 3.935 secs +/- 0.043 secs

There is only a 1-2 % difference to linux-tkg. There must be something else fundamentally different in the xanmod kernel.

[xanmod]

Try echo 2 | sudo tee /proc/sys/kernel/yield_type

[mabod]

tee: /proc/sys/kernel/yield_type: No such file or directory

[mabod]

Now I see. Only the xanmod kernel has /proc/sys/kernel/yield_type. But the value does not make a difference. with '2' I get similar results:

xanmod with yield_type=2

opencl: average= 2.659 secs +/- 0.013 secs
cpu:    average= 4.300 secs +/- 0.177 secs

What is yield_type for?

[xanmod]

I built this image with more optimizations that I have been testing for a while.
https://dl.xanmod.org/rc/6.6.32-xanmod2-rc1/

[mabod]

I want to add one more piece of information for this test:
My PC starts a vmware during boot. This is running in the background with a debian nextcloud installation. Of course the nextcloud is idle. That is for sure. But it is idle in the background all the time.
I am wondering if xanmod gives this vm more attention than a stock arch kernel or linux-tkg. This would take away resources from darktable. Can that be?

EDIT:
I tested without nextcloud VM and the result is the same. The performance drop has nothing to do with the enxtcloud VM

[mabod]

In the meantime have also tested linux-xanmod 6.9.3 and the results are similar:

opencl: average= 2.622 secs +/- 0.034 secs
cpu:    average= 4.390 secs +/- 0.072 secs

[mabod]

I have repeated the benchmark with kernel 6.6.58 and there is no doubt, that the xanmod kernel makes darktable slower by 16-17 % compared to the arch kernel or the linux-tkg kernel. And this is even true for opencl and not just CPU load.

kernel                      opencl          CPU
----------------------------------------------------------
6.6.58-273.1-tkg-eevdf      2,3 s           3,7 s
6.6.58-x64v3-xanmod1-1-lts  2,7 s  (+17 %)  4,3 s (+16 %)
6.6.58-1-lts                2,3 s           3,8 s

This is how I do the benchmark:

I am using darktable and the debug output of darktable tells me how much time it spends in the "pixel pipeline". That is basically how much time it needs to convert a picture from RAW to jpg. I am doing this benchmark with opencl support enabled, running on my AMD GPU, and with opencl support disabled, purely running on the CPU.

The darktable command is:

darktable-cli bench.SRW /tmp/test.jpg --core --disable-opencl -d perf -d opencl --configdir /tmp
(bench.SRW and the sidecar file can be found here: https://drive.google.com/drive/folders/1cfV2b893JuobVwGiZXcaNv5-yszH6j-N )

The line to look for in the debug output is:

4,2765 [dev_process_export] pixel pipeline processing took 3,811 secs (81,883 CPU)

The results are very surprising to me. Why is darktable running slower on xanmod. Any idea?