darktable performance regression with kernel 6.7 and newer.

[mabod]

@priort

This is what I was alluding to earlier…I wonder if you had a version compiled on your PC using any available CPU enhancements specific to your hardware if you would see something different over the generic one that you likely have now???

Yes, I also did compile my own darktable binary with “-march=x86-64-v3 -mtune=generic”
The effect is the same. The binary is a little faster in general but the performance difference between kernels stays.

[paperdigits (Mica)]

Have the compile flags changed between kernel versions?

[mabod]

Of course not. It was always the same binary compared between kernel versions. Apples with Apples.

Anyways, this discussion is going in the wrong direction. This performance degradation has nothing to do with compiler flags.

[mabod]

@paperdigits

It’s an AMD

Same with me: AMD RYZEN 9 5900X

[mabod]

I have opened a bug report for the kernel devs: 219366 – Performance regression from kernel 6.6 to 6.7 and newer

Lets see what is coming out of that.

[paperdigits (Mica)]

@mabod

Anyways, this discussion is going in the wrong direction. This performance degradation has nothing to do with compiler flags.

Then point it in the right direction.

You don’t get to criticize the people using their free time to try and help you resolve your issue without at least also advancing the issue forward.

[mabod]

@paperdigits

Then point it in the right direction.

You don’t get to criticize the people using their free time to try and help you resolve your issue without at least also advancing the issue forward.

I did not criticize people. I was just saying that a discussion about compiler flags is going in the wrong direction. The performance degradation is independent of compiler flags.

I opened this thread on advice of the darktable devs to get more evidence about the performance degradation. I am looking for darktable users doing the test and confirming the findings.

So far only @reox has reproduced the test. I am thankful for that. I would have wished for more participation, but ok.

[paperdigits (Mica)]

The next step is to bisect the kernel and find the exact version that introduced the regression, then from there, bisect down to the commit that caused it and then raise that patch to the kernel maintained.

[mabod]

Funny!

I am not a developer. I can compile my own darktable with the arch tools, because that is easy. But I am certainly not bisecting the kernel. This is beyond my payroll.

I am just a darktable user experiencing a performance regression. And I am reporting this. That is it. I was hoping that other darktable users reproduce the findings with only little effort. At the end of the day this has to go back to the developers.

And if there is no kernel or darktable developer picking this up I can not help it. Then we all have to live with this performance regression.

[paperdigits (Mica)]

@mabod

This is beyond my payroll.

There is no payroll. Literally we are all volunteers.

[mabod]

@paperdigits

There is no payroll. Literally we are all volunteers.

Please! “This is beyond my payroll / pay grade” is just a saying.

beyond my pay grade

[paperdigits (Mica)]

@mabod

Please! “This is beyond my payroll / pay grade” is just a saying.

Yes I know.

[paperdigits (Mica)]

Can you try mitigations=off in the grub kernel options? That’ll tell us if its some security fix that slows everything down.

[mabod]

I just did this. I verified with spectre-meltdown-checker that the PC is vulnerable.

It has no effect. With kernel 6.10.13 I am getting 4.734 s in the pixel pipeline, which is similar to what I reported for kernel 6.10.6 in my opening post.

[paperdigits (Mica)]

Can you provide me exact kernel versions, if I can reproduce this on my intel laptop, I will probably try and bisect; sadly my AMD cpu compuers are too new to run with 6.6 it seems.

[mabod]

I tested and confirmed this behavior with several kernel version.

Fast kernels:
6.5.13
6.6.51 / 52 / 53 / 54

Slow kernels:
6.7.1 / 12
6.10.6 / 10 / 11 / 12 / 13

I tested out of the box arch kernels as well as self compiled linux-tkg kernels. All combinations I have tested confirm the performance degradation.

If you do a bisect I suggest you start with slow kernel 6.7.1. This is the first kernel I have tested which is showing the performance degradation.

[reox]

@mabod

Fast kernels:
6.5.13
6.6.51 / 52 / 53 / 54

Slow kernels:
6.7.1 / 12

Almost there! Thus, the slowness was introduced either in 6.7.0 or 6.7.1

[paperdigits (Mica)]

6.6.54 and 6.11.2 both show ~23 seconds for atrous on my 8th gen Intel laptop, so I could not reproduce.

[priort (Todd Prior)]

Am I missing something…this seems overly trivial and clearly the expectation or most common practice is that DT is using the opencl code on a GPU. In any serious use the program, there I believe there is no real issue… this just seems like a big song and dance about a scenario that in reality is rare ie running a CPU only pipeline. Its perhaps curious to note but this is not impacting DT in a major way or am I missing something…

[priort (Todd Prior)]

I came across several posts going back to different kernel versions and there were quite a few instances where people noted that they saw a regression with a new or updated kernel but it was corrected by compiling the kernel from “pure sources” was the term used… rather than taking one provided… again it might not be the fix as this is one module and not the whole pipeline…