Darktable speed

[Karl]

FWIW I bought a 4k monitor last April, and there was no noticeable slowdown for me. I do run the Nvidia proprietary drivers though.

[chiraag (ಚಿರಾಗ್ ನಟರಾಜ್)]

Huh, interesting. I switched to using the proprietary drivers, but still had the slowdown in lighttable (obviously using OpenCL will speed things up in actual processing). I confirmed it was the 4K resolution because switching down to 1920x1080 fixes things and everything’s back to the speed I was used to.

[Karl]

Weird. Maybe a difference in desktop environment? I run Slackware (KDE desktop). Maybe that has something to do with it?

Around a year ago I recall reading a post somewhere about high CPU usage in lighttable view, but I can’t find it now. Perhaps it’s the same issue you’re experiencing? Simply moving the mouse around the lighttable does use a lot of CPU on my machine (~40% of one core) but it’s not enough to cause a noticeable slowdown.

Edit: and on reading the rest of the comments here it looks like someone else has mentioned the post I couldn’t fnd!

[chiraag (ಚಿರಾಗ್ ನಟರಾಜ್)]

For those who are interested and have a similar problem, here is my current workaround. Since the problem is the 4K resolution, I created a script which sets the right resolution (just half in each dimension — e.g. my regular display is 3840x2160, so my “half” resolution is 1920x1080), starts darktable (along with resetting GDK_SCALE so that the interface isn’t gigantic), and cleans up by restoring the original resolution (--preferred argument to xrandr) after I exit darktable. This way, I get the awesome 4K display most of the time as well as responsive photo editing when I want it!

Of course, this workaround is most easily implemented in linux (is there even a command-line tool to set the screen resolution in Windows or macOS?). Here is the script I’m using:

#!/bin/bash

OUTPUT="eDP-1-1"
HALF_RES="1920x1080"
HALF_RES_DPI="96"
HIGH_RES_DPI="192"
DARKTABLE='firejail darktable --noiseprofiles ~/.config/darktable/noise/presets.json'

# Change resolution

xrandr --output $OUTPUT --mode $HALF_RES --dpi $HALF_RES_DPI;

# Start darktable

GDK_SCALE="" $DARKTABLE "$@";

# Change resolution back

xrandr --output $OUTPUT --preferred --dpi $HIGH_RES_DPI;

(I use firejail to sandbox programs, but that’s irrelevant — you can just as easily remove that and just start darktable "$@".)

[Tobias]

Have you tested, how much firejail low down darktable?

[mimoklepes]

Hi chiraag, thanks for an interesting workaroud. I may use this when working on my 15-inch laptop (which isn’t often), but my main desktop display is a 31-inch one, so unfortunately 1920x1080 would be too huge and ugly.

Weird thing though - with resolution set to 1920x1080 lighttable seems a bit snappier than on full 4K with darktable window resized to one quarter of the screen (which should result to about the same number of the redrawn pixels). Maybe something else is in play as well.

[chiraag (ಚಿರಾಗ್ ನಟರಾಜ್)]

@Tobias

Have you tested, how much firejail low down darktable?

firejail does not slow down darktable at all (one of the first things I tried was running darktable without firejail - it didn’t help). I’ve never noticed a performance impact of firejail on any application (including darktable).

[chiraag (ಚಿರಾಗ್ ನಟರಾಜ್)]

@mimoklepes

Hi chiraag, thanks for an interesting workaroud. I may use this when working on my 15-inch laptop (which isn’t often), but my main desktop display is a 31-inch one, so unfortunately 1920x1080 would be too huge and ugly.

What is your native resolution? I use awesome window manager, so if I hide the top bar, the only indication that anything’s changed is that the wallpaper looks horrendous. But given that I reset it right back after darktable ends, as long as I don’t switch applications, I don’t think anything should be too noticeable? I think anything that’s DPI-aware shouldn’t have too many problems (although you would have to reset e.g. GDK_SCALE if you want to start a new program while darktable is running).

@mimoklepes

Weird thing though - with resolution set to 1920x1080 lighttable seems a bit snappier than on full 4K with darktable window resized to one quarter of the screen (which should result to about the same number of the redrawn pixels). Maybe something else is in play as well.

Huh, that is weird. I wouldn’t really know since I use a tiling window manager (so I don’t normally resize clients manually).

[Pascal_Obry]

Driver Nouveau has poor perfs on 4K display maybe? Have you tried the proprietary Nvidia driver?

[chiraag (ಚಿರಾಗ್ ನಟರಾಜ್)]

@Pascal_Obry

Driver Nouveau has poor perfs on 4K display maybe? Have you tried the proprietary Nvidia driver?

1. I was initially running this on my on-board GPU (Intel).
2. No other application had problems with 4K, either on nouveau or intel.
3. I switched to running the Nvidia driver because I wanted OpenCL.
4. Neither of those had an effect on darktable’s issue with 4K monitors. Changing the resolution did.

[mimoklepes]

My native resolution is 4096x2160. I wasn’t able to set the resolution to exactly half (2048x1080), looks like my GPU card does not support it. But anyway, I tested with 1920x1080 and the pixels are just too large on the 31 inch display - the fonts look ugly etc. Also I often switch windows from darktable to a file manager etc, so unfortunately this solution is not suitable for me.

FYI I’m using Radeon RX-480 with the AMDGPU-PRO driver.

[chiraag (ಚಿರಾಗ್ ನಟರಾಜ್)]

@mimoklepes

My native resolution is 4096x2160. I wasn’t able to set the resolution to exactly half (2048x1080), looks like my GPU card does not support it.

Damn, that’s a pity.

@mimoklepes

But anyway, I tested with 1920x1080 and the pixels are just too large on the 31 inch display - the fonts look ugly etc. Also I often switch windows from darktable to a file manager etc, so unfortunately this solution is not suitable for me.

Yeah, that makes sense. For me, if I’m in darktable, I often don’t leave it until I’m done editing my photos. Here’s to hoping performance is fixed on 4K monitors in the near future

[paperdigits (Mica)]

@chiraag

Here’s to hoping performance is fixed on 4K monitors in the near future

To fix it, we’d need to know exactly what is wrong, that it is actually a problem in darktable and not in another library in debian sid or a gtk issue. So best as I can see, there isn’t information this precise, but sounds like you’re narrowing it down.

Have you tried to run strace yet? What version of GTK is running in sid?

[LebedevRI (Roman Lebedev)]

@paperdigits

To fix it, we’d need to know exactly what is wrong, that it is actually a problem in darktable and not in another library in debian sid or a gtk issue.

It’s https://redmine.darktable.org/issues/10764.
That is a true, valid issue. That code is … retarded, not in a good way.

[chiraag (ಚಿರಾಗ್ ನಟರಾಜ್)]

@mimoklepes

I believe that the sluggish response in lighttable is related to this issue: https://redmine.darktable.org/issues/10764 - darktable is redrawing all visible images on even the smallest mouse movements. It’s just much more noticeable on HiDPI displays, since it has to redraw more pixels. Unfortunately it seems this didn’t get much love yet, probably the impact on the user base is not that big.

Most likely it’s tied to this (especially since changing the resolution largely fixes the problem). I can certainly upload strace logs here or to that bug report…which is preferred? The current version of GTK+ is 3.22.26 in sid, but I’m pretty sure (looking at the changelog) that there hasn’t been an update to libgtk-3-0 since November (and my old laptop died towards the end of January).

[chiraag (ಚಿರಾಗ್ ನಟರಾಜ್)]

@LebedevRI

That is a true, valid issue. That code is … retarded, not in a good way.

I’d love to help resolve this issue in any way I can What needs to happen for progress to be made on this? Will this require refactoring a large portion or is this a fairly self-contained issue?

[paperdigits (Mica)]

I stand corrected, sorry @chiraag! Thanks @LebedevRI

[Pascal_Obry]

To assess if this is really only the redraw lightable issue you may want to try this branch:

https://github.com/TurboGit/darktable/tree/faster-lighttable

Note that this is not a proper fix (has some redraw issue IIRC) and it won’t be merged. But if testing this code you do not see anymore the sluggishness of the lighttable then we are talking about the same issue.

[mimoklepes]

Thanks for this suggestion, I’ve compiled the faster-lighttable branch and tested. There is some change, but unfortunately it didn’t solve the slow responsiveness problem. Although it helped with the high CPU usage when mouse moves only within the same thumbnail:

On regular dt 2.4.1 log like this (with “-d all”) keeps repeating whenever mouse moves inside one thumb:

[sql] /build/darktable-CEnZBI/darktable-2.4.1/src/common/image.c:632, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0244 sec
[sql] /build/darktable-CEnZBI/darktable-2.4.1/src/common/image.c:632, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0248 sec
[sql] /build/darktable-CEnZBI/darktable-2.4.1/src/common/image.c:632, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0284 sec
[sql] /build/darktable-CEnZBI/darktable-2.4.1/src/common/image.c:632, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0305 sec
[sql] /build/darktable-CEnZBI/darktable-2.4.1/src/common/image.c:632, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0270 sec
[sql] /build/darktable-CEnZBI/darktable-2.4.1/src/common/image.c:632, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0253 sec
[sql] /build/darktable-CEnZBI/darktable-2.4.1/src/common/image.c:632, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0249 sec
[sql] /build/darktable-CEnZBI/darktable-2.4.1/src/common/image.c:632, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0245 sec
[sql] /build/darktable-CEnZBI/darktable-2.4.1/src/common/image.c:632, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0134 sec
[sql] /build/darktable-CEnZBI/darktable-2.4.1/src/common/image.c:632, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0134 sec
[sql] /build/darktable-CEnZBI/darktable-2.4.1/src/common/image.c:632, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0141 sec
[sql] /build/darktable-CEnZBI/darktable-2.4.1/src/common/image.c:632, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0156 sec
[mipmap_cache] thumbs fill 98,88/2048,00 MB (4,83%)
[mipmap_cache] float fill 0/4 slots (0,00%)
[mipmap_cache] full  fill 1/4 slots (25,00%)
[mipmap_cache] level | near match | miss | stand-in | fetches | total rq
[mipmap_cache] thumb |   3,23% |   3,23% |   -nan%  |   0,00% | 100,00%
[mipmap_cache] float |   -nan% |   -nan% |   -nan%  |   0,00% |   0,00%
[mipmap_cache] full  |   -nan% |   -nan% |   -nan%  | 100,00% |   0,00%


[lighttable] expose took 0,2732 sec

With faster-lighttable log changes to only this (again, repeating while mouse moves):
[lighttable] expose took 0,0000 sec

Also the CPU consumption went down from ~200% to ~40% (on a quad-core intel i5). This does not affect the user experience much, but indeed lowers the overall CPU usage.

Hovever when moving between thumbs (with mouse or with an arrow key) the CPU usage is still around 200% and this is logged once on every jump to a neighbouring thumb:

[sql] /home/mimo/tmp/dt/darktable/src/libs/metadata_view.c:222, function _metadata_view_update_values(): prepare "SELECT imgid FROM main.selected_images LIMIT 1"
[sql] /home/mimo/tmp/dt/darktable/src/common/image.c:151, function dt_image_film_roll(): prepare "SELECT folder FROM main.film_rolls WHERE id = ?1"
[sql] /home/mimo/tmp/dt/darktable/src/common/image.c:196, function dt_image_full_path(): prepare "SELECT folder || '/' || filename FROM main.images i, main.film_rolls f WHERE i.film_id = f.id and i.id = ?1"
[sql] /home/mimo/tmp/dt/darktable/src/common/metadata.c:179, function dt_metadata_get_xmp(): prepare "SELECT value FROM main.meta_data WHERE id = ?1 AND key = ?2 ORDER BY value"
[sql] /home/mimo/tmp/dt/darktable/src/common/metadata.c:179, function dt_metadata_get_xmp(): prepare "SELECT value FROM main.meta_data WHERE id = ?1 AND key = ?2 ORDER BY value"
[sql] /home/mimo/tmp/dt/darktable/src/common/metadata.c:179, function dt_metadata_get_xmp(): prepare "SELECT value FROM main.meta_data WHERE id = ?1 AND key = ?2 ORDER BY value"
[sql] /home/mimo/tmp/dt/darktable/src/common/tags.c:351, function dt_tag_get_attached(): prepare "SELECT DISTINCT T.id, T.name FROM main.tagged_images AS I, data.tags AS T WHERE I.imgid IN (SELECT imgid FROM main.selected_images) AND T.id = I.tagid ORDER BY T.name"
[sql] /home/mimo/tmp/dt/darktable/src/common/image.c:675, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0248 sec
[sql] /home/mimo/tmp/dt/darktable/src/common/image.c:675, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0246 sec
[sql] /home/mimo/tmp/dt/darktable/src/common/image.c:675, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0256 sec
[sql] /home/mimo/tmp/dt/darktable/src/common/image.c:675, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0274 sec
[sql] /home/mimo/tmp/dt/darktable/src/common/image.c:675, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0268 sec
[sql] /home/mimo/tmp/dt/darktable/src/common/image.c:151, function dt_image_film_roll(): prepare "SELECT folder FROM main.film_rolls WHERE id = ?1"
[sql] /home/mimo/tmp/dt/darktable/src/common/image.c:196, function dt_image_full_path(): prepare "SELECT folder || '/' || filename FROM main.images i, main.film_rolls f WHERE i.film_id = f.id and i.id = ?1"
[sql] /home/mimo/tmp/dt/darktable/src/common/metadata.c:179, function dt_metadata_get_xmp(): prepare "SELECT value FROM main.meta_data WHERE id = ?1 AND key = ?2 ORDER BY value"
[sql] /home/mimo/tmp/dt/darktable/src/common/metadata.c:179, function dt_metadata_get_xmp(): prepare "SELECT value FROM main.meta_data WHERE id = ?1 AND key = ?2 ORDER BY value"
[sql] /home/mimo/tmp/dt/darktable/src/common/metadata.c:179, function dt_metadata_get_xmp(): prepare "SELECT value FROM main.meta_data WHERE id = ?1 AND key = ?2 ORDER BY value"
[sql] /home/mimo/tmp/dt/darktable/src/common/tags.c:333, function dt_tag_get_attached(): prepare "SELECT DISTINCT T.id, T.name FROM main.tagged_images AS I JOIN data.tags T on T.id = I.tagid WHERE I.imgid = 4430  ORDER BY T.name"
[sql] /home/mimo/tmp/dt/darktable/src/common/image.c:675, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0249 sec
[sql] /home/mimo/tmp/dt/darktable/src/common/image.c:675, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0245 sec
[sql] /home/mimo/tmp/dt/darktable/src/common/image.c:675, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0245 sec
[sql] /home/mimo/tmp/dt/darktable/src/common/image.c:675, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0135 sec
[sql] /home/mimo/tmp/dt/darktable/src/common/image.c:675, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0135 sec
[sql] /home/mimo/tmp/dt/darktable/src/common/image.c:675, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0170 sec
[sql] /home/mimo/tmp/dt/darktable/src/common/image.c:675, function dt_image_altered(): prepare "SELECT operation FROM main.history WHERE imgid = ?1"
[lighttable] image expose took 0,0138 sec
[mipmap_cache] thumbs fill 98,88/2048,00 MB (4,83%)
[mipmap_cache] float fill 0/4 slots (0,00%)
[mipmap_cache] full  fill 1/4 slots (25,00%)
[mipmap_cache] level | near match | miss | stand-in | fetches | total rq
[mipmap_cache] thumb |   1,79% |   1,79% |   -nan%  |   0,00% | 100,00%
[mipmap_cache] float |   -nan% |   -nan% |   -nan%  |   0,00% |   0,00%
[mipmap_cache] full  |   -nan% |   -nan% |   -nan%  | 100,00% |   0,00%


[lighttable] expose took 0,2702 sec

So it seems that the actual slow-down is caused by “expose” when skipping to next thumb, not the constant redraw. Quarter of a second may not seem like much, but when mouse moves quickly by several thumbs the highlight can take up to few seconds to catch up.

Is this something that can be optimized, or maybe influenced by some configuration? Thanks!

[Pascal_Obry]

Thanks for the experiment. I’m not sure there configuration option to optimize this.