[SOLVED][Performance git-master] darkroom much less performant, dt-2.7 vs. dt-2.6.2

Dear Ladies and Gentlemen,
boyz ‘n’ gals,

I do think, there is a performance issue in darkroom with current git-master of darktable.

Hence I did this bug-report
https://github.com/darktable-org/darktable/issues/2484

I would like to open this new topic here for casual talk, awareness and share experiences, while keeping the issue on github for the developers and solution oriented topics.

I hope that split is appreciated and could be implemented?

Just when I was on the way to edit this topic, I found the announcement from @aadm with regards to his performance issues and re-build of the lib and database. However, I did try to reproduce with empty or new ~/.config/darktable and the “feeling” is the same…

Credits:
Thank you @Claes for your always very welcomed advices :slight_smile:

Remember that rebuilding the database does not mean starting from “an empy or new darktablerc”, you need to physically delete or move the library.db files so that DT will create a new database.

Or you can follow the manual instructions to launch DT with option --library <library file>:

darktable keeps image information in an sqlite database for fast access. The default location of that database file is “$HOME/.config/darktable/library.db”. You may give an alternative location, e.g. if you want to do some experiments without compromising your original library.db. If the database file does not exist, darktable creates it for you. You may also give “:memory:” as a library file in which case the database is kept in system memory – all changes are discarded when darktable terminates.

I have read your issue report on github and while I don’t have time to make yet another test, what you describe does feel similar to what I was experiencing, i.e. overall a less responsive system on 2.7 than 2.6.2. I am now missing some little features in 2.7 like the overlay of exif data on top of the images etc., but overall this (2.6.2) feels better and faster than ever.

1 Like

@aadm
There was a typo, now corrected.

I meant totally new conf dir

Here are some fresh timings, to back the claims made by @AxelG:

Between every change of dt version, I deleted all traces of darktable in .config as well as in .cache.

Have fun!
Claes in Lund, Sweden

1 Like

Hey guys,

I just tested this PR 2486 locally thanks to the strong support of @Edgardo_Hoszowski (I assume is the same than @edgardoh on github). I did a very first quicky, as you can see here

@Claes could you imagine, to go into this as well. I am a bloody newbie but with that step by step guidance (Again, thanks for that), it was foolproof :slight_smile:

UPDATE 1
That PR 2486 might get split up, as it addresses two issues, in order to bring the improvement faster. So pls. stay tuned on github

Hello all together,

due to the strong support of our wonderful devs, from dt-2.7+1099 this issue seems to be solved from my point of view (still I can see differences in different approaches for the solution, but I am not at all in the position to judge, why and how to decide the technologies used).

So GREAT THANK YOU to (alphabetical order):
@anon41087856
@Edgardo_Hoszowski
@Pascal_Obry

Maybe @aadm could backup his config-dir and give it one more try (if you don’t edit productively, you can easly go back) :smile:

Yes, current master reacts way faster now and feels snappy on my side again.
Thanks!

tempted but a bit scared to waste even more productive time… I have a backlog of images to process that will take months of continuous work (quite impractical for all purposes)…

@aadm Understand you very well. I also edited a lot with 2.7 and hence cannot go back, as 2.6.2 cannot read the XMP files Version 3

Me I’m fine now, as dt reached a level currently wich I can live with. Sometimes pain is necessary to move forward :relaxed:
I learned so a lot…

Also you could run 2.7 with - - config dir for trials :innocent:

I did some performance comparison between the recent dt-2.7 versions

…maybe someone is interested…

@pk5dark could you do me a favour and compare the instantaneous reaction between master and 2.6.2 when you grab the WB slider with the mouse and track it for and back? For me, 2.6.2 is still more responsive, means less lag in screen update…

I can’t promise that I will find time to do this as I need to get 2.6.2 running before. For me draging the WB slider in current git master is fast enough, but I played with the opencl settings before to get rid of the slugish behaviour. IRC opencl_sync_cache made a difference if changed.

Do you use the same opencl settings for your tests in darktablerc for 2.6.2 and 2.7.0?

@pk5dark
you are brilliant, that was it!
as 2.6.2 do not support (AFAIK) opencl_synch_cache=active module, so I set it there to opencl_synch_cache=false

Changing to false made the trick in 2.7

I wonder, should I issue a bug report? For this I have two suspicious topics:

  • just now I googled after “opencl_synch_cache=active module” and surprisingly I found redmine is still acvtive, I thought everything moved on to github
  • If I remember right, *
    just now I googled after “opencl_synch_cache=active module” got introduced somwhere on github, but with google I cannot find it anymore, otherwise I would just comment at that place…

I found it on github #2230

@pk5dark
in this respect, what did you mean with IRC?

So yes, setting it to opencl_synch_chache=active module costs some performance when editing something with a short pixlepipe (probably logic) but setting it to false is also not a solution. If one maybe wanna edit “levels” at the end of the que in such heavily edited pic.

Here “active module” really gains a lot (at least sometimes, in other times you need to switch the module off and on first)
:slight_smile:

So compiled 2.6.2 and running beside 2.7 gives me identical response time on moving the WB slider. Adding some instance of profiled denoise and equalizer makes no difference. Using retouch healing is slow in both versions. So no differences for me …

Should have been IIRC :innocent: (if i remember correctly)

@pk5dark Thank you very much!

Which opencl_synch_cache setting you used?

Hardware spec?
Here is core I9 9900K no OC, 32GB 3333MHz, 2x GTX1060

I have been lurking with interest. On inspecting my darktablerc, which I have never tweaked, I discovered:
opencl_sync_cache=false
and
opencl_omit_whitebalance=

I’d be happy to tweak away at any or all of the opencl settings, but is there documentation that gives a commentary on what possible values the individual options have, and - indeed - what the
values mean? For example:
opencl_device_priority=*/!0,*/*/*
opencl_enable_markesteijn=true

Again, the numeric values are probably defaults either for a generic hardware, or - possibly I guess - defaults chosen with the detected hardware in mind.

Many configuration files are peppered with comments - often the comments comprise 4 or 5 lines per setting. This one has not a single comment, nor a nod to a documentation file.

It is nice to have the popup bubble helptext when using the darktable preferences gui, although this has on occasion been cryptic. I’d love to see this level of commentary for all option lines in darktablerc.

@martin.scharnke

for opencl_synch_cache=active module (I ever wonder is sync nor synch?) AFAIK is new in 2.7 and you can find some things here on github

Other opencl tweaks are described here: usermanual/en/darktable_and_opencl_optimization and the following pages

That one is about having more than one GPU and you find things here: usermanual/en/darktable_and_opencl_multiple_devices

A quick googling on this also didn’t bring anything to me :slight_smile:

currently dt seems to have a function, where all masked entries in darktablerc will be moved to the top and the rest gets sorted, so they are in order.

I support your idea, in that case better keep the settings grouped.

Have you ever considered to file a feature request on github?

@pk5dark
I hope I did the right thing in filing this bug report

It is really not easy to describe, what I experiance there… :slight_smile:

system i9900k + GeForce GTX 1060 6GB

settings for 2.6.2 (default as I used a new config dir)

opencl_async_pixelpipe=false
opencl_avoid_atomics=false
opencl_checksum=2160716749
opencl_device_priority=*/!0,*/*/*
opencl_disable_drivers_blacklist=false
opencl_library=
opencl_mandatory_timeout=200
opencl_memory_headroom=300
opencl_memory_requirement=768
opencl_micro_nap=1000
opencl_number_event_handles=25
opencl_scheduling_profile=default
opencl_size_roundup=16
opencl_synch_cache=false
opencl_use_cpu_devices=false
opencl_use_pinned_memory=false

for 2.7

opencl_async_pixelpipe=FALSE
opencl_avoid_atomics=FALSE
opencl_checksum=2160716749
opencl_device_priority=0/!0,*/*/*
opencl_enable_markesteijn=true
opencl_library=
opencl_mandatory_timeout=200
opencl_memory_headroom=0
opencl_memory_requirement=768
opencl_micro_nap=100
opencl_number_event_handles=175
opencl_omit_whitebalance=
opencl_runtime=
opencl_scheduling_profile=very fast GPU
opencl_size_roundup=16
opencl_synch_cache=active module
opencl_use_cpu_devices=false
opencl_use_events=FALSE
opencl_use_pinned_memory=FALSE