Darktable eats all memory, system freeze

Software darktable
1

Today, I imported 13k images to my library, some of them are very large scans (50 MB). The problem is, darktable crashes at some point, but not always after the same time. It is a new problem. I think it is a memory problem, because the whole system freezes. I replicated the issue with a memory monitoring program open, so I could confirm my suspicion of memory problems. And yes, I could see, at a certain point within 3 or 4 seconds darktable is rapidly eating all memory resources, then filling the swap file, then my system freezes.

Help, I have never had this kind of behaviour.
Maybe it has to do with thumbnailing files that are too big for memory, because the issue mainly occurs in the lighttable. But shouldn’t these files be split up and tiled?

The attached PDF file contains the console output of flatpak run org.darktable.Darktable -d opencl -d perf -d verbose -d common

My platform is Linux Mint, 16GB Memory, 2GB Swap file, 4GB GPU memory.
I would be glad if anyone could point me to a solution so I can use my beloved darktable again.
darktable_crash.pdf (87.5 KB)

2

I don’t see an actual crash in the log you’ve provided.

What if you add like 2k images at a time?

3

Under preferences/processing, what are your resources set to?
https://darktable-org.github.io/dtdocs/en/preferences-settings/processing/#cpu--memory

4

Your swap file should be twice your system memory, so in this case 32GB

5

I am not sure that this is necessary in most circumstances these days. I have 32GB main memory, and my system never uses swap.

This case looks like an exception, while the swap is low, I am not convinced that increasing it would solve the problem.

6

It’s likely the oom killer stepping in

3 Likes
7

For disk swap this is true, but we have far better technologies now.

IMO the best action here is to setup zram. It has a small CPU performance penalty but it’s not a lot and can effectively provide you with 8GB+ of usable memory. Some people even run twice their system memory in zram without issues.

8

Hello, thank you for asking. The resources are set to ‘large’.

9

The problem is not about importing the images, that succeeded fairly quickly. now after the import, the system freezing occurs. And there is no crash because it is a freeze, and yes, sometimes the oom killer steps in and other times I have to use a ‘Magic Key Sequence’ to manually start the oom killer. (Alt + PrintScreen + F)

1 Like
10

For sure I can increase my available RAM, by either method, but I wonder what is causing this condition. usually when memory is too small for the task at hand, tiling the image should take place instead of freezing the system. I am not very willing to do this workaround even if it might help me, as I am intrigued by the nature of this problem.

1 Like
11

Try default or small and see if the problem goes away.

12

It could be that background generation of thumbnails triggers a memory leak.
Also, try -d memory as a debug option - and, if you are on Linux, use > /path/to/debug.txt to send the output directly to a file, then attach that, instead of a PDF.

So, for example:
darktable -d common -d memory > /tmp/debug.txt

3 Likes
13

Thank you for this suggestion.
Here is the output of -d common -d memory:
darktable_log2.txt (120.0 KB)

I suspect the background thumbnail generation to be at fault as well.

14

I tried it with default resource level, and I noticed one obvious effect: The memory overload took longer to occur (128 seconds vs. 55 seconds). Here is the log, maybe you can find more hints in there:
darktable_log3.txt (91.4 KB)

15

You can try and run darktable-generate-cache as a work around for now. That’ll generate your thumbnails.

16

Did you try with “Preferences → Lighttable → Thumbnails → Generate thumbnails in background” disabled ?