Less than optimal CPU usage

Software RawTherapee
1

Hello!
I have just started to use RawTherapee and I’m quite blown away by the functions available in the program.

However when batchprograssing it seems to only use about 38% of the CPU. My system is Windows 10, 12 core Ryzen 3900x, 64Gb ram, read/write to fast NVMe disk capable of 1bout 1.5Gb/s. I’m reading Nikon NEF, and processing to uncompressed .TIFF

My Preferences->Performance->Threads is 0. Program seems to use all cores, but only about 1/3 of full capacity. They all do have some spikes, followed by some idle time. If I use more advanced functions in program, the CPUload seems to go down. If doing simple NEF->TIFF conversion, I get about 60% cpu load.

Why does it not go at full throttle and is it possible to force more than 24 threads?

2

Can you share a .pp3 file you used?

From the informations you gave, at least 2 steps are only single threaded, which are the decoding of the NEF and the writing to TIFF.

Another operation which does not use all cores is Tonemapping.

3

thanks for quick answer!
Please check the included .pp3.

Is it possible to execute queue by running several instances of RT from CMD? Maybe several queues and start each queue as parameter to RT?

image-000729.nef.pp3 (12.3 KB)

4

Hello @heckflosse

single threaded which are the decoding of the NEF and the writing to TIFF.

Just out of curiosity, since I am a Nikon user as well :slight_smile:
The multi-threading decoding of NEFs files is not implemented because it is not possible to do so or, more likely, because nobody coded this option so far (due to lack of time, interest etc)?

5

Because it’s quite hard to implement a parallel Huffman decoding. Google parallel huffman decoding…

1 Like
6

@marz71 In your pp3 Tonemapping should be the cuplrit. As I wrote above it uses not all cores of your cpu.

Not by using the queue, but by using rawtherapee-cli from for example msys2 command line like this:

OMP_NUM_THREADS=6 rawtherapee-cli ...
OMP_NUM_THREADS=6 rawtherapee-cli ...
OMP_NUM_THREADS=6 rawtherapee-cli ...
OMP_NUM_THREADS=6 rawtherapee-cli ...

This way you could start 4 instances of rawtherapee-cli each of them using up to 6 threads. That should put more load on your cpu

7

ah, do I need to process each and every file or will the 4 instances process all my files?
If I need to process one and one file, maybe better to use parallel from msys2? (I have never used msys2, be see there is a msys2 parallel package)

8

If you want to process the files this way, you need to split them into different folders and then use rawtherapee-cli on each of this folders in parallel.

Use rawtherapee-cli -h to get help for the parameters.

You can do this also with windows command-line instead of msys2 command line, but for windows command line I do not know how to pass the OMP_NUM_THREADS=6 because I never use the windows command line. Maybe someone else knows that…

Edit: keep in mind that using 4 instances of rawtherapee-cli in parallel will also need much more memory. Though, as your machine has 64 GB of RAM, it should work for your NEF files.

9

thanks for good tips! This will get me started.

1 Like
10

If you have WSL, cygwin, or similar, then GNU Parallel can help as well: GNU Parallel - GNU Project - Free Software Foundation

11

If anybody is interested this worked for me:
-install msys2 (easy)
-install parallel (pacman -S parallel)
then this line does the trick:

ls *.nef | xargs -iarg echo "/d/usr/gfx/RawTherapee/5.8/rawtherapee-cli.exe -Y -t -s -o out -c arg " | parallel -j14

my files are saved to directory out (-o out), but tou can of course adjust this to your needs. This gives my cpu more than it can chew and the whole processing is much faster than running from gui.

3 Likes