Slow export to JPG

I live on Ubuntu 18 using Darktable 3.8.1
Editing Sony alpha 7III raw files: adjust exposure, sharpening, lens correction, local contarst, shadows and highlights and changing the color tempriture. This leads to a one night long export process (500 images to make 10 second timlapse). I have an old HP laptop without any GPU.
The interesting part is the followings: start Virtualbox, Windows 8.1, Sony imaging software, what do the job around 2 hours.
Do you have any idea what should I do to get the same result on Ubuntu without Windows?
Thanks in advance

You are comparing the Sony software vs darktable?

1 Like

@tommy Nope: not enough information.

For starters: what settings on the Sony software (keep in mind that it knows the camera processing and can read all the makernote settings)?
Then: same export conditions on both, like file format, output size, drives? (exporting to local hard drive will be way faster than exporting to an external (USB2 or network) drive.

And finally: did dt really use sixteen hours for the export?

Sample raw file and sample sidecar file?

16 hours for 500 images means nearly 2 min per image. The export strongly depends on the modules used (diffuse and sharpen can for example be really time consuming). If you want a boost in speed use a graphics card for calculation (OpenCL). So maybe you could find another (desktop) computer to use.

I would translate that as ca. 8h?

That could be your problem, even though your information is quite vague. As others have pointed out, more precise information would be useful.

If you want some comparisionā€¦ I have: Ryzen 5 3600, 32 GB, AMD RX 6600. Without OpenCL it would take ca. 1h 40m to export 500 photos with this processing pipeline:

grafik

With OpenCL it takes 40 - 45m.

When I now translate ā€œoldā€ with something like ā€œ2012ā€ and suppose a midclass i5 mobile CPU or similar of that time, needing like 8h for 500 photos sounds reasonable to me.

About the comparison to the Sony software: Thatā€™s just how it is. DT isnā€™t the fastest.

Sorry about my long delay and thank you for your answers.
Yes, my laptop is really old, 10 yearsā€¦ishā€¦ HP6560b probook, i5 processor, 16GB RAM, 256GB SSD. But I like it. Iā€™m not here to complain. Just find it to be intresting that limited resources Virtualbox ā†’ Win8 ā†’ Sony software can do a faster job. And at the end a HD video produced.
Here is two XMP files. ā€œSimpleā€ means nothing touched, just import the raw files and hit the export button. 1,5 hour to export 500 images.
The second XMP contains some module usage, 4 hour export time.
If I make changes on colors (adjustments using moduls) that leads to a one night long export. (no XMP file, sorry)
Exposure, local contrast, shadows and hughlits DT_10054.ARW.xmp (6.9 KB)
simple DT_10054.ARW.xmp (5.7 KB)

Is it a CPU problem or an I/O problem? Are you using a SSD or an old HDD? I have found that VM sometimes work faster tha n native on HDDs because their filesystem isnā€™t spread over the whole HDD and this reduces seek times.

Using SSD and the VM can use only one core and 8GB RAM. DT can use both core and 16GB RAM.
Forgot to tell at the beginning, but Sony have to downsample all images on the fly to HD format because on Win 8 canā€™t produce 4K video.

So, much smaller images. In any case, it is near impossible to give a usable answer without knowing was is slowing down the process: CPU, GPU (or lack of), I/Oā€¦

I like how correct I was with my estimation :blush:

And that leads me to the point, that I personally think there is no particular problem, itā€™s just what you can get out of your machine with this software.

My laptop has a 2018 quad-core , but the ultra low voltage kind, and I even made sure there itā€™s limited at 15 watts because - as a working machine - i want it to be quieter , not faster.

But it also doesnā€™t have a GPU, And opencl is disabled .

If i take a 24mp image , with multiple instances of diffuse (sharpen no aa sharpen lens blur , local contrast fast ) , and lens corrections , and chromatic fixes , two instances of denoise (profiled) and the whole modern workflow module set , it starts crawling speedwise. But even then it takes like 30 to 40 seconds for an export .

For comparison , DxO export with their sharpening and their latest famous denoising takes 1m20 for a single image.

Now, your i5 model is older so a dual core , not a quad-core. And yes, intel did very little year on year to improve itself during those times , but still i think a laptop CPU from 2012 is quite a bit slower ā€˜core for coreā€™, and then you have two less cores. So yes , around 2 minutes per image sounds reasonable then. But that is eith the most expensive modules absolutely loaded (denoise and diffuse ). Without those modules , my export time drops down to under 5 seconds per image.

Now , for reference . My desktop machine with an - now also quite old :stuck_out_tongue: - amd Ryzen 2700 eight core machine with an (original price ) cheap gaming CPU (GTX 1060) can do the same loaded export in under 10 seconds .

The DxO export drops down from 1m20 to 12 seconds. Thats mostly due to the fact there is a GPU to work with.
It can make a massive difference if speed is important for you.

And I guess compared to the original Sony softwareā€¦ Do you know anyone actually using it ? I installed it once hoping to try out their updated sharpening algorithms on my older camera , but they donā€™t use the newer , better algorithms on files shot with a camera that came before that times ā€¦ So i removed it directly after it.

Speed and quality and usability differs greatly per raw processing software , makes comparing them kinda useless IMHO. Everyone is looking for something else.

On1 photo raw for example takes 30 seconds ro switch from image to image . Not export time, but ā€˜loadingā€™ time. It makes it completely unusable for me. I know lots of people are happy with it , but for me that is a deal breaker.

If Darktableā€™s export speed on your system is a deal breaker for you , thatā€™s perfectly fine and understandable. But donā€™t expect it to get any faster, only heavier and more gpu reliant Iā€™m guessing :).

1 Like

If you only need downsampled images, you can try disabling high quality resampling. Often, you wonā€™t notice the difference.

https://docs.darktable.org/usermanual/3.8/en/module-reference/utility-modules/shared/export/#global-options

1 Like