Shortly I will buy a new computer to dual boot Linux and Windows. OS and software will be kept on separate partitions or drives, while image files will read and write to a shared drive. One thing I’d like to do is batch export possibly hundreds of images at a time from either RawTherapee, darktable or both.
I am wondering what hardware will help make this process as swift as possible?
Does anyone have knowledge/experience on SSD v HDD, Nvme/Pcie v Sata? Does it make a difference for either OS/Software drive, files shared drive, or both? I’m aware which performs better in general speed tests, but I’m interested in the practical difference when performing batch operations out of RT or DT.
Does amount or mhz of RAM make a difference?
Does CPU make a difference? (Currently looking at a 3rd gen Ryzen)
Does GPU make a difference? (Currently looking at GTX 1660)
Anything else worth considering?
Much appreciated.
GPU does not matter.
CPU does matter. The more cores, the better, as almost all tools in RT make use of all cores.
For modern CPUs it’s worth to make a native RT build. For my old AMD FX8350 this gives a speedup of ~10% compared to a generic x86/64 build. And this speedup is for free
RAM speed does matter if your CPU has a lot of cores and you run into memory bandwidth limit.
RAM size of course does matter. I would not go for less than 16 GB.
Regarding SSD vs HD, don’t even think about it. Any SSD will outperform any HD by far, and if your CPU is fast, you may want writting to disk as fast as possible.
Nvme/PCIe vs SATA: if your motherboard supports it, you will see differences of up to 6x the speed of SSD over SATA, but only if the PCIe lanes are at least 4x. You have to look specifically if the motherboard says something as 8x shared lane, which means that if you connect more devices into that lane, the bandwidth will be split among all devices. So take care of what you connect into a 4x shared lane or you will end up with an expensive Nvme drive performing like a SATA SSD.
Any significant difference between say 6 or 8 cores?
This (soon to be ex) mac user will need to learn how to native build. 10% speedup is significant.
Yes had 16GB pencilled in, with space for 32 if I wish to upgrade later.
Thanks for the tip. I need openCL for darktable, so will find a distro that is happy with Nvidia.
Is this specific to Rawtherapee? In other words, a 1hr batch becomes a 10 minute batch? That would certainly be worth it. And would I need it on both drives (OS/software & shared files) or just one?
For darktable, by far, the biggest impact will come from your GPU. A mid-range GPU like the 1660 will take a couple of seconds to export a typical image. Compare that to:
CPU: without a GPU, it would take tens of seconds. A fast CPU is still important because some of darktable’s modules are CPU-only, but it’s not the bottleneck until you have a very fast GPU
general RAM has probably little impact, but the memory in the GPU is very important. Choose at least 6GB to avoid tiling (when an image is processed in parts because it does not fit in there).
Sata/Nvme: think it this way: a “slow” Sata HDD still reads/writes data at around 100MB/s, it will read a typical image in half a second or less, even if not cached. Then darktable will take several seconds to process it. Of course, an SSD will make everything faster, and Nvme faster still, but it won’t be the bottleneck during export.
In my view, for darktable the order of money allocation should be: GPU > SSD = CPU > RAM.
The NVidia proprietary drivers usually work perfectly on Ubuntu and derivatives.
You may think about hard drives the other way around: it’s not that the process will be quicker with a fast HD, but instead that it will be slower with a slow HD.
You would only need a fast HD for your shared files, but certainly having a fast drive on your OS drive won’t hurt either.
A bit off topic (because it is not related to batch processing), but still:
When using RawTherapee on dual boot with Linux and Windows, you should have a look at this page http://rawpedia.rawtherapee.com/File_Paths#Cache
for Custom config and cache folders
A general one: I’m really happy how this one performs running darktable, GIMP and, as of late, RawTherapee. Often multiple versions at once.
A specific one: I started out with 16GB and thought, as mentioned in that topic, that that would be enough. Turns out that it isn’t, not really. I just upgraded to 32GB and that works much better.
Just so you know: I’m on Debian (10/Buster) and run Nvidia’s propriety drivers (450.66) on kernel 5.8.10. Without any problems whatsover. I do use some back-ported stuff though!
though i have code that will run ~20MP images through a typical darktable-like processing pipeline in under 20ms. on a system with an okay SSD (not up to today’s standards) for batch export clearly it becomes a bottleneck loading these files from disk, decoding in rawspeed, and uploading to GPU (which is why i interleave these things, the GPU can asynchronously upload and process while rawspeed can be a third parallel process). so thinking ahead a couple of years i think disk i/o may well become our bottleneck. i think it is very easily for thumbnail creation.
Ha! As I was writing that I had that idea in the back of my mind (“this is NOT TRUE for vkdt!”). I guess once your developments are ready for mass consumption it will be GPU = SSD >> CPU > RAM
Keep in mind that there are raw formats (Canon CR2 for example), for which just one or two cores can be used to decode them (no GPU). For this formats the frequency of the cpu matters as well, not only the number of CPU-cores and the GPU.
Thanks! So about 18% faster for those two modules.
Its a small sample size, but if we were to use the average of those two modules as representative of all modules, and we used 20 modules per image, then 8 cores would take ~26 seconds per image, and 6 cores ~32. If we had 1000 images, that is ~7.2 hours compared to ~8.9. So for individual images the difference between 6 and 8 cores is not that significant, but for batching large amounts it is.
Is the moral of the story to put the cache folder on the shared drive available to both OS? Or to keep it separate? Most of my RT work will be done on the Linux side, but it may be convenient to install it on both.
