Hello all, just getting started with darktable and trying to find a processing workflow that works well for me.
For some context, my current workflow I’m looking to improve: I have made the unfortunate decision to mainly shoot with a Sigma SD Quattro camera. The camera’s native raw format, x3f, is not supported by darktable or RawTherapee, and the DNG mode offered by the camera makes files 3x as large as the raws, and from what I understand offers lower quality output by some technical details, I think lowering the effective bit depth at some ISO levels or something to that effect. As such, I just shoot x3f and then back those up to the cloud as my primary archive. So far, when I’ve wanted to do any “serious” edits for a shot or shots, I’d export 16bit TIFFs in ProPhoto RGB space to preserve as much info as I can, write the file to a scratch folder and then edit from there in darktable. The TIFFs that come out of Sigma’s software are all uncompressed and so are 150MB each, so I don’t keep them around but so far I’ve been keeping the .xmp files. Ideally I’d have those written back to the directories where the original raws came from so they would be auto-synced to the cloud as well, but I don’t want those TIFFs syncing so those have to live somewhere else.
So my main question is: is there a better way of managing input files here? Ideally I’d use some kind of lossless compressed format as my intermediate, but I’m not familiar with an easy way to set up batch processing for something like this. Is there a way to get darktable to compress and overwrite the input images? Or a way to set up something to watch a folder and convert any TIFFs I export there from Sigma’s software? Or even just a suggested/easy way to make batch converting as painless as possible? Any suggestions as to file formats that might be best as an intermediate in the absence of raw support?
You can likely or maybe reduce the DNG…you might be set to embed the raw in the DNG which makes it really large…you could maybe do some more experimenting with DNG as well and that would I think be a better source file than a 16bit tiff… at least I think so…do you have a sample file that you could share…
I wonder if Adobe’s Free DNG convertor would convert your Sigma raw files to a DNG that could be used by DT and other programs. That is what I do with all unsupported cameras.
I can see the link @priort, and yes, the non-bayerness is both what I find interesting/attractive (as a nerd, if not for its actual abilities as a camera) as well as the cause of a lot of headaches when processing. Not only is it not bayer-pattern, but the red and green color channels are half the resolution of the blue channel. Super interesting stuff, but not at all well supported.
From the image you shared, the f/8 ISO 400/200 column demonstrates what I’m trying to avoid with avoiding the out of camera DNG. Does the source of that image mention what converter they used to do the conversion for the X3F → DNG samples? If those are compressed reasonably and can be done in batch that might be perfect.
As you mentioned, it’s the very design of the Quattro that makes this difficult, if not impossible/futile…
In any case, that special internal Quattro X3F layer structure needs to be first translated to full-res RGB, which increases the pixel count and file size to begin with. (This can only be avoided by software that natively supports Quattro X3Fs; apart from Sigma’s, perhaps LibRaw built w/ X3F support can do it, but again, not sure if its “demosaic” quality beats Sigma’s. It’s a shame Adobe can’t do it, a lossless DNG would’ve been a neat solution).
Onto compression then: any truly mathematically lossless scheme (even the modern JPEG XL one) provides around 2:1 ratio on average. Lossless TIFF files (horiz. predictor + deflate) are typically 50-70% of uncompressed size which is already decent, and it is a shame Sigma doesn’t output it directly.
Finally, in any further conversion (you can use ImageMagick/GraphicsMagick for batch conversion to JPEG XL or whatever), you also need to think about preserving metadata, as sometimes it doesn’t carry over well when changing formats (esp. proprietary MakerNotes that some tools might drop or mess up).
IMHO, the ideal solution would’ve been keeping any lossless compressed format straight out of Sigma SW (e.g. lossless TIFF/DNG). You might want to ask them to support it. Barring that, I’d forget about any further conversion and live w/ the large TIFF files (as mentioned, you’d be saving “only” 50% on average, but with a huge faff; you can alternatively use a compressed filesystem if you want to save some disk space).
P.S. You could try simply re-compressing Sigma’s TIFFs w/ LibTiff’s tiffcp, but again, not sure if the file structure and the metadata will remain intact, YMMV…
Can you share an image with appropriate licensing so some of us could test it? I am stunned that Adobe didn’t work because they tend to be on top of most cameras. Does Sigma provide software for handling the files?
Sigma does provide “SIGMA Photo Pro” for download for (rather limited) processing, and I believe they also have a plugin for lightroom and/or photoshop, but I don’t own either of those to be able to test that aspect.
On the other side of the coin, if I do accept that I have to live with proxy files, are there tricks to keeping the xmp files in sync with the original files? Specifically, could I have the xmp files refer to the original raws and store them beside the raws in my cloud sync folder, while then performing the actual operations based on the converted proxy files in a mirrored directory? Or, at the very least, is there a way to configure the xmp files to live in a directory other than where the images are?
While a good find, it doesn’t seem to like the SD Quattro files very much, at least as an out of the box experience when importing them into darktable, but perhaps I’m missing something?
Maybe a wb thing or maybe some other nuance…would wb scalars be 1 for all channels with this camera or some other non standard ratio to get it to display correctly…
