I just found out that RawTherapee does not read the exposure compensation I set in FastRawViewer. Basically, it does not read the XMP sidecar files at all. I could live with this if I had a decent oneway converter script, but I just found this (way too) thin one.
I could expand on that, but would like to know if there are better/existing options first. Does anyone know of any existing projects I could use or send pull requests to?
See there are others having thought about the same.
RawTherapee only reads a very limited amount of data from embedded XMP tags (only ratings, I believe). I’m pretty sure you will have to cook up your own script to convert processing parameters from other software to be read by RawTherapee. Sorry, XMP sidecar support is not coming anytime soon.
I started working a small C program to generate skeleton pp3 profiles from sidecar XMP.
I’m doing so for the exact same reason, FastRawViewer.
Just a simple one way only scrubbing the rating. I’ll get back to you when I make the program public on GitHub.
However I “abandoned” the project when I discovered a quite simple workaround:
Cull the native raw images with FastRawViewer, then run the folder through Adobe DNG converter. The sidecar XMPs will get picked up and embedded into the new DNG file.
DNG is great anyway all the FUD over it doesn’t make much sense. Better compression, checksums, and much more confidence that my files will be useable and accessible for years or decades to come.
This workflow works great for me, because I need to run my files through the converter anyways because Rawtherapee doesn’t yet support the Lossless Compression used in my ILCE-1 (a1) or in ILCE-7M4 (a7 IV)
Tangent:
The really frustrating bit that remains is that the devs at libraw/FastRawViewer refuse to embedded XMP data into DNG files, and insist on using sidecars for everything. This is a problem since it creates 2 copys of metadata, and is blatantly disregarding the Metadata Working Group specification on how to use XMP metadata with DNG. As much as I hate Adobe, they made the DNG standard, and have made the quite understandable decision to ditch sidecars, and its probably best to stick to their decision of always writing to the embedded XMP for consistency and compatibility with other software, and avoid the aforementioned issue of having 2 potentially conflicting records of metadata.
What FUD?
I think there is a general consensus not to write or rewrite metadata into raw files. Raw files are an artifact of your shooting session and should be treated as such.
I’ve heard a lot of people concerned that DNG might not be ideal for long term storage, and that they might not be supported in a decade, or two, or more, and compatibility with raw processors. I believe that this doesn’t make much sense. How could a proprietary and undocumented format be better supported than an open specification based on the TIFF standard. I mean, just look at the situation with CR3 files.
I agree, honestly this might make some of the FUD valid. Using sidecars for DNG would be ideal. I’m confident writing directly to them because I have a robust system of backups in place. That being said, I’m curious as to how often people are actually experience issues when writing metadata to embedded XMP.
I’m by no means claiming that the way I do things is the “Best/Correct” way of doing things. It’s a system that works well enough in my situation.
Some cameras produce DNGs, those are fine. The DNGs that are no ideal are the ones produced by Adobe DNG converter or other software like topaz. This is because those files no longer contain raw data. You’re basically working with a tiff at that point and you no longer have the camera artifact that you created. While DNG is open, Adobe isn’t exactly the best stewards of their open formats, for example PDF.
How so? My understanding is that the DNG file still contains all the data from the bayer sensor doesn’t it? But rather than being wrapped up in an undocumented and propriety file format, its basically a TIFF container, as you said.
I haven’t actually used any topaz software, however I was under the impression that topaz demosaics and generates linear DNG files. Obviously this can’t be compared to real bayer sensor data.
But does this really matter? It’s the same data, just represented differently.
My understanding is that it is demoasiced. So not raw. Adobe themselves also say that DNG converter strips out some metadata and adds others.
The answer is personal. To you, it seems it doesn’t. But to me it does; I want the files from my camera, not what Adobe thinks is correct in that particular moment.
Granted, there are other DNG converters like DNGlab, but I still would never throw away my raw files.
This might be the root of our misunderstanding. Adobe DNG converter does not demosaic your raw files. I also want all the data in my raw files, and the DNG converter doesn’t throw any of that away.
RawPedia, (or at least someone on RawPedia) seems to have a pretty good opinion of DNG, and makes what I think are some pretty good points.
http://rawpedia.rawtherapee.com/How_to_convert_raw_formats_to_DNG
That understanding is incorrect. Files converted using DNG converter are not demosaiced. (Maybe there’s an option to do so, but by default, no.)
That said, there have been enough cases where DNG Converter mangled/stripped metadata that later proved useful that using it is ill-advised. That’s the big issue with DNG Converter - frequent documented scenarios of corrupted metadata, or sometimes even corrupted output.
Converting XMP to PP3 is not going to happen - partly because, fundamentally, XMP is not that standardized. You can’t even use XMP sidecars from Adobe products in darktable (or vice versa), for example, because all of this software has very different image processing pipelines where there simply is no 1:1 mapping of the processing definition from one piece of software to another. For example, watch an Adobe product s**t the bed as it tries and fails to understand what “Filmic v6” in a darktable-sourced XMP sidecar is/does. At best it’ll just completely ignore the entry, and at that point, what’s the point of trying to use the XMP?
I believe that the original issue was to just grab the rating from the XMP and generate a PP3 with that rating, only because culling in RT is much slower than culling in FastRawViewer. A script that reads all ratings in .XMP files in a directory, and creates .PP3 file with only that rating seems possible, as a one-time, one-way conversion, with no intention to ever write back to the XMP sidecar, or to write to an existing .PP3, only generate a “blank” PP3 with the rating if one does not already exist.
RT already grabs the rating from embedded XMP data and seems to do that very thing. I’m sure I could make a very primitive script to read sidecars for ratings and generate .PP3 files for my own personal use.
Thank you for the correction + extra info.
I thought the advantage of XMP was that common parts are standardized, such as rating and label and other simple metadata, while it still provides eXtensibility for things that are generally not standardized, such as processing pipelines.
Exposure compensation, I am fairly certain, is not one of those standardized parts.