Probably because I use lots of different programs to process raw files I definitely favor non-destructive editing. To me that means the raw files are NOT altered in any way. The mention of “Makernotes” does ring true to me. When I tried using Adobe DNG Converter to solve my raw format problem I found that the Makernotes completely disappeared from the resulting .dng format files. Likewise, it omitted the camera produced preview image in favor of one of its’ own making. Bottom line is that while .dng format might preserve the raw sensor data the file itself differs from the camera produced raw file in a lot more ways than just format.
This is an option that interests me. Might that be a reason for wanting to preserve the processing parameters in a sidecar file rather than a Duplicate?
Again, duplicates and sidecar files are not mutually exclusive, as your phrasing makes it seem. If you have XMP writing turned on in darktable, when you use the duplicates feature, a new sidecar file is created. The Duplicates feature let’s you select which edit you want to use by using buttons in the GUI.
1 Like
Just once again to clarify in DT the duplicate is just an extra xmp file with your additional edits…say two versions of your edit…one + filmic and one without as an example…Your option to make a duplicate creates a sidecar copy that is either an exact copy of your first edit or one that would be the standard created when you open and image (called original in the duplicate manager)…. I could be wrong but is this clear as to how it works… the duplicate option is not distinct from the use of sidecars??
I thought Darktable was writing a sidecar file that contained the processing parameters for the file being exported with the name of the exported file (i.e., the image file being created) suffixed with .xmp. However, that is NOT happening now.
In the settings there is a selection identified as “Storage>write sidecar file for each image” which has 3 options that are “on import, never, after edit”. At present it is set to “on import” which I think is the default that has not been changed. However, this sounds to me like it pertains to the file being edited (e.g. raw file) as opposed to the file being exported (e.g., image file created from raw file).
I can NOT find a setting for writing a sidecar file when an image file is created (i.e., exported). I think this should be a sidecar file with the same content as an up-to-date version of the one associated with the raw file being edited.
Could use some help finding the referenced setting?
That sidecar would be the sidecar that is saved along side your image…
On import creates a basic one on import
On edit the same …so this was added as users don’t like sidecars for files not yet edited…
None will use the database I presume I have never set it that way…
This is how the files are managed…
If you use duplicates and name them…
You can append something like _$(VERSION_NAME) from my example exporting the first one will create a file with _Edit1 appended and the second Edit2 appended.
The xmp files will have a version number in this case 0 and 1. So if you look in you image directory this is where you will find those two sidecars along with your raw image.
Maybe you have a look at the manual if a software doesn’t work as you ‚thought‘.
1 Like
Maybe you’re thinking of RawTherapee, I don’t think darktable has ever done this. Hence putting the processing instructions inside the metadata of the exported file.
Just curious …outside DT to what extent would processing data in an xmp be useful to other applications. I guess some of the metadata might be but the actual editing steps aren’t of any use except in DT are they??
Yes but those files are the ones I’d say are associated with the raw (?input) file rather than the exported image file.
It is probably fair to say that my thinking is a bit contaminated by my use of other programs. When an image file (e.g., tif format) is created from a raw file using Rawtherapee (RT) you would also get a sidecar file named “PXL_20220508_14425766.tif.out.pp3” as well as “PXL_20220508_14425766.dng.pp3”. Of course ART being a fork of RT also does the same thing but so does Canon DPP4 even though I have to request it whenever I do an export.
Note: RT also allows to save and reload as desired multiple sidecar files while editing the image. It looks like that is what Duplicates are about. Duplicates might be good, for all I know, but at present I’d say it is a more complicated way of doing the same thing.
What that allows, is to do more editing with RT on the raw file after having created the image file which do change the sidecar associated with the raw file BUT NOT the one associated with the image file. Therefore, I can come back later, maybe years, and resume editing the same raw (by loading that sidecar file) to reset parameters to the ones used to create the image file. If you really do it years later, as I have done, that is when it is a good idea to do it with the same version of RT used to create the subject image file. That is why running in portable fashion where I never loose a prior version is advantageous.
I think what I’m learning here is that what I just described is the reason for DT including the processing parameters in the metadata. While I’m wanting to avoid unnecessary changes to my workflow it is looking like this may be necessary should I want to use DT. I can see an advantage which is that the metadata is harder to separate from the image file than a sidecar file. However, metadata is another subject that has its’ share of complexities.
If what I’m saying is correct I’ll probably start by making my own copy of the sidecar file following a naming convention like RT in order to reserve the option of omitting the subject metadata from the image file (i.e., resulting work product).
Have a look at the discussion concerning Lightroom imports, different programs, same situation. Those threads explain why most editing steps can not be transplanted between programs (things like crop, and similar adjustments might translate, things like “filmic” or “tone equaliser” certainly won’t)
I would expect it is NOT useful at all which has something to do with why I dislike the idea storing that data in the image file as metadata. I do know from experience that metadata can have unintended consequences in other software. I think this is a case where the image files being produced are intended to be used by all manner of other software that we cannot possibly anticipate what might go wrong.
I have been spending a lot of time in the User Manual. At present, I’d say it is what I like most about Darktable (DT). However, as Mica has subsequently posted, it now looks like what I’m thinking did work is NOT the way DT actually works, which is also consistent with a very helpful reply from Todd. In that, NOT to be found in the manual.
With respect to dt’s processing metadata, if another application reads and and mangles something, that application is broken in respect to processing. XMP is the eXtensible Metadata Platform, and the extensible part is made by using a namespace in the XMP XML. F you look at darktable’s processing instructions, you see <darktable:exposure>1.234533</darktable:exposure> if another programs reads that an gets it wrong… Well 
all the code is public and commented in the source.
The manual explains very specific things in a very specific way. We omit a lot of conceptual things in the manual so we don’t have like 2,000 pages. The manual is already 300 pages long, which is a lot.
I never touched the settings, but I think I know how it works. So, take in mind that I might be wrong:
Darktable has a database, which contains the location of imported files, duplicates and all the metadata and edits. So, a few points:
- The original RAW file you imported should never be changed by Darktable. Just in case you were thinking that.
- Any edits you do, are always stored in the database.
Now, write sidecar file for each image (in Settings → Storage) on import does exactly that. Darktable will create (or update) .xmp files for each imported file, when you import them. So if you start editing afterwards (and modifying tags or other metadata) they will not end up in the xmp file (automatically).
The after edit function does more what you want I guess: It means that after every edit you do (or after you leave the Darkroom mode, not sure) the .xmp file is created / updated and will contain any metadata updates and any edits you made. This is handy if you keep the xmp files together with the raw files in your backup / storage, that way you keep the edits you made to the file always together with the file. But the edits are written as ‘a description of what Darktable has done’, and that means that (most) other software can’t do anything with those edits. Maybe simple stuff like cropping or orientation, but mostly only tags and simple metadata.
never is self explanatory I guess, Darktable will just never touch any xmp files unless you do it manually.
Now, reading of xmp files is done one startup (if you enable the option in Settings → Storage) and maybe when you open a RAW file for editing again (not sure about that one).
This all has nothing to do with the metadata that is written to a jpg file if you export a jpg file from Darktable. I believe that will always contain metadata, and so always contains the edits as metadata. So as long as another program hasn’t stripped the Darktable-metadata from the file, you can manually do load sidebar file... and then select a jpg that is written by Darktable.
In the Lighttable mode (photo picker mode as I call it), on the right side is a panel history stack, and it has two buttons at the bottom: load sidecar file... and write sidebar files.
The ‘load sidecar file’ allows you to pick a xmp or jpg file to read the metadata and edits from. Handy when you saved multiple copies (with different filenames) and want to revert to something. Or when you want to try out edits from this forum.
‘write sidebar files’ will forcibly write xmp files for all the current selected images. You can’t change the filename (as far as I know), this will always be <base filename>.xmp for any <base filename>.RAW file, in the same folder.
So, back to your original question.
I don’t know ‘how you cannot figure it out’. It should do this by default I guess. You double click an image from the Lighttable mode, you end up in Darkroom mode and you make edits. Those edits are then saved to the database. So the next time you double click the same image again, it should open up with the edits you did last time.
Now, if you want to throw away your edits and reset it to a certain .xmp (or .jpg) file:
- In the Lighttable mode, select the image you want.
- In the right sidebar, in the panel
history stack, you might want to click ‘discard history’ to throw away all your edits and reset the image back to defaults. - Now you can click
load sidecar file...and pick an xmp file or jpg file which contains Darktable edits. Now you can doubleclick the image again to open it in Darkroom mode and you’ll see the edits you just loaded.
(Instead of using ‘discard history’, there is also an ‘overwrite mode’ for loading a sidebar that does this in one go, but it’s set to ‘append mode’ by default. And discarding edits is always something I want to explain explicitly in case you do it by accident.)
If I compare to other rawconverters i know of:
- Lightroom / Adobe Camera Raw will not create .xmp files by default, but write them to their database. But a setting can be made to always write xmp files (so basically kinda like Darktable). I don’t know if there is a specific ‘load xmp file’ option though.
- DxO PhotoLab always writes in it’s own
.dopfile format. There is no database of edits, so if you throw away the file, you start over with the your edits. But they are always (over)written, so you can’t easily go back to a previous version of the edit, unless you made a copy of the.dopfile yourself. - ON1 PhotoRaw writes it’s own sidecar files with edits, but they are primarily stored in the database (so sidecar files are there as ‘a safety’ thing). I don’t even know how to reset the edits and load edits from a sidecar file.
- Capture One writes it’s own sidecar files, I don’t know what they contain. But the edits are stored for sure in a Session / Database. I don’t know if it’s possible to reset the edits and read them from a sidecar file.
- Rawtherapee works purely with it’s own sidecar files. Edits are written to them, and you can overwrite or merge edits with those from a .pp3 file. A pp3 file has the option to conain multiple set of edits (snapshots) though, so trying things out doesn’t has to mean you have to mess with multiple sidecar files for a single image.
So I’m a bit struck by this comment, because I think this is far from intuitive and common in other software. They all have their own way of doing things.
It might also be that you are just looking for a ‘snapshot’ feature. A method of having one RAW file, but having multiple versions of edits for it, so you can try around and compare edits. In Darktable this is called a ‘duplicate’.
In the Lightroom mode, it can be done from the right sidebar, the selected image[s] panel. There is a duplicate button there. I call it ‘a virtual copy’.
In the Darkroom mode I recently learned, there is a duplicate manager in the left sidebar. You’ll see previews of the other versions of the file, and you can click around to select them. I believe there is even a ‘mouse hover preview’ function by holding alt (and clicking? Not sure on this). And there is a duplicate button to create a duplicate as is (from your current version), and an original button to create a duplicate but start again from defaults.
This is a nuance I am not getting…not that I get everything…those xmp files are basically the script with the instructions of how to create that export so they are intimately related to it, DT just doesn’t offer to write a second copy with the exported file. If you apply them to another image it will blindly apply those steps which may or may not be the result you want on a new image. If you use shift ctr c and shift ctrl V you can do selective copy and pastes from the history stack on to a duplicate to make a new edit or starting point or on to another image in a series so this might be something you can also expt with.
There is a click and hold…so if you click on a duplicate and hold it will come up …when you let go the image you are currently editing comes back so you can do some on off comparisions among your duplicates with the currently displayed image
When I say associated with the raw (input) file I mean those are the ones automatically being updated by the software and synchronized with the database. If I want to freeze something that is certain NOT to be updated any further or can be used for restoration should the database go away (as happens when using the :memory option for --library) a different file is needed. That is the kind I’m talking about.
Okay well I guess DT takes the approach that they could get lost or need managed so just embed the data. I suspect something like a tiff would be your most common export to pass along so why not just make sure to use a copy of your exported file in the next step and then your export is intact, has the metadata locked in time and sync with the export and if the next program in your workflow mangles the DT data no big deal…I don’t think it would be that difficult to manage that way and if you use duplicates and selective copy and paste and name your duplicates you have a well organized set of exports from DT. If you really wanted this I am sure you could implement it in scripting or a batch file. Use the command line to do your exports and then also copy the xmp used for processing to the export directory…maybe even renaming it in a way it could not be modified like xmpx for exported and then you would have to physically rename it to use it again…this should protect it.