I am using darktable 3.6 on Windows 8.1 and want to make sure I understand the .xmp file.
When I import an image, I know it creates the xmp file. Three questions, 1) “as is”, is that file considered a back up file to the raw once its created during importing. 2) If I am editing an image, is that .xmp file the back up to what I have edited? I just want to be sure how that file works as I want to back up images that I have edited on my Passport. 3) Is the .xmp file what I see referred to as the “Side Car” file? If so can that be opened if for some reason my raw file cannot?
XMP files are the back up of the editing parameters, which are also stored in the library.db file. XMP files are not a backup of your raw file. To make your output, you need both the XMP and raw file.
yes, that xmp file contains a description of your edits. A sort of step by step data recording.
yes, that xmp file is what we call a sidecar file. No, that cannot be opened like a raw file.
Loading your raw file for editing will most likely also read in a xmp file in the same directory. Darktable has a special button to load a xmp file or write one (for example, if you want to try out an example file from the forum , you can download the raw and xmp file. Then after picking the picture in darktable ‘image browser mode’ click the button to load an xmp and pick the one you downloaded).
@aptillie , be aware that if you make any change to an image, then the XMP file is updated. So if you output a great image, e.g. a jpeg, last week, and now just try changing something, then unless you’ve backed it up, you will no longer have the XMP that made your great image. My way of ensuring I don’t get into this problem is this -
edit the image and output to e.g. jpeg
go to the folder where these files are, most recent at the top, and see you have for example
IMG1234-Nice-Cat.jpg
IMG1234.NEF.XMP
Copy the XMP to the same folder so you now have a third file, for example
IMG1234.NEF-Copy.XMP (or whatever your file manager app calls it)
Rename the copy to be the same as your jpeg. So you now have a pair,
IMG1234-Nice-Cat.jpg
IMG1234-Nice-Cat.XMP
plus your raw file IMG1234.NEF
Backup these three.
darktable will not touch your named XMP, it only maintains “raw file”.XMP
But at a later date, you can apply IMG1234-Nice-Cat.XMP to your raw file and hence re-create the edit. This is covered in the manual.
In most cases unless you have a massive edit you can also use an exported JPG as a side car file and load it in light able with a raw file to apply similar edits…either the source raw or even a different one to apply those edits…so it’s another option if you have an accident or misplaced somehow your xmp file
Thank you. So what I have been doing is correct then. After I edit an image even if I’m not done with it or have not exported it yet, I copy that .xmp file to my backup directory on my passport. Ive also been copying the original .nef file so I know what I’m working on.
I usually import all of my images after a shoot, to see what I have. Am I better off taking a copy of both the original imported .NEF file and the .XMP, of an image I am going to work on and placing them in my back up directory and just always work on that file so my originals that were imported stay intact?
If you are really happy with a certain edit, you might additionally want to backup the output file (e.g. 8 bit PNG). You never know if the software/pipeline breaks or changes in some way in the future.
However, darktable tries to keep backward compatible.
I’ll try that as well. So I will backup my NEF, XMP and a .PNG file. They will all go in one specific backup directory separate from where my original files are stored. Then if I want to continue to work on that file in that directory the .xmp will be updated as well.
Thank you
Sorry to state the obvious - just for the sake of completeness / safety:
darktable (and RawTherapee, ART, Filmulator, rawproc, …) don’t change your input files (whether they are raw files or JPGs, TIFFs etc.); you only need to back those up once, not after each editing step / session.
in darktable, your editing history is usually stored in the library database, as well as in .xmp sidecar files
it is possible to have several sidecar files (‘duplicates’) for an image; those can be created both on the lighttable as well as in the darkroom. Here I created several such duplicates and then deleted one of them (notice that _02 is missing): . You can use duplicates as ‘backups’: if you reached a point while working on the image, and want to make sure further modifications will not ruin your work even if you make mistakes and e.g. compress the history stack. Alternatively, they can represent completely independent development (e.g. different crop, black and white vs colour, etc.). This does not duplicate the source file, very little extra space will be used.
(this has been mentioned already) when exporting, darktable will, by default, save the editing history (basically, the same info as you have in the .xmp files) in the exported image’s metadata. Such images (e.g. JPG files) can then be loaded into darktable as sidecar files (as if they were .xmp files), even to other images. So, for example, you can take an old image, and apply the editing steps to an unrelated, new image, even one from a different camera.
the history (the library, the .xmp files, and metadata in exported images) does not contain the lookup tables you may apply using the 3D lut module (only the name of the LUT file will be stored). For maximal safety, you should also back up LUTs every now and then.
I’m going to review your reply when I have a few minutes to digest this, but looks very helpful. I’m sure I may have a few follow up questions for you after I’m done looking through this.
Thank you as always!
Still relatively new to using darktable, so your information has been very helpful. I have yet not done that much editing to images. Mostly cropping, vertical & horizontal adjustment, some exposure compensation, but I want to make sure that whatever I have done is not lost if for some reason I have a hard drive issue. I understand that darktable is not destructive so I can go back at any time to the original settings. I just want to be sure that I am going
to back up the correct file so that if there is an issue, I have it.
You mention: in darktable, your editing history is usually stored in the library database, as well as in .xmp sidecar files Can you tell me where that library data base is on the hard drive, and what file(s) I would backup along with the .xmp file?
Im not sure why I wouldn’t back up the .xmp file each time I am done editing if I plan on coming back to it later. Wouldn’t I back it up to keep the most current edits? If the file was corrupted for some reason while waiting to come back to it, the original .xmp would not contain any of the editing I had done to that image previously.
I need is some help and advice as to the best way to save, backup etc the files needed while I may be editing an image over a period of time.
The default location of that database file is $HOME/.config/darktable/library.db.
I wrote that you don’t have to back up your source image (raw or not) for each modification. You should back up your sidecars.
Personally, I use dirvish to back up to an external hard drive; it makes it easy to maintain date-based backups, without much storage overhead. I’m sure there are more modern solutions out there. For encrypted cloud backup, I use duplicati from the command-line, with BackBlaze as storage.
I figured the .xmp would have to get backed up each time I do some edits. I think at least I know now what to backup and where to store them.
Thank you
As you are on Windows you could make use of a little program I use…because I often update Dt and over the years a few times the new version has broken my edits or doesn’t work well with the old edits…I must say that was in the past…but still in addition to backing things up I use AutoVer. A free program that will watch a directory and keep versions of your files …so I have it watch my image folders for xmp files and it versions them according to the options…so I can go back to any version of an edit that I have ever done without too complicated a backup strategy…what I back up is the AutoVer folder as it has all versions of the xmp files I have ever created for an image…it also has been useful for a couple of situations where I was working on the same images from two PC and accidently updated the database from the cloud…long story but its a nice safety blanket … https://beanland.net.au/autover/
. You can use duplicates as ‘backups’: if you reached a point while working on the image, and want to make sure further modifications will not ruin your work even if you make mistakes and e.g. compress the history stack. Alternatively, they can represent completely independent development (e.g. different crop, black and white vs colour, etc.). This does not duplicate the source file, very little extra space will be used.