I was using Apple Aperture for more than a decade (Library 50k images / jpgs & raws) and didn’t care about a longterm strategy, I was young and just didn’t think about it. Now aperture is deprecated and I need to export my library. I want to move to a program that has no proprietary database so I don’t have to do this ever again. XMP seems like to be the thing to go for but it also seems to be less standardised than I first thought.
Darktable is fulfilling all my needs and would be awesome if I could use it. The only issue is, that the Aperture Exporter XMP files have the following schema: image.XMP (uppercase)
Darktable automatically creates an additional XMP file: image.JPG.xmp or image.ARW.xmp
Then I have 2 XMP files. Not good.
I could remove all by Aperture created XMPs and only use the ones created by darktable. Sounds good, but then I have the feeling that I am locked into a program again because on Mac common apps (Preview, Adobe Bridge) don’t read the file created by darktable and also a Metadata superpower tool like XNview can read it, but then creates a image.xmp (lowercase)
What is a good future proof approach working with XMP files?
TLDR if xmp compatibility is very important to you look elsewhere.
xmp should be considered stable and long term only for meta data such as tags, keywords and exif data. Edits stored in xmp will only be realistically valid for the software that you made the edits in. This may be obvious to you but a lot of people misunderstand the difference between xmp metadata and edits stored in the xmp file (simplified language)
Unfortunately darktable devs don’t care for following the file naming standard. There are long threads about it in this forum. So far barely any sympathy has been shown for meta data interoperability. For darktable the flexibility of storing edits trumps the metadata standards the industry use.
the .xmp files themselves. Those are text files, in a well-defined format (xml). So metadata can always be read and transformed to other formats, as long as those definitions are available (cf. xml namespaces) differences in extensions are mostly a nuisance (*).
the metadata concerning the processing. That is specific to darktable, and will be (mostly) unusable by any other program. Not only are those data stored in a binary format, the other program would have to know the exact algorithm used to reproduce the processing.
So things like keywords, captions, titles and other textual metadata will stay available, edits are as long term as with any other program storing the edits as a set of instructions. To really have long term persistance of your edits, you’ll have to export them to a well-defined image format.
(*: dt needs the double extension to allow sidecars for files with the same name, but different formats, as it never writes to an original file)
Put them in git or another revision control system. Check the modified files in after each editing session. Then you’ll have all your editing history at little to no cost.
Long term thinking should be: is the metadata stored in am open, structured, well documented format? This way you can transfer that data elsewhere when needed with some scripting. XMP answers “yes” to all those things.
If you quickly consider the above quotes you realize that it’s much more than a nuisance. A “simple” rename of all xmp files won’t do as you can have clashes. This is ignoring that renaming 100k files is not just a nuisance for normal users it’s a risky and for many impossible task.
Ask yourself this:
How are you going to add different licensing information if you can only have one sidecar per base filename? Or how are you going to handle different edits? Keep in mind that dt does not add sidecars to exports, only to imported files.
I never said renaming 100k files would be simple, or easy. It’s possible, so you can translate to a different requirement. Using a database or (god forbid) a binary format would be way worse. As for “risky” that’s one of the reasons we have backups.
Thank you for all the answers so far. I had noticed that there was a lot of discussion about in the forum, but wasn’t clear about the outcome. Image edits in XMP are not important to me, I’d rather prefer not to have them at all, because of the lack of interoperability.
I’ll need to think now If I either want
a system that works well know but need some export/modification of xmp or
a application with less comfort that can keep more strict xmp interoperability (only digikam if I did proper research)
It sounds like you need to define what you want in your tooling, what your idea workflow looks like, what interoperability you want; then the tools you need and how theyl need to work.
That’s why xmp files exist as a complement to a database. To safeguard interoperability and robustness. But that feature is broken when the file naming convention is broken.
Using digikam or similar software in a more traditional DAM role is certainly possible. I think darktable reads the metadata from files adhering to the standard. Perhaps it’s then an option to disable darktable xmp writing for a cleaner file system.
I think the way to think about xmp is like a cooking recipe, but not the final meal. If what you are after is a long term access of your images, then you need to export them instead of just keeping the raw + xmp. I keep raw + xmp + export of the finial edits.
I don’t want to duplicate the true facts that others have stated. There is likely no exact solution to your request. However, just some additions to think about:
xmp is no common standard in the sense that image operations are universal. Besides metadata such as tags, the sidecar xmp files (or pp3 or whatever) are saving all image manipulations. However, these are only meaningful for a single software. It does not matter if lightroom or darktable or any other software wrote that file, a reproduction of the image processing steps is not possible as the image operations themselves are proprietary.
With darktable, there are several benefits that may make the long-term situation better over other approaches:
Darktable reads lightroom xmp and applies some basic operations. This contradicts the rule above, but as it only includes very basic operations, it is only of little help. As you are not coming from lightroom, it won’t help anyway.
Any software can vanish at some point in time. However, due to its free/libre/open source nature, you have at least the chance to archive a copy of darktable, maybe even together with an OS image, to be able to run it in some future emulator even when x86 is something of the distant past. In particular, you are also independent of running license servers. If this is necessary for you, I can’t tell.
At many occasions, darktable devs have stated that image operations are only deprecated from UI but old edits will still open correctly down to version 0. As it is a complex software, there is no warranty that it technically always works out like this, but at least it is better than nothing, and so far I never had an issue, and I am running darktable for many years already.
I know others are of different opinion, but the way darktable deals with xmp file naming is IMO the most useful one. Disregarding original file extensions require an enormous effort to support files of the same name with different extension, and prevent a useful handling of sidecar files on the file system level (e.g. separating/moving single files + their xmp is not easily possible). This original idea of multi-file handling in xmp (by adobe) is insanely overcomplicated for no benefit but with lots of drawbacks.
The “library” database of darktable is only proprietary in the sense that most of its content is only useful in combination with darktable. Other than that, it’s a regular sqlite database which you can easily access with the sqlite command line tool or any sqlite gui.
Digikam uses the same sidecar naming convention as darktable, files are mutually read without problems. Digikam has an option to ensure “Sidecar filenames are compatible with commercial programs”, which is off by default.
Note that in the context of the question, disabling sidecar writing adds nothing to a longterm strategy of data protection.
See above @luap isn’t interested in storing edits in xmp. Sensible as they can’t really be compatible across software.
Yes, that’s the critical setting. Coupled with never doing any tagging in darktable and disabling xmp writing you should have a pretty good system I believe.
Thank you all for your answers so far. Stating again, that I don’t need image adjustments in XMP, crucially only tagging and rating: How would I in, lets say 10 years when Darktable is not working anymore, transform my XMPs into a more standardised format?
As I wrote in the beginning, key for me is to not have the same headache again in the future as I have now with moving out of a comfy library.
You can actually use them as a form of backup. What happens if some entries in darktable’s own database are corrupted, but you don’t notice that immediately, and keep backing up the corrupted DB? With edits present in the XMP, you can restore files on an individual basis. I never keep finished edits in the database, for example: once I’m done with a ‘film roll’, I remove (not delete) the images from darktable, knowing that I can always get everything back by reimporting them (which also loads all edits stored in the XMPs). Some go as far as completely disabling the database, and relying on the XMPs exclusively (re-importing the directory on which they work every time they open darktable).
XMP sidecars are standard tag-based XML format text files. Keywords and ratings are saved by darktable in standard tags inside the XMP (dc: subject, lr:hierarchicalsubject and xmp:rating if I remember correctly). You don’t have to do anything to make it “standard”, any sane program will read them.
This is false. The contents of the files may follow the standard but the file naming is crucial for it to work. darktable intentionally breaks the file naming standard. Only a couple of foss softwares will be able to read sidecar files and associate it with the correct file when the filename.ext.xmp filename is used.
To make it work you have to go through a filenaming and matching exercise that isn’t simple at all. If you have touched the files with a standards compliant software you have two xmp files determining which to use is non trivial. Breaking the file naming breaks xmp.
That has nothing to do with tagging and metadata. The use case xmp was designed for and @luap is asking about. This is the issue with darktable. The at best secondary feature of storing edits have broken the main feature of storing meta data. Dev’s are unable to understand the origins of xmp and the problem it was meant to solve.
As you keep referring to a naming “standard” for sidecar/xmp files, where is this standard published? Note that I expect a reference to the “standard” for naming sidecar files, the contents (data model and main namespaces) are described in an ISO standard.
That’s a rather bold statement to make, the more as I haven’t seen any alternate solutions to use sidecars where you can have multiple versions with the same base filename.
Keep in mind that the editing information is not the only thing that could change between versions, I can imagine situations where you want different licensing information (or different tags) for different edits/versions.