Linux applications and their non-standard XMP sidecar naming convention

[damonlynch]

Several but not all Linux photographic applications write out XMP sidecar files in violation of the XMP sidecar naming specification promulgated by Adobe and used today in a large number of applications on multiple operating systems.

That is, instead of image1.CR2 and image1.XMP, several applications write out image1.CR2.XMP or some variation of that.

Naming the XMP file image1.CR2.XMP instead of image1.XMP represents a decisive break with both historical practice and also with how the great majority of other XMP aware applications operate.

Sidecar type files existed for quite some years before XMP appeared on the scene. To take two examples:

• THM files that contain a thumbnail of a video file, e.g. movie1.MPG and movie1.THM.
• WAV files that contain audio that accompanies photos, e.g. image1.CR2 and image1.WAV

From the spec:

“XMP is not directly embedded within MPEG-2 files, but is specified as a sidecar file. This is a separate file containing just the XMP packet, which is stored at the same location as the MPEG-2 file, and uses the same filename, with the file extension .xmp replacing the original file extension.”

and

“For applications that need to find external XMP files, look in the same directory for a file with the same name as the main document but with an .xmp extension. (This is called a sidecar XMP file.)”

Major photographic applications follow this naming convention, i.e. filename.xmp, not filename.ext.xmp. For example we read at Photo Metadata that “For file formats that have no internal support for XMP data, the data is stored in separate .xmp files with the same base file name. Many photo cataloging applications have support for this file format.”

Some discussion as to how some Linux applications came to default to filename.ext.xmp is available online, e.g. for Digikam. To me caulier.gilles’ justification for going with filename.ext.xmp is wholly unsatisfying. With respect to PNG and JPEG files, the XMP specification says to write the XMP metadata into the file itself, not as sidecar files. Moreover, unless I’m mistaken, Adobe developed the XMP spec itself to handle the use case of the same base file name with different extensions. See for instance the tag dc:format.

One might ask why does this all matter? Does it matter that some Linux applications have gone off in a different direction to the rest of the photographic software world?* Personally I think it does matter, because standards really do matter. Conflicting standards make life difficult for users and for applications developers. In my case I wrote Rapid Photo Downloader to follow the standard, but some Linux photography application users who use Rapid Photo Downloader as a file renaming tool are justifiably bewildered by the fact that Rapid Photo Downloader doesn’t detect their non-standard XMP files.

So as an application developer I face a dilemma:

• forget the XMP standard and make life easier for those exclusively in the Linux photographic community, or
• go with the standard and ignore non-standard XMP files, suggesting to users that they use ExifTool or Exiv2 to rename their existing collection of files.

I’m yet to decide what to do.

*Bibble Pro used to write out files using the filename.ext.xmp convention, but as I recall its successor Corel Aftershot offers the option to step back from that and follow industry standard practice.

[Ofnuts]

Linuxesque cop-out: use a soft or hard link from Foobar.CR2.xmp to Foobar.xmp or vice-versa.

[paperdigits (Mica)]

I had thought about doing this, bit that’s a lot of symlinks!

[houz]

We don’t follow the standard because the standard is stupid. They glued on the fix to put several XMP blobs into one sidecar file, but that is so ugly to handle that i wouldn’t even consider discussing it. That part of the standard is basically just a leftover from the old DOS ages and short filenames. If Adobe likes to keep that, good for them. We don’t.

[chris]

Thanks for bringing this up wrt RPD. I think @houz is right, it was really a bad design decision to go without the .ext part for the xmp. That’s also true from a user’s perspective. More than 1 xmp blob in 1 sidecar breaks basic user expectations about copying images around. If you really care about xmp with RPD, then you still have option 3: make it user configurable and take a reasonable choice as default. E.g., decide by OS which to use, or ask the user which software is used and fall back to the OS choice, or simply have .ext.xmp as default ;-).

BTW, what’s the exact functionality of RPD that handles sidecars? Writing them during import? Copying them during import if availabe from source? Especially in the latter case, to give the user the opportunity to convert from one to the other format, maybe even with smart handling of the multiple blob case, would be a great feature for RPD. Please don’t try to oversimplify the UI, photography, and especially digital photography has a technical side that simply requires basic knowledge from the user and user decisions.

[Ofnuts]

Script? WDNNS script!

for f in *.CR2.xmp; do ln -s $f $(basename $f .CR2.xmp).xmp;done

[paperdigits (Mica)]

Yes, of course script, but I mean my filesystem will be littered with symlinks. I mean it already is since I use git-annex and that does symlinks too.

[damonlynch]

@houz

We don’t follow the standard because the standard is stupid.

A rough back of the envelope calculation tells me that more than 99% of software by volume of photographic data throughput uses the Adobe led approach. It obviously works for them. AFAIK 100% of smartphone apps uses the Adobe approach with respect to those that actually create XMP files. You guys follow your own star, and are proud to do so, so it’s up to you to fix your user’s problems when it comes to interoperability with the 99%.

@chris

what’s the exact functionality of RPD that handles sidecars?

Rapid Photo Downloader’s entire reason for existence is to get data off the camera / memory card / external drive and onto the computer, with on-the-fly back ups and file renaming being optional. It’s not a general purpose renaming tool for folks who are changing their existing file naming scheme or moving their files about on their computer. ExifTool and exiv2 do a fine job of that. Moreover Rapid Photo Downloader doesn’t create XMP files. It’s not the Linux equivalent of Photo Mechanic.

[houz]

Ok, let me be a little more precise in saying why I don’t like the specs: it requires dedicated tools with insight into the XMP files for all the basic things like copying files around, backups, restoring from backup, sending edits to someone else and probably many more.
With a simple 1:1 mapping of media file to sidecar you can use the most stupid tools your operating system offers, be it cp, mv, rm or rsync for backups. All that breaks down horribly as soon as you put more than one media file’s XMP data into the same sidecar.
Since you need to put something into the individual XMP segments to identify what media file it belongs to (filename, checksum, …) you may never change that detail of your media file without editing the XMP, too, which again requires dedicated tools. So you can either not rename your image files, not change the content (for raw files that is ill advised anyway, but sidecars are used for other formats as well).
Regarding interoperability, I don’t see a problem with that. While XMP is a standard that can be understood by many programs hardly anything is interoperable in reality. Basic things like ratings are, tags are to some extent (there are application specific namespaces used already, for example by Adobe) and color labels are totally crazy. Once it comes to processing information there is basically no way to share the data between applications.
So what is left are some basic things that might be shared between applications, however as that is such a limited set almost no one does that in the free software world. And the few that actually do can use symlinks as described in other posts to get that working. And when trying to move the sidecars written by one application over to another without the intent of going back and forth (say, a camera writes a sidecar with ratings for every image it takes and I want to get that over into darktable, or I want to stop using darktable and move on to Lightroom) then a single rename step is enough.

TL;DR: I am all for following specs, even if it makes life a little harder, as interoperability is a good thing. But not for a price like this. Especially as basically no user missed that so far.

PS: I would be interested in seeing your calculations that said free graphics software had a market share of 1%. That sounds way too high. I wouldn’t even believe 1‰.

[darix]

you can save one exec per file:

for f in *.CR2.xmp; do ln -s $f ${f%.CR2.xmp}.xmp ; done

For more see the refcard of the Advanced Bash-Scripting Guide

[darix]

my 2 cents on the subject … how about having an option to change from one pattern to the other on import?

[paperdigits (Mica)]

I’ve been hoping the option to switch would show up on geeqie, as I use and prefer filename.ext.xmp. I’ve been using darktable’s features in the meantime.

[damonlynch]

@houz

TL;DR: I am all for following specs, even if it makes life a little harder, as interoperability is a good thing. But not for a price like this. Especially as basically no user missed that so far.

So just to be clear, since TIFF, JPEG and PNG files are meant to have their XMP data incorporated in the file itself, what multiple files for one XMP sidecar are you talking about? Do you support video + XMP? Audio + XMP? If so, sure, you might need to support a situation where you have a video, audio and RAW file with the same basename. But otherwise I’m struggling to think of a situation where it even comes up in a practical sense.

@houz

PS: I would be interested in seeing your calculations that said free graphics software had a market share of 1%. That sounds way too high. I wouldn’t even believe 1‰.

Where I come from (NZ), 99% is often a term that signifies not a literal or exact number but figure of speech that means “almost everyone” or “almost everything”.

[houz]

For one, darktable never ever changes source files. So embedding XMP data is not an option. There are basically two reasons for that: bugs might happen and destroy the user’s images, and quite a few people use automated backups of their image files (rsync, cloud based, …) which would trigger a slow transfer of the image every time something in the XMP changes.
Another reason to not embed XMP data into the source image is that the processing instructions are not part of the source but the to-be generated new image which does get the XMP embedded. Having the same instructions embedded in both files would result in double processing once exported images get opened in darktable. You also don’t want to give your processing info away when handing originals over to someone else.
Secondly, we support duplicates for images by having more than one sidecar file for a single image file. That isn’t possible with standard compliant naming either. Granted, that’s not directly related to the file extension being part of the name, but it’s still breaking compatibility.
Another thing we have is creating an HDR DNG from raw files. By default the base name of those is different from the source files, but that might change.

The longer I think of this the more problems I see with the “correct” behaviour.

May I ask why RPD cares about sidecar files? If it’s just in the business of copying images from camera/card to the computer, does it need to understand the concept of sidecar files? Are there cameras that create such XMP files themselves?

[damonlynch]

@houz

May I ask why RPD cares about sidecar files? If it’s just in the business of copying images from camera/card to the computer, does it need to understand the concept of sidecar files? Are there cameras that create such XMP files themselves?

For two reasons:

1. They are sometimes created by the “camera” (loosely defined). For example at least one Android application used by a Rapid Photo Downloader user creates them.
2. The code to handle it was trivial, because I already had to handle the use case of audio files and THM files, which work identically.

For those who are wondering why I don’t just create the code to handle the darktable / Digicam way of handling XMP sidecars, the answer is pretty simple: it takes longer than you might guess to implement. I have a long TODO list of things of stuff to code and not enough time to implement them. Handling deviations from the spec takes time now and in the future, because I need to monitor changes to XMP handling that projects will implement in future. For instance Geeqie handles XMP differently again from darktable and Digicam. I don’t know what it’s approach will be a few years from now. A GUI is needed to expose these various options to the user, making the application more complex.

[paperdigits (Mica)]

@damonlynch I always appreciate your thoroughness and the fact that you share your thoughts in such great detail. Thank you!

[damonlynch]

@paperdigits

@damonlynch I always appreciate your thoroughness and the fact that you share your thoughts in such great detail. Thank you!

Thank you very much Mica!

[nosle]

I’m with Damon on this one. As a Darktable user I’m alway annoyed that other apps can’t see my tags and Darktable can’t see tags I’ve added in other software. Unacceptable really when a standard is availabe. Darktable is great in many other ways though!

The idea that people use only one software to handle their raws is quite strange in a Linux context. For instance I might tag in one software develop most but not all in some other, fix specific problems in a third etc. For this to work the metadata such as tags needs to flow smoothly.

I’ve always thought that Darktable should use the xmp’s for the normal xmp metadata and a Darktable specific sidecar file for all the software specific stuff. A bit messy file structure but at least you can use the correct tool for each job without loosing metadata.

[capponz (Martin)]

Hi Damon and community,
thank you very much for your work Damon.
I’m new to RPD. I just realized that my sidecar files from my SD card haven’t been copied to the file/folder structure. I’m using DT on Linux computers. so my sidecar files are named name.nef.xmp. I’ve read about the Adobe standard.
couldn’t there be an choosable option in the settings? I’d love RPD if it would be able to copy everything which is on the SD card.
thanks

[damonlynch]

Hi Martin,

My thoughts have not changed on the matter. Perhaps someone can volunteer to write a simple script to rename .ext.xmp sidecar files according to the expectations engendered by darktable / digiKam / Geeqie users? The script would be more general if it could also have the option to convert .ext.xmp files to conform to the xmp standard, so that users will not lose their metadata when they import their files into industry standard systems like other raw convertors, photographic management systems, etc.