We don’t follow the standard because the standard is stupid. They glued on the fix to put several XMP blobs into one sidecar file, but that is so ugly to handle that i wouldn’t even consider discussing it. That part of the standard is basically just a leftover from the old DOS ages and short filenames. If Adobe likes to keep that, good for them. We don’t.
16 Likes
Thanks for bringing this up wrt RPD. I think @houz is right, it was really a bad design decision to go without the .ext part for the xmp. That’s also true from a user’s perspective. More than 1 xmp blob in 1 sidecar breaks basic user expectations about copying images around. If you really care about xmp with RPD, then you still have option 3: make it user configurable and take a reasonable choice as default. E.g., decide by OS which to use, or ask the user which software is used and fall back to the OS choice, or simply have .ext.xmp as default ;-).
BTW, what’s the exact functionality of RPD that handles sidecars? Writing them during import? Copying them during import if availabe from source? Especially in the latter case, to give the user the opportunity to convert from one to the other format, maybe even with smart handling of the multiple blob case, would be a great feature for RPD. Please don’t try to oversimplify the UI, photography, and especially digital photography has a technical side that simply requires basic knowledge from the user and user decisions.
Script? WDNNS script!
for f in *.CR2.xmp; do ln -s $f $(basename $f .CR2.xmp).xmp;done
1 Like
Yes, of course script, but I mean my filesystem will be littered with symlinks. I mean it already is since I use git-annex and that does symlinks too.
A rough back of the envelope calculation tells me that more than 99% of software by volume of photographic data throughput uses the Adobe led approach. It obviously works for them. AFAIK 100% of smartphone apps uses the Adobe approach with respect to those that actually create XMP files. You guys follow your own star, and are proud to do so, so it’s up to you to fix your user’s problems when it comes to interoperability with the 99%.
Rapid Photo Downloader’s entire reason for existence is to get data off the camera / memory card / external drive and onto the computer, with on-the-fly back ups and file renaming being optional. It’s not a general purpose renaming tool for folks who are changing their existing file naming scheme or moving their files about on their computer. ExifTool and exiv2 do a fine job of that. Moreover Rapid Photo Downloader doesn’t create XMP files. It’s not the Linux equivalent of Photo Mechanic.
1 Like
Ok, let me be a little more precise in saying why I don’t like the specs: it requires dedicated tools with insight into the XMP files for all the basic things like copying files around, backups, restoring from backup, sending edits to someone else and probably many more.
With a simple 1:1 mapping of media file to sidecar you can use the most stupid tools your operating system offers, be it cp, mv, rm or rsync for backups. All that breaks down horribly as soon as you put more than one media file’s XMP data into the same sidecar.
Since you need to put something into the individual XMP segments to identify what media file it belongs to (filename, checksum, …) you may never change that detail of your media file without editing the XMP, too, which again requires dedicated tools. So you can either not rename your image files, not change the content (for raw files that is ill advised anyway, but sidecars are used for other formats as well).
Regarding interoperability, I don’t see a problem with that. While XMP is a standard that can be understood by many programs hardly anything is interoperable in reality. Basic things like ratings are, tags are to some extent (there are application specific namespaces used already, for example by Adobe) and color labels are totally crazy. Once it comes to processing information there is basically no way to share the data between applications.
So what is left are some basic things that might be shared between applications, however as that is such a limited set almost no one does that in the free software world. And the few that actually do can use symlinks as described in other posts to get that working. And when trying to move the sidecars written by one application over to another without the intent of going back and forth (say, a camera writes a sidecar with ratings for every image it takes and I want to get that over into darktable, or I want to stop using darktable and move on to Lightroom) then a single rename step is enough.
TL;DR: I am all for following specs, even if it makes life a little harder, as interoperability is a good thing. But not for a price like this. Especially as basically no user missed that so far.
PS: I would be interested in seeing your calculations that said free graphics software had a market share of 1%. That sounds way too high. I wouldn’t even believe 1‰.
9 Likes
you can save one exec per file:
for f in *.CR2.xmp; do ln -s $f ${f%.CR2.xmp}.xmp ; done
For more see the refcard of the Advanced Bash-Scripting Guide
1 Like
my 2 cents on the subject … how about having an option to change from one pattern to the other on import?
I’ve been hoping the option to switch would show up on geeqie, as I use and prefer filename.ext.xmp. I’ve been using darktable’s features in the meantime.
So just to be clear, since TIFF, JPEG and PNG files are meant to have their XMP data incorporated in the file itself, what multiple files for one XMP sidecar are you talking about? Do you support video + XMP? Audio + XMP? If so, sure, you might need to support a situation where you have a video, audio and RAW file with the same basename. But otherwise I’m struggling to think of a situation where it even comes up in a practical sense.
Where I come from (NZ), 99% is often a term that signifies not a literal or exact number but figure of speech that means “almost everyone” or “almost everything”.
For one, darktable never ever changes source files. So embedding XMP data is not an option. There are basically two reasons for that: bugs might happen and destroy the user’s images, and quite a few people use automated backups of their image files (rsync, cloud based, …) which would trigger a slow transfer of the image every time something in the XMP changes.
Another reason to not embed XMP data into the source image is that the processing instructions are not part of the source but the to-be generated new image which does get the XMP embedded. Having the same instructions embedded in both files would result in double processing once exported images get opened in darktable. You also don’t want to give your processing info away when handing originals over to someone else.
Secondly, we support duplicates for images by having more than one sidecar file for a single image file. That isn’t possible with standard compliant naming either. Granted, that’s not directly related to the file extension being part of the name, but it’s still breaking compatibility.
Another thing we have is creating an HDR DNG from raw files. By default the base name of those is different from the source files, but that might change.
The longer I think of this the more problems I see with the “correct” behaviour.
May I ask why RPD cares about sidecar files? If it’s just in the business of copying images from camera/card to the computer, does it need to understand the concept of sidecar files? Are there cameras that create such XMP files themselves?
9 Likes
For two reasons:
- They are sometimes created by the “camera” (loosely defined). For example at least one Android application used by a Rapid Photo Downloader user creates them.
- The code to handle it was trivial, because I already had to handle the use case of audio files and THM files, which work identically.
For those who are wondering why I don’t just create the code to handle the darktable / Digicam way of handling XMP sidecars, the answer is pretty simple: it takes longer than you might guess to implement. I have a long TODO list of things of stuff to code and not enough time to implement them. Handling deviations from the spec takes time now and in the future, because I need to monitor changes to XMP handling that projects will implement in future. For instance Geeqie handles XMP differently again from darktable and Digicam. I don’t know what it’s approach will be a few years from now. A GUI is needed to expose these various options to the user, making the application more complex.
1 Like
@damonlynch I always appreciate your thoroughness and the fact that you share your thoughts in such great detail. Thank you!
1 Like
Thank you very much Mica!
1 Like
I’m with Damon on this one. As a Darktable user I’m alway annoyed that other apps can’t see my tags and Darktable can’t see tags I’ve added in other software. Unacceptable really when a standard is availabe. Darktable is great in many other ways though!
The idea that people use only one software to handle their raws is quite strange in a Linux context. For instance I might tag in one software develop most but not all in some other, fix specific problems in a third etc. For this to work the metadata such as tags needs to flow smoothly.
I’ve always thought that Darktable should use the xmp’s for the normal xmp metadata and a Darktable specific sidecar file for all the software specific stuff. A bit messy file structure but at least you can use the correct tool for each job without loosing metadata.
Hi Damon and community,
thank you very much for your work Damon.
I’m new to RPD. I just realized that my sidecar files from my SD card haven’t been copied to the file/folder structure. I’m using DT on Linux computers. so my sidecar files are named name.nef.xmp. I’ve read about the Adobe standard.
couldn’t there be an choosable option in the settings? I’d love RPD if it would be able to copy everything which is on the SD card.
thanks
Hi Martin,
My thoughts have not changed on the matter. Perhaps someone can volunteer to write a simple script to rename .ext.xmp sidecar files according to the expectations engendered by darktable / digiKam / Geeqie users? The script would be more general if it could also have the option to convert .ext.xmp files to conform to the xmp standard, so that users will not lose their metadata when they import their files into industry standard systems like other raw convertors, photographic management systems, etc.
I finally got hands on the ISO xmp standards and what a bummer, they do not discuss storage of the xmp data, they even explicitly exclude it from their scope. Adobes specs part 3 that talks about storage of xmp only has the one paragraph you quoted above, but does not define the situation of multiple files with the same basename at all. So I am still wondering: If one were to conform with Adobe’s specs, how would one correctly deal with metadata from multiple files? Did I miss that somewhere?
Files can embed the XMP metadata inside them, like is already done for IPTC metadata, or create a sidecar. That’s how every program outside of the darktable / digiKam / Geeqie triumvirate does it (although there are bound to be exceptions I don’t know about I suppose).
Maybe some other developers privately think the standard is “stupid” too. Who knows? But it doesn’t matter because the standard exists and is widely adhered too, just like the TIFF standard exists and is widely adhered too. Incidentally the TIFF standard itself comes from Aldus / Adobe, as does PDF.
1 Like
Just because you can pick the hammer and smash your fingers with it, does not mean it is the smart thing to do. (see: exiftool/digikam breaking raw files by doing exactly what you are suggesting to do, embedding sidecars.)
1 Like