you can save one exec per file:
for f in *.CR2.xmp; do ln -s $f ${f%.CR2.xmp}.xmp ; done
For more see the refcard of the Advanced Bash-Scripting Guide
1 Like
my 2 cents on the subject … how about having an option to change from one pattern to the other on import?
I’ve been hoping the option to switch would show up on geeqie, as I use and prefer filename.ext.xmp. I’ve been using darktable’s features in the meantime.
So just to be clear, since TIFF, JPEG and PNG files are meant to have their XMP data incorporated in the file itself, what multiple files for one XMP sidecar are you talking about? Do you support video + XMP? Audio + XMP? If so, sure, you might need to support a situation where you have a video, audio and RAW file with the same basename. But otherwise I’m struggling to think of a situation where it even comes up in a practical sense.
Where I come from (NZ), 99% is often a term that signifies not a literal or exact number but figure of speech that means “almost everyone” or “almost everything”.
For one, darktable never ever changes source files. So embedding XMP data is not an option. There are basically two reasons for that: bugs might happen and destroy the user’s images, and quite a few people use automated backups of their image files (rsync, cloud based, …) which would trigger a slow transfer of the image every time something in the XMP changes.
Another reason to not embed XMP data into the source image is that the processing instructions are not part of the source but the to-be generated new image which does get the XMP embedded. Having the same instructions embedded in both files would result in double processing once exported images get opened in darktable. You also don’t want to give your processing info away when handing originals over to someone else.
Secondly, we support duplicates for images by having more than one sidecar file for a single image file. That isn’t possible with standard compliant naming either. Granted, that’s not directly related to the file extension being part of the name, but it’s still breaking compatibility.
Another thing we have is creating an HDR DNG from raw files. By default the base name of those is different from the source files, but that might change.
The longer I think of this the more problems I see with the “correct” behaviour.
May I ask why RPD cares about sidecar files? If it’s just in the business of copying images from camera/card to the computer, does it need to understand the concept of sidecar files? Are there cameras that create such XMP files themselves?
9 Likes
For two reasons:
- They are sometimes created by the “camera” (loosely defined). For example at least one Android application used by a Rapid Photo Downloader user creates them.
- The code to handle it was trivial, because I already had to handle the use case of audio files and THM files, which work identically.
For those who are wondering why I don’t just create the code to handle the darktable / Digicam way of handling XMP sidecars, the answer is pretty simple: it takes longer than you might guess to implement. I have a long TODO list of things of stuff to code and not enough time to implement them. Handling deviations from the spec takes time now and in the future, because I need to monitor changes to XMP handling that projects will implement in future. For instance Geeqie handles XMP differently again from darktable and Digicam. I don’t know what it’s approach will be a few years from now. A GUI is needed to expose these various options to the user, making the application more complex.
1 Like
@damonlynch I always appreciate your thoroughness and the fact that you share your thoughts in such great detail. Thank you!
1 Like
Thank you very much Mica!
1 Like
I’m with Damon on this one. As a Darktable user I’m alway annoyed that other apps can’t see my tags and Darktable can’t see tags I’ve added in other software. Unacceptable really when a standard is availabe. Darktable is great in many other ways though!
The idea that people use only one software to handle their raws is quite strange in a Linux context. For instance I might tag in one software develop most but not all in some other, fix specific problems in a third etc. For this to work the metadata such as tags needs to flow smoothly.
I’ve always thought that Darktable should use the xmp’s for the normal xmp metadata and a Darktable specific sidecar file for all the software specific stuff. A bit messy file structure but at least you can use the correct tool for each job without loosing metadata.
Hi Damon and community,
thank you very much for your work Damon.
I’m new to RPD. I just realized that my sidecar files from my SD card haven’t been copied to the file/folder structure. I’m using DT on Linux computers. so my sidecar files are named name.nef.xmp. I’ve read about the Adobe standard.
couldn’t there be an choosable option in the settings? I’d love RPD if it would be able to copy everything which is on the SD card.
thanks
Hi Martin,
My thoughts have not changed on the matter. Perhaps someone can volunteer to write a simple script to rename .ext.xmp sidecar files according to the expectations engendered by darktable / digiKam / Geeqie users? The script would be more general if it could also have the option to convert .ext.xmp files to conform to the xmp standard, so that users will not lose their metadata when they import their files into industry standard systems like other raw convertors, photographic management systems, etc.
I finally got hands on the ISO xmp standards and what a bummer, they do not discuss storage of the xmp data, they even explicitly exclude it from their scope. Adobes specs part 3 that talks about storage of xmp only has the one paragraph you quoted above, but does not define the situation of multiple files with the same basename at all. So I am still wondering: If one were to conform with Adobe’s specs, how would one correctly deal with metadata from multiple files? Did I miss that somewhere?
Files can embed the XMP metadata inside them, like is already done for IPTC metadata, or create a sidecar. That’s how every program outside of the darktable / digiKam / Geeqie triumvirate does it (although there are bound to be exceptions I don’t know about I suppose).
Maybe some other developers privately think the standard is “stupid” too. Who knows? But it doesn’t matter because the standard exists and is widely adhered too, just like the TIFF standard exists and is widely adhered too. Incidentally the TIFF standard itself comes from Aldus / Adobe, as does PDF.
1 Like
Just because you can pick the hammer and smash your fingers with it, does not mean it is the smart thing to do. (see: exiftool/digikam breaking raw files by doing exactly what you are suggesting to do, embedding sidecars.)
1 Like
I know you can and should embed xmp data in the file, but there are circumstances where this is not possible (even if just due to implementation) or not preferable, e.g. for ease of transfer/backup. So simply saying this scenario should not happen is not a solution. Adobe’s specs agree with that: “It is occasionally appropriate to store metadata separately from the file it describes;”. So there must be some way to handle the situation of more than one file with the same basename, which I still haven’t read how to do at all, let alone in a standardized way. I don’t want to argue about better/worse, I would just like to know how to do it without using the extension at all.
IIRC there is the possibility to add the metadata of several files into one xmp. But I agree to the darktable/dikikam/whatever folks that think this is a stupid solution. It makes moving files around a mess, e.g. if you have shot raw+jpeg, have xmps with metadata for both, moving just the jpegs means you have to copy the xmp files and you have doubled the metadata in either folder. Plus, it is something you would not do at the end of a work day, since it is too complex. Having one or more metadata sidecars for every single file makes sense since moving files around becomes very simple. Another example would be sharing metadata files, e.g. here on pixls.us. If you want to share only one of your edits of a raw file, you would first have to “export” to a new xmp that contains only this single edit, given that such an export exists.
Ouch, just started using digikam and was worried about what helpful things it might do in the background.
Can someone expand on the usefulness of having files with the same basename with differing metadata? I’m interested in case I’m missing a good way of organising files. Personally so far I’d much prefer to capture all files with one xmp. I like Geeqies grouping feature and wish all software could treat files with the same basename as the same content in different containers.
I guess I’m sort of url:ified with my paths/filenames. A path basename should point to content, files with the same basename should have the same semantic content just stored in different formats for different media etc.
As I’ve mentioned before I’d prefer (I understand this is unlikely to happen) the xmp to contain only capture metadata and human readable metadata such as tagging, titles, captions, ratings etc. It becomes conceptually and presumably programming wise complicated when the raw software history becomes mixed in with the content metadata. The software parameters could be saved in .dt sidecars.
I also like Geeqies option to embed or sidecar the metadata depending on filetype. I really don’t want to embed in raw files but really wan’t to embed(content metadata) in all other files as they should be portable.
I tag caption, retag, recaption refine my metadata and it’s a bit frustrating as is.
For me, I’d use a tag or note field to state the intent of the file, a jpeg for web will be sRGB, a tiff I print at home will be adobeRBG, while printing from a lab will be something else according to their spec. If all files share the same xmp sidecar, denoting the differences becomes more difficult.
Almost all RAW file formats are proprietary. I never suggested that XMP metadata be stored in RAW files. I don’t think anyone has during this conversation – where did you get that idea from? I wouldn’t do that myself, just like I wouldn’t embed IPTC metadata in the RAW. That’s just dumb. That’s why there is the option for XMP sidecar files. However, XMP metadata can be stored in jpeg, tiff, and other standard formats, precisely because they are standards. It’s pretty simple really.
Like I say, follow your own star if you want – ignore whatever standards you think are beneath you. Just make sure to do the responsible thing and make it easy for your users to use their files outside of your projects. Please don’t leave it up to others to do so.
1 Like
digiKam devs turn that exiv2’s feature off when compiling the official appimage. And yeah, no one should modify RAW files
1 Like