Asked and answered previously. I want to store my edits and also store metadata for each different edit of my image, and the semantic metadata might contain information related to those edits but I should be free to have two xmps related to the same source file with different metadata. For example, a description that says “Processed to B&W from color original in darktable”. Or on one I might crop out one feature and label it “apple” and on another edit, crop another feature and label it “banana”.
1 Like
I don’t understand why everyone in this thread thinks that the xmp sidecar works or should work cross application. I’ve never managed to use the xmp generated by one program in a different program, that would imply that everey program is the same. As far as I care, the sidecar could as well be in binary format. It’s not interoperable between apps and I’d never count on that. Any relevant metadata is found in the Exif data is it not?
Why would you want to publish your edit history?
Why does XMP standard impllements the word “photoshop” in the tag naming?
https://iptc.org/std/photometadata/specification/IPTC-PhotoMetadata#credit-line
It’s not about the edits. Xmp is not primarily a config file for image editors. That’s a peripheral use case that somehow several people here seem to think is the main one. I really don’t want all my edits in published files.
I’m having real trouble making people understand that. Why are people convinced it’s a format for editor configs and why would anyone want to standardise that when the tools aren’t the same? Sure you can but is it a great idea?
The main thing about xmp is sharing meta data and things such as IPTC. So that authorship, content, subject, etc can be used and meaningfully understood by various systems such as “the internet” but also various DAM and publishing tools. Bylines will be populated automatically etc.
1 Like
After reading this thread I am becoming more appreciative that RawTherapee saves my edits in a pp3 file and leaves my xmp file from Photo Mechanic alone.
Unfortunately Rawtherapee ignores the xmp files and fails to propagate the metadata to the output jpeg right? I’d like to think I’m missing a trick but I don’t think that’s the case.
Understand that what is standardized in the case of storing edit history and tool settings is the namespacing and some XML structure, not the tools themselves nor their settings.
Correct. I use RawTherapee solely to develop the raw file to a jpeg. I have RawTherapee set to pass the exif data through unchanged. I ignore the IPTC tab. I then use Photo Mechanic to embed the xmp data in the jpeg. This works well for me now that I have gotten the routine. Flickr picks up the xmp data from the jpeg largely without issue. I say largely because there is a minor incompatibility between Photo Mechanic and flickr with respect to geodata. I think it has to do with whether latitude / longitude are signed or always positive with E/W, N/S designations. It is easy to fix after uploading but a bit of a nuisance.
I am a fan of standards. In some ways this discussion reminds me of the fights that used to occur about html. But that war is in the past; no one argues against web standards any more. Even Microsoft threw in the towel.
What an interesting read this has been. Quite the rollercoaster here!
What I would like to add is the following. A file extension is basically metadata itself. You don’t always need it to identify the function of a file (take a look at magic numbers), and there are probably also cases where you don’t really care about the function of the file and can simply leave out an extension. But in many cases it helps greatly in identification and avoids the need for advanced heuristics to determine file type and, for example, figuring out how the OS should handle opening the file.
I believe that because of this, file extensions are pretty much ubiquitous in modern OSes as an integral part of the filename. It’s not without reason that we use terms like basename and extension, implying they are both part of a whole.
So, what happens when you want to describe metadata related to a uniquely defined file? Ideally, you embed the metadata in the file. That is perfectly portable. You could store the metadata in some sort of relational database, but that is much less portable, although many DAMs allow for this solution. If neither of these work you go for a sidecar file.
Then we get to the core of the matter. If you consider a file is uniquely defined by its basename and extension, then the only sensible way I see is to go for the basename.extension.metadataextension. The metadata extension uniquely identifies that file as a metadata-type-file pertaining to a particular, unique other file of a certain type.
If you do not consider a file to be uniquely defined by its basename, then of course, the sensible way is to go for basename.metadataextension. Personally, I really dislike this point of view… and the fact that this seems to be a standard concerns me.
Things get more complicated when you consider metadata related to a group of files, or metadata not for files but some abstract idea of content (e.g. JPEG,NEF,DNG,TIF that all contain the same image). I honestly believe that filesystems are not the correct solution to store metadata like that, because filesystems are not designed for one-to-many relations [citation needed]. That makes any filesystem based solution to this problem inherently flawed imo.
Just my two cents…
5 Likes
What are the flaws with this solution? It seems super simple to me. The only cost is the mental adjustment of having the basename describe the file content and the extension the format available. Both are actually the norm.
The idea that the metadata, in the form of exif and iptc should match one and only one file makes no sense in publishing or any other dam situation?
By adjusting to the very basic file naming scheme you can suddenly manage your files as their content not their files. A magic gain in abstraction and practicality. Whilst still avoiding databases and speciality tools. Again Geeqie does this completely naturally (grouping sidecar extensions setting). Tag the content not the 3 files. 50% less work with one quick stroke.
What are the potential issues?
So I think we’d all agree that basename.extension.metadataextension is technically a better solution but basename.metadataextension seems to be the “industry standard”.
Now, in my opinion, it is not important why basename.metadataextension was chosen in the past, it’s not that great but what’s done is done and we have to live with it.
Now what is important is what that industry standard brings to the table? Why would we choose to go with the a little bit less flexible implementation. What would we have to give up and what would we gain?
If the gains are greater than the losses then I think we should go for it, if not then we shouldn’t.
I’ve never thought or cared about XMP metadata, it’s just not that well communicated in the community as a very useful thing. I think it’s time I and everyone like me takes another look at that and try to see some potential there.
Now @damonlynch mentioned that by adhering to the industry standard it would gain us the interoperability with other software and with online services.
I won’t pretend to know how that works on Goggle Photos, Flickr or other places and software. But I will say that he brings up a very fair point! Because that’s a big enough reason for many users to choose to make a trade off and go with the “industry standard” way of doing things.
If all of them are using basename.metadataextension then we might be loosing more than we are getting by diverging from the “industry standard”.
I would also like to hear @anon41087856 and @wpferguson opinions on this. They are developing a WordPress plugin that would publish photos from daktable afaik. I’m guessing they’d want to take a look at and evaluate potentially utilizing the xmp files too! In that endeavor I’d probably want to go with the “industry standard” if I ever wanted it to go beyond Darktable. But that’s why I’m mentioning them here to voice their thoughts.
One day maybe we’d like to have a Nextcloud photo extension that also makes use of the XMP metadata. We’d like that to adhere to a wide standard too I guess.
Anyway, I’ve learned two things today. I should care about properly naming my files as it just makes sense, it is nice to have and it’s very easy thanks to RPD.
And I should care about the xmp metadata and should look into how I can use it to my advantage.
My own opinion is that Darktable should definitely support the “industry standard” because the potential pain points pale in comparison to the gains and some users might be willing to make that trade off. It’s not the best possible thing, the best would be for IPTC to change their standard but it is what it is and we have to get some and give some.
There might be an option in the settings to change from let’s say Darktable standard to IPTC standard when creating xmp files. That way everyone would be happy.
I’d like to extend this discussion a bit. I’d like to know in what way and for which purposes are people using the xmp metadata? How do online services make use of those exactly? I only ever thought about those files as just history, but I’d like to know more.
I’d also like to ask people what they think of this thing?
https://github.com/darktable-org/darktable/pull/2819
Obviously design is just horrible but personally I’d like to see a powerful metadata editor in it’s own dedicated view space in Darktable. But it must have a much better design and UI than it’s propose in this MR.
It would be a cool way to enrich our images and maybe start thinking about metadata too, not just the photo itself.
4 Likes
The case for technical superiority of filename.ext.xmp has so far been based on xmp as a config file for raw software. The advantage for tagging metadata in the form of exif, xmp and iptc has not been adressed? I have adressed why filname.xmp is advantageous but havent really seen counter arguments by anyone actually using the features. You can always freely add and remove tags from individual jpegs if you wish but then you are no longer in sync and loose some batchability. A fair choice.
Not only that: I DO have different metadata (including tags) for different exts of even the same file. It’s advantageous for me, it works for my usecase, it’s flexible… Why would I not use it?
yes - because “commercial” apps don’t understand filename.ext.xmp.
I haven’t received a reply on why can’t “commercial” apps support filename.ext.xmp? Why can’t they be more robust in terms of .ext support?
And I want to ask - It’s 2nd thread on same topic with same pushback from open source devs and same arguments. Has the topic creator or you created support questions/threads for commercial apps suggesting they could support filename.ext.xmp too? You know, just to be fair?
If I am not wrong, XMP spec is not only aimed at semantic metadata, replacing IPTC but also with edits metadata see https://wwwimages2.adobe.com/content/dam/acom/en/devnet/xmp/pdfs/XMP%20SDK%20Release%20cc-2014-12/XMPSpecificationPart2.pdf paragraph 3.3 and many other metadata.
So you should and must not reduce the role of XMP only to semantic metadata.
1 Like
I think everyone is aware of the spec allowing edits to be stored. However this is a separate issue and much less critical as edits can’t be fully compatible between software.
The choice of using xmp for edits is of much, much less concern for users and more an issue of one less sidecar in the folder and what format the developers want to write their configs in. It has very little impact on users. If config is stored in xmp it’s crucial to follow the spec though to avoid breaking the exif/iptc part.
Is it though? Not even Adobe, the creator of the XMP standard, holds to its own standard for handling all types of metadata. Take a look at what Premiere Pro produces after analyzing video files:
This is still metadata according to Adobe’s own definition: “Metadata is any data that helps describe the content or characteristics of a file.” Apparently the company who designed and pushed for adoption of a standard is okay to store this type of metadata differently? But some people think it is really bad that not every (FOSS) software follows the XMP standard? Aren’t those double standards?
This seems the core of the argument, really. There are multiple levels of metadata. Metadata per file, metadata per group of files, even metadata for multiple groups of files, and so on. These different levels of metadata seem hard to reconcile if you restrict yourself to a single XMP file.
Along with the different organizational levels also come different types of metadata. On a per file basis you may want to store things like creation date, but perhaps also some color label or ranking (for example, if you have multiple renditions of the same image that you label differently), and also things like editing history. Then on a group basis you may want to store things like shooting location, descriptors of the content (‘family pictures’ etc.), things that would result in redundant information if stored on a per file basis. And one level higher you may want to store information about author.
So, how does the XMP standard allow for this kind of control? And how does it rhyme with the idea of using the filename.xmp convention?
1 Like
Meanwhile the Adobe, etc., CEOs be tapping their cigars saying to each other:
What are we going to have to do to derail this whole
"Free"software thing.
3 Likes
@nosle
If I am not wrong, XMP spec is not only used for semantic metadata, replacing IPTC but also for" edit metadata" see https://wwwimages2.adobe.com/content/dam/acom/en/devnet/xmp/pdfs/XMP%20SDK%20Release%20cc-2014-12/XMPSpecificationPart2.pdf paragraph 3.3 and or whatever other metadata useful to manage the content ( type of coding, resolution, version, owner of rights…)
So you should not and must not reduce the role of XMP only to semantic metadata.
In my mind the master child relationship is important and inluences how metadata propagates. For a reasonable workflow you tag the master file, and any sooc children, with all the relevant metadata in one go. Creating the master .xmp file that applies to all files with identical basenames.
All files produced from the master will then have the base metadata included. Those children will naturally have some properties that diverge be it colour space, resolution, changed content due to crop file format specific metadata, notes that help identify it etc. Depending on the changes and workflow a rename (suffix perhaps to maintain relationships) can be made. You can also embed additional metadata in the child file as long as the format allows it. When outputting the file again overwriting it, after tweaks say, you have to redo that extra tagging but that’s inevitable. Allowing tagging only at export would be an interesting feature.
Of course you can maintain any personal metadata system on top of this if you wish but this can’t be expected to propagate or be understood buy the world. xmp and the standardised tagging is there to for that data you want to work externally.
That restriction is only for files with identical basenames in the same folder that haven’t been individually tagged. Not much of a restriction imho.
^ I’m not sure this is was an intentional retorical repost or a mistake as I answered it in the post you replied to.
As a user you can to whatever you like with your tagging and metadata. Override with fine grained per file embedded metadata, run parallell system, have a database, a text file, all metadata in file names etc. This is all possible whilst having well behaved standards compliant software than can interface with the rest of the world.
I’m super confused about this, a joke that this thread is to heated risking to undermine free software? A conspiratorial anti-capitalist worldview? @johnny-bit is expressing similar “rebellious” anti corporate stuff but it’s pretty hard to understand. Perhaps because I doubt anyone else on this forum has as “hard”, long term and “extremist” views on those issues as I do. There are only a few positions below and none to the left of me on the political compass.
I’m not talking about this guy:
Talking about this guy,
the capitalist management of corporations whose existence relies solely on intellectual property and logic.
You’re trying an anti-corporate BS as a counter argument against “the standard”?
I’m saying it clearly that filename.metaext is inferior and even @Thanatomanic showed that Adobe itself can use filename.ext.metaext. You say it’s “not much of a restriction”… Sure it’s not “that big of a deal”, but clearly allowing filename.ext.metaext is more flexible and robust for ALL standards that relate to file connections.
So, to circle back to anti-corporate BS: I have nothing against corporations (in this discussion), I have nothing against actual standard. I have a problem with interpretation that filename.metaext is “THE STANDARD” and therefore filename.ext.metaext is bad.
Simple: You guys are drumming for filename.metaext to open source devs, have you drummed for filename.ext.metaext to commercial stuff? I think more flexible and robust interpretation of filename is generally better than blind reliance on less flexible and less robust first implementation.
2 Likes