missing features: film roll and sidecars?

[CarVac]

Thanks.

First off I need to understand what exactly hierarchical tags are.

Are they actually a tree structure with parents/children, or is it just a flat series of tags separated by pipes?

[paperdigits (Mica)]

@CarVac

Are they actually a tree structure with parents/children, or is it just a flat series of tags separated by pipes?

Depends on what program you’re using. In darktable, its a list of tags separated by a pipe.

See for example:

[XMP:XMP-digiKam] Tags List                     : Species/Human##Species/Human/Position/Left##Site/Timbavati##Individual/NA##Species/Human/Number/1##Species/Human/Gender/Female
======== made_via_exiftool.JPG
[XMP:XMP-Panthera] Hierarchical Subject         : Site|Timbavati, Species|Human, Individual|NA, Species|Human|Gender|Female, Species|Human|Number|1, Species|Human|Position|Left
[XMP:XMP-Panthera] Subject                      : Timbavati, Human, NA, 1, Female, Left
[XMP:XMP-dc]    Subject                         : Timbavati##Human##NA##1##Female##Left
[XMP:XMP-lr]    Hierarchical Subject            : Site|Timbavati##Species|Human##Individual|NA##Species|Human|Gender|Female##Species|Human|Number|1##Species|Human|Position|Left

[CarVac]

And exiv2 can handle these myriad schemas in xmp files?

Or do I have to manually parse any of these many schemas?

[paperdigits (Mica)]

@CarVac

And exiv2 can handle these myriad schemas in xmp files?

It can read and write them.

[damonlynch]

@mlaverdiere

Of course, with FOSS, it’s up to the developers to decide on how they want to spend their time, what features they want to include in their app., etc., but IMHO, relying on a “homegrown” solution would not be a wise choice on the long run (and in the interest of users who usually don’t appreciate any app lock down when they want to use more than one tool to deal with their collection or when they want to move it in another app).

This is true even if one were to hypothetically never use another photographic program on their desktop computer. If you upload your tagged photos to Flickr, for example, Flickr automatically reads the XMP tags and incorporates them on the photo’s page, meaning they show up in keyword searches etc. That’s the power of an industry standard.

[CarVac]

That’s embedded XMP though, not sidecar XMP.

I can get behind that, though, as I tag my manual lens models on Flickr.

[damonlynch]

@CarVac

That’s embedded XMP though, not sidecar XMP.

If you’re going to go to all the work of supporting embedded XMP, then sidecar XMP is trivial.

Sidecars exist because writing metadata to reverse engineered files like RAW files is foolish and dangerous. JPEG is an ISO standard so no worries there — embed the XMP.

[paperdigits (Mica)]

@CarVac

That’s embedded XMP though, not sidecar XMP.

Sidecar XMP for RAW becomes embedded XMP in the exported JPEG

[grubernd]

Looking for sidecars was one of the first things I did when testing out your nice software.

Without them it is impossible for me to use it.
Not that is any of your concern, but let me give a little context.

My archive spans five major RAW converters: old ACR, Bibble, AfterShotPro, RawTherapee, darktable.
All of them with have sidecars.
All the metadata I entered in any of them among a quarter million images can be used in any of them.
Yes, sometimes I have to run some conversion scripts to inject keywords from one sidecar into another … but that can be automated.
That’s one side of the story.

The other:
My workflow involves three computers and two NAS drives.
Plus automated backup drives in case of a drive failure.
With sidecars I can just push one folder from one machine to another and know all my metadata will be there.
Edits can - and will - be revised, changed, etc.
But keywords, location and other things remain.
(darktable with their non-IPTC speciality metadata system is throwing some spanners into the works, but it is solvable).

So … reading existing FILENAME.RAW.XMP¹ files is a must in my book.
And while that xmp-lib is already there, well, use it to write your Filmulator section back to it, too.

¹) please ignore Adobe and their stupid FILENAME.XMP standard. It is backwards, not intuitive, produces only problems with JPG+RAW workflows.

[CarVac]

Good to know. I’ll keep this in mind.

[damonlynch]

@grubernd

(darktable with their non-IPTC speciality metadata system is throwing some spanners into the works, but it is solvable).

This sort of problem is why following industry standards is important.

@grubernd

So … reading existing FILENAME.RAW.XMP¹ files is a must in my book.
And while that xmp-lib is already there, well, use it to write your Filmulator section back to it, too.

¹) please ignore Adobe and their stupid FILENAME.XMP standard. It is backwards, not intuitive, produces only problems with JPG+RAW workflows.

Let me be blunt: you are asking an ISO standard to be overridden. You had better have a really good reason to do so. Chances are very strong the people responsible for the ISO standard know a lot more about metadata and program interoperability than you do. The people responsible are experts in their fields. Are you a recognized metadata expert who is invited to speak at industry presentations, for example? Do you think you know as much as or more than David Riecks, for instance?

If you can produce statements by some of the people responsible for the ISO standard that says “hey we made a mistake, it should be FILENAME.RAW.XMP and not FILENAME.XMP”, then that would be a perfectly good reason to adopt this practice.

Otherwise it’s best to go with what the ISO standard says. That’s what pretty much everyone else does. The only people who don’t are developers who think the XMP standard is “stupid” and assume they know better. In reality they don’t know better, they just think they do. All they end up doing is making life more difficult for everyone else. Files like TIFF and JPEG embed the XMP; files like RAW files use a sidecar.

P.S. it is not just software where deviating from standards happens and causes all kinds of problems. For example, Apple step outside the PTP standard for their iOS devices, which is fine for them and macOS, but extremely annoying otherwise. gPhoto2 doesn’t work properly with modern iOS devices. Windows PTP doesn’t either. What a pain in the rear end. Such arrogance on the part of Apple!

[grubernd]

Damon, with all due respect … that part of that standard is a result of someone hacking something together without thinking it through and then just selling it to a committee that did not bother either.

A long time ago, when I was in the alpha-testing team on BibbleLabs and they went from their homemade .bib sidecars to xmp I was one who initially fought them on the filename.raw.xmp idea. But that changed the moment I shot a job in RAW+JPG and was able to edit them side-by-side as I pleased while maintaining complete flexibility. I was happy that they ignored this idiotic piece of the standard. Today those 15 year old xmps open happily in darktable with all the metadata present. Interoperability with any Adobe tools has been a hassle, as always. So yes … the filename.raw decision by hack plus committee was a bad one.

Of course I do not know what the exact story behind filename.xmp is, but the result is clear: use Adobe or die. Other members of this forum have already written thorough explanations, but since your argument trails around “it’s a standard” I have no hope of convincing you otherwise … I’ll just leave this link here:

Linux applications and their non-standard XMP sidecar naming convention

Ok, let me be a little more precise in saying why I don’t like the specs: it requires dedicated tools with insight into the XMP files for all the basic things like copying files around, backups, restoring from backup, sending edits to someone else and probably many more. With a simple 1:1 mapping of media file to sidecar you can use the most stupid tools your operating system offers, be it cp, mv, rm or rsync for backups. All that breaks down horribly as soon as you put more than one media file…

[CarVac]

I’m for whatever’s logical, ISO standard or not.

[mlaverdiere]

@damonlynch

Sidecars exist because writing metadata to reverse engineered files like RAW files is foolish and dangerous. JPEG is an ISO standard so no worries there — embed the XMP.

Indeed, the common understanding is that metadata shouldn’t be written (embedded) directly within RAW files. But here’s an interesting analysis made by the ExifToolGUI creator saying that it might not be that risky to do it: Writting metadata: To raw or not to raw?

[CarVac]

Filmulator keeps track of files in its internal database by hash… if you edit the metadata and move a file and try to teach Filmulator the new location it will not recognize it as the same file.

Perhaps in hindsight I should have figured out a way to hash the image data specifically, but that’s set in stone for now.

[grubernd]

@mlaverdiere

it might not be that risky to do it

Given the maturity the metadata tools have these days, I agree with the general idea.

But.

From a maintenance point of view writing to RAW files is a nightmare. Or even original JPGs out of cam for that matter.

The data ratio between RAW vs XMP is approximately a factor of 100 (JPG) to 5000 (RAWs). So any change you make in your editor will result in any disk operation to take at least 100 times longer than they need to. And that is for measly JPGs.

With average RAW files these days hovering between 20 and 40 MB and XMPs seldom exceeding 20KB we are talking a factor of more than 1000. For every single operation.

Add a keyword to 100 files? Change the whitebalance on 50 files? Insert your contact information into 10000 files? Run backups? Maybe over the internet? Everything will take longer.

You might be able to do it, but it is not a really clever thing to do.

[anarcat]

@CarVac

Filmulator keeps track of files in its internal database by hash… if you edit the metadata and move a file and try to teach Filmulator the new location it will not recognize it as the same file.

Perhaps in hindsight I should have figured out a way to hash the image data specifically, but that’s set in stone for now.

FWIW, I think that’s fine.

@grubernd

From a maintenance point of view writing to RAW files is a nightmare. Or even original JPGs out of cam for that matter.

I would avoid writing anything to imported files. Those are precious and, in fact, in my workflow, are generally read-only (they are locked by git-annex). Writing to the sidecar is fine, of course, and so is writing to another JPG file, where embeddeding the XMP makes sense. But I would never write to the originals.

[jorismak]

I know them in the interface as multiple levels of parent / child tags. Just like a tree view.

Imaging adding a tag ‘animals’. Under that a child tag ‘pets’. Under that add child tags with the names of the pets.

Now during import / organize I can click on the name of a pet. And that enables to picture to be found under the name of the pet, but also under generic ‘pets’ and also under generic ‘animals’.

Now, how that tree view is stored in a xmp I have no clue. Xmp looks like xml to me, so the tree hierarchy could (should :)) parent and child tags in my mind…

… But I wouldn’t be surprises as that someone somewhere decided to make a standard about splitting the levels with a separator character :(.

[CarVac]

Is there a distinction between that and just having all the tags individually?

Is it a way of making sure the main tag always comes with the hierarchy above?

[paperdigits (Mica)]

From what I’ve been able to see of lightroom, it also uses the pipe dileaneated list for hierarchical tags. The left most tag is the top of hierarchy.

<rdf:li>landscape | Yosemite | Half Dome </rdf:li>