I did a fresh install of Ubuntu 17.10 and darktable 2.4.0 on one of my partitions and then merrily imported my photo library (stored on a different partition). The problem I am struggling with is that some of my images have lost all of their edit history.
When I look at the XMP file for one of these images, I can clearly see that there are no ‘darktable:history’ elements at all. If I start with a ‘good’ XMP file and then import the image in darktable, the XMP file gets overwritten with a nearly empty one. Fortunately, I have my library fully backed up elsewhere, and I can compare the XML file with its prior state. One curious observation is that during import, if I deselect ‘Apply metadata on Import’, the XMP file still contains meta-data.
The loss of history information isn’t consistent across all of my images, only some percentage of them. A manual scan of a dozen folders showed dozens of damaged images (but certainly not all images). I haven’t discerned a pattern yet. I’m hoping someone else can provide some insight
One example file I have been experimenting with is from a Nikon D7200. Other cameras may have issues too.
I had previously been running darktable 2.2.5 without issue.
I don’t yet know how to include files with this post, but I have an example NEF file and its associated xmp file if you want to see them. I also captured darktable debug (-d all) output, although it is gibberish to me.
I can copy a ‘good’ copy of an XMP file into my library (from my backup) and view its contents to verify that it is still intact. When I startup darktable 2.4.0 and import that image, darktable writes it back out with the contents overwritten.
What I found is that your XMP file is corrupt. It states that darktable:xmp_version is 1 while it is clearly a new one, so it should be 2. Are you sure that those files were written by darktable 2.2.5? Or darktable at all? And not by some other tool? What’s also odd is that darktable:history_end is 0. Did you maybe try to open that image in an even older version of darktable? That would explain the corruption.
That being said, it should be quite easy to fix those files. If you are able to write a small script then (after making a backup of the files!) look for all XMP files that have darktable:xmp_version="1" and use sed to replace that with darktable:xmp_version="2" and also set darktable:history_end to something ridiculously big, like 1000.
If you are not able to do that yourself I can surely come up with something.
My files were created using a variety of older darktable versions up through 2.2.5.
I just checked my backups and found a large number of them are ‘xmp_version=1’. This worries me a great deal. I hate to think that I’ve lost editing history. I’ll spend some time sampling some of them to see if they still have history attached. I’ll do that today.
I don’t have a problem scripting the XMP modifications, but I have some concerns:
What are the differences in xmp syntax (or rules) between xmp_version=1 and 2? If I simply change the version number, will my xmp files always be valid xmp_version=2 files? Are there other file contents that will need to be changed?
Will setting ‘darktable:history_end’ to a large number cause any other problems? Will darktable generate errors?
If you have XMP files around that were originally written by old darktable versions then having version 1 is fine. It’s just a problem when a version 2 file (as written by dt 2.2.0 and newer) was opened in older darktable versions.
A way to find those is looking for XMP files of version 1 that contain <darktable:history>. In theory they could however contain a 2nd history stack written by the old darktable.
Setting history_end to something big should be fine, darktable sanitizes that value when loading the XMP file.