Darktable: image corruption on import of existing library


(Inchoate) #1

Problem
I’ve upgraded my system to the latest Kubuntu release (18.04). Since I keep all of my data (and photos) on a separate partition, I installed the latest darktable (2.4.4) and happily imported my photo library (I’m a long-term user darktable, so I have images collected with it going back several years).
After importing my photos, I started scanning my library and discovered several of my favorite images had been visibly altered. Alarmed, I started doing diff’s between the XMP files that had just been imported and my backups from my server.
What I’m seeing (so far; I haven’t dug too deeply yet) is that for some reason the mask data is getting messed up. In fact, it seems that each of the darktable:mask... variables has their contents duplicated multiple times!

Example

Backup Version

   <darktable:mask_id>
    <rdf:Seq>
     <rdf:li>1501099164</rdf:li>
     <rdf:li>1501099193</rdf:li>
     <rdf:li>1501099199</rdf:li>
     <rdf:li>1501099206</rdf:li>
     <rdf:li>1501099212</rdf:li>
     <rdf:li>1501099218</rdf:li>
     <rdf:li>1501099224</rdf:li>
     <rdf:li>1501099238</rdf:li>
     <rdf:li>1501099264</rdf:li>
     <rdf:li>1501099278</rdf:li>
     <rdf:li>1501099284</rdf:li>
     <rdf:li>1501099297</rdf:li>
     <rdf:li>1501099317</rdf:li>
     <rdf:li>1501099328</rdf:li>
     <rdf:li>1501099347</rdf:li>
     <rdf:li>1501099354</rdf:li>
     <rdf:li>1501099375</rdf:li>
     <rdf:li>1501099430</rdf:li>
     <rdf:li>1501099452</rdf:li>
     <rdf:li>1501099166</rdf:li>
     <rdf:li>1501099459</rdf:li>
    </rdf:Seq>
   </darktable:mask_id>

Imported Version

   <darktable:mask_id>
    <rdf:Seq>
     <rdf:li>1501099164</rdf:li>
     <rdf:li>1501099193</rdf:li>
     <rdf:li>1501099199</rdf:li>
     <rdf:li>1501099206</rdf:li>
     <rdf:li>1501099212</rdf:li>
     <rdf:li>1501099218</rdf:li>
     <rdf:li>1501099224</rdf:li>
     <rdf:li>1501099238</rdf:li>
     <rdf:li>1501099264</rdf:li>
     <rdf:li>1501099278</rdf:li>
     <rdf:li>1501099284</rdf:li>
     <rdf:li>1501099297</rdf:li>
     <rdf:li>1501099317</rdf:li>
     <rdf:li>1501099328</rdf:li>
     <rdf:li>1501099347</rdf:li>
     <rdf:li>1501099354</rdf:li>
     <rdf:li>1501099375</rdf:li>
     <rdf:li>1501099430</rdf:li>
     <rdf:li>1501099452</rdf:li>
     <rdf:li>1501099459</rdf:li>
     <rdf:li>1501099166</rdf:li>
     <rdf:li>1501099164</rdf:li>
     <rdf:li>1501099193</rdf:li>
     <rdf:li>1501099199</rdf:li>
     <rdf:li>1501099206</rdf:li>
     <rdf:li>1501099212</rdf:li>
     <rdf:li>1501099218</rdf:li>
     <rdf:li>1501099224</rdf:li>
     <rdf:li>1501099238</rdf:li>
     <rdf:li>1501099264</rdf:li>
     <rdf:li>1501099278</rdf:li>
     <rdf:li>1501099284</rdf:li>
     <rdf:li>1501099297</rdf:li>
     <rdf:li>1501099317</rdf:li>
     <rdf:li>1501099328</rdf:li>
     <rdf:li>1501099347</rdf:li>
     <rdf:li>1501099354</rdf:li>
     <rdf:li>1501099375</rdf:li>
     <rdf:li>1501099430</rdf:li>
     <rdf:li>1501099452</rdf:li>
     <rdf:li>1501099459</rdf:li>
     <rdf:li>1501099166</rdf:li>
     <rdf:li>1501099164</rdf:li> <<<-- duplicate!
     <rdf:li>1501099193</rdf:li>
     <rdf:li>1501099199</rdf:li>
     <rdf:li>1501099206</rdf:li>
     <rdf:li>1501099212</rdf:li>
     <rdf:li>1501099218</rdf:li>
     <rdf:li>1501099224</rdf:li>
     <rdf:li>1501099238</rdf:li>
     <rdf:li>1501099264</rdf:li>
     <rdf:li>1501099278</rdf:li>
     <rdf:li>1501099284</rdf:li>
     <rdf:li>1501099297</rdf:li>
     <rdf:li>1501099317</rdf:li>
     <rdf:li>1501099328</rdf:li>
     <rdf:li>1501099347</rdf:li>
     <rdf:li>1501099354</rdf:li>
     <rdf:li>1501099375</rdf:li>
     <rdf:li>1501099430</rdf:li>
     <rdf:li>1501099452</rdf:li>
     <rdf:li>1501099459</rdf:li>
     <rdf:li>1501099166</rdf:li>
    </rdf:Seq>
   </darktable:mask_id>

Note how the list of values has been duplicated. Each of the darktable:mask… variables seems to be similar with repeated data contents.

My System

Dell XPS 15 I7
16 G ram

I had just installed Kubuntu and have not yet installed the NVIDIA drivers, so no GPU was used durinig the import.

I have images taken from several cameras (Nikon 7000, 7200, Olympus Pen-F) and some images from each of them seems to be affected.

Has anyone experienced any problems importing an existing library into darktable before?
Any suggestions?

Here is the xmp file after import:
badimage.orf.xmp (23.3 KB)

This is the original:
OLYF4140.orf.xmp (12.9 KB)

This is the raw image file that goes along with thisOLYF4140.orf (17.2 MB)

[Edit: added image file]


(Mica) #2

I haven’t noticed any problem in importing images. I assume you have checked the options to check for updated xmp files on start up?

Also I wonder why you felt the need to reimport the library? You can either use the command line option to point to your custom library.db inside the .desktop file, or just symlink the location, ~/.config/darktable/library.db to your custom location.


(Inchoate) #3

Well, this was a full clean installation of Kubuntu, so I didn’t have an existing database to work with. I guess I didn’t have the foresight to put my library.db in my separate partition. I do have a partition where I keep my user files (and photos), but it is not my /home directory.
Looks like I need to put library.db and data.db in a safe place.

What should be darktable’s expected behavior when importing images that have sidecar files? (Given a new installation of darktable and no existing databases).

Thanks


(Mica) #4

I believe the expected behavior is to read up the sidecar file and update it. Different versions of the application have different xmp formats.


#5

To not break anything. I’ll have a look. There were some changes to remove unneeded masks, but I never saw duplicated masks.


#6

This should be fixed now. Thanks for reporting.


(darix) #7

JFYI: our 2.4.x snapshots for linux already have the fix.


(Inchoate) #8

Thanks so much for finding and fixing this!

I really appreciate how much effort you put into this software.

Inchoate


(Inchoate) #9

Are these changes available in a PPA?
Or do I need to build darktable myself?

I’m kind of stuck without a fix. I get hundreds of corrupted images when I import my library.


(darix) #10

you probably want the graphics:darktable:stable package.


(Inchoate) #11

I just installed darktable from the graphics:darktable:stable as mentioned above, and I’m still seeing the same problem.
I have an image I had previously edited with 3 spot removal fields. When I import this image into darktable, the xmp file ends up with 3 copies of the darktable:mask_... variable data. For example, the darktable:mask_name list looks like this:

<darktable:mask_name>
    <rdf:Seq>
     <rdf:li>path #1</rdf:li>
     <rdf:li>path #3</rdf:li>
     <rdf:li>path #4</rdf:li>
     <rdf:li>grp spot removal</rdf:li>
     <rdf:li>path #1</rdf:li>
     <rdf:li>path #3</rdf:li>
     <rdf:li>path #4</rdf:li>
     <rdf:li>grp spot removal</rdf:li>
     <rdf:li>path #1</rdf:li>
     <rdf:li>path #3</rdf:li>
     <rdf:li>path #4</rdf:li>
     <rdf:li>grp spot removal</rdf:li>
    </rdf:Seq>
   </darktable:mask_name>

The other mask related tables show the same thing. Something is weird. Is my original xmp file corrupted in some way?


(Mica) #12

Can you try the snapshot release?


(darix) #13

that is the stable branch snapshot


(Inchoate) #14

Ok, I’ve installed darktable from the snapshot release: darktable 2.5.0~git389.2c4d40683.

I’m still seeing a number of seriously modified files. The worst are the ones where all of the darktable:history entries are deleted!

I have a moderate sized library of perhaps 50k images. I started with an empty darktable database (having removed all images) and then imported my files. I then ran diffs on the xmp files against my backup copies and examined the differences for a portion of them.

For the subset that I looked at, there were about 5000 total images with about 130 of them showing some differences. Out of those differences, about 25 were what I would consider to be ‘corrupted’, since they lost all of my image editng history.
Most of the other diffs seemed to be re-ordering of the <darktable:mask_… > lists (associated mostly with spot removal edits). I’m assuming that these changes are probably not significant, although I could be wrong.

I have been using darktable for many years now, having upgraded to new versions as they come along. It’s possible that my photo library has accumulated corruption from earlier darktable bugs.

I really appreciate your help! Any suggestions as to what to try?

Thanks


#15

Please share one XMP before and after darktable corrupted it.


(Inchoate) #16

Sure. Here is Before.ORF.xmp and After.ORF.xmp.

After.ORF.xmp (2.18 KB)

Before.ORF.xmp (8.03 KB)


(Inchoate) #17

Here is the raw Olympus file that goes along with the XMP’s I posted.

Can someone try importing the Before.ORF.xmp to see if they see a similar loss of edits?OLYF3482.ORF (18.7 MB)


(Eduardo Battaglia) #18

Lost all edits inside the XMP, not on the import. Before.ORF.xmp seems fine

Before:
Captura%20de%20pantalla%20de%202018-07-02%2018-46-49
After:
Captura%20de%20pantalla%20de%202018-07-02%2018-47-02


(darix) #19

are you using other software like digikam with those directories? I had digikam overwriting XMP files e.g.


#20

That xmp loads fine for me.

If you can reproduce the deletion of the history when importing an image with that sidecar, could you please do so while running darktable in a terminal so you can see error messages printed there?