Darktable: image corruption on import of existing library


(Inchoate) #1

Problem
I’ve upgraded my system to the latest Kubuntu release (18.04). Since I keep all of my data (and photos) on a separate partition, I installed the latest darktable (2.4.4) and happily imported my photo library (I’m a long-term user darktable, so I have images collected with it going back several years).
After importing my photos, I started scanning my library and discovered several of my favorite images had been visibly altered. Alarmed, I started doing diff’s between the XMP files that had just been imported and my backups from my server.
What I’m seeing (so far; I haven’t dug too deeply yet) is that for some reason the mask data is getting messed up. In fact, it seems that each of the darktable:mask... variables has their contents duplicated multiple times!

Example

Backup Version

   <darktable:mask_id>
    <rdf:Seq>
     <rdf:li>1501099164</rdf:li>
     <rdf:li>1501099193</rdf:li>
     <rdf:li>1501099199</rdf:li>
     <rdf:li>1501099206</rdf:li>
     <rdf:li>1501099212</rdf:li>
     <rdf:li>1501099218</rdf:li>
     <rdf:li>1501099224</rdf:li>
     <rdf:li>1501099238</rdf:li>
     <rdf:li>1501099264</rdf:li>
     <rdf:li>1501099278</rdf:li>
     <rdf:li>1501099284</rdf:li>
     <rdf:li>1501099297</rdf:li>
     <rdf:li>1501099317</rdf:li>
     <rdf:li>1501099328</rdf:li>
     <rdf:li>1501099347</rdf:li>
     <rdf:li>1501099354</rdf:li>
     <rdf:li>1501099375</rdf:li>
     <rdf:li>1501099430</rdf:li>
     <rdf:li>1501099452</rdf:li>
     <rdf:li>1501099166</rdf:li>
     <rdf:li>1501099459</rdf:li>
    </rdf:Seq>
   </darktable:mask_id>

Imported Version

   <darktable:mask_id>
    <rdf:Seq>
     <rdf:li>1501099164</rdf:li>
     <rdf:li>1501099193</rdf:li>
     <rdf:li>1501099199</rdf:li>
     <rdf:li>1501099206</rdf:li>
     <rdf:li>1501099212</rdf:li>
     <rdf:li>1501099218</rdf:li>
     <rdf:li>1501099224</rdf:li>
     <rdf:li>1501099238</rdf:li>
     <rdf:li>1501099264</rdf:li>
     <rdf:li>1501099278</rdf:li>
     <rdf:li>1501099284</rdf:li>
     <rdf:li>1501099297</rdf:li>
     <rdf:li>1501099317</rdf:li>
     <rdf:li>1501099328</rdf:li>
     <rdf:li>1501099347</rdf:li>
     <rdf:li>1501099354</rdf:li>
     <rdf:li>1501099375</rdf:li>
     <rdf:li>1501099430</rdf:li>
     <rdf:li>1501099452</rdf:li>
     <rdf:li>1501099459</rdf:li>
     <rdf:li>1501099166</rdf:li>
     <rdf:li>1501099164</rdf:li>
     <rdf:li>1501099193</rdf:li>
     <rdf:li>1501099199</rdf:li>
     <rdf:li>1501099206</rdf:li>
     <rdf:li>1501099212</rdf:li>
     <rdf:li>1501099218</rdf:li>
     <rdf:li>1501099224</rdf:li>
     <rdf:li>1501099238</rdf:li>
     <rdf:li>1501099264</rdf:li>
     <rdf:li>1501099278</rdf:li>
     <rdf:li>1501099284</rdf:li>
     <rdf:li>1501099297</rdf:li>
     <rdf:li>1501099317</rdf:li>
     <rdf:li>1501099328</rdf:li>
     <rdf:li>1501099347</rdf:li>
     <rdf:li>1501099354</rdf:li>
     <rdf:li>1501099375</rdf:li>
     <rdf:li>1501099430</rdf:li>
     <rdf:li>1501099452</rdf:li>
     <rdf:li>1501099459</rdf:li>
     <rdf:li>1501099166</rdf:li>
     <rdf:li>1501099164</rdf:li> <<<-- duplicate!
     <rdf:li>1501099193</rdf:li>
     <rdf:li>1501099199</rdf:li>
     <rdf:li>1501099206</rdf:li>
     <rdf:li>1501099212</rdf:li>
     <rdf:li>1501099218</rdf:li>
     <rdf:li>1501099224</rdf:li>
     <rdf:li>1501099238</rdf:li>
     <rdf:li>1501099264</rdf:li>
     <rdf:li>1501099278</rdf:li>
     <rdf:li>1501099284</rdf:li>
     <rdf:li>1501099297</rdf:li>
     <rdf:li>1501099317</rdf:li>
     <rdf:li>1501099328</rdf:li>
     <rdf:li>1501099347</rdf:li>
     <rdf:li>1501099354</rdf:li>
     <rdf:li>1501099375</rdf:li>
     <rdf:li>1501099430</rdf:li>
     <rdf:li>1501099452</rdf:li>
     <rdf:li>1501099459</rdf:li>
     <rdf:li>1501099166</rdf:li>
    </rdf:Seq>
   </darktable:mask_id>

Note how the list of values has been duplicated. Each of the darktable:mask… variables seems to be similar with repeated data contents.

My System

Dell XPS 15 I7
16 G ram

I had just installed Kubuntu and have not yet installed the NVIDIA drivers, so no GPU was used durinig the import.

I have images taken from several cameras (Nikon 7000, 7200, Olympus Pen-F) and some images from each of them seems to be affected.

Has anyone experienced any problems importing an existing library into darktable before?
Any suggestions?

Here is the xmp file after import:
badimage.orf.xmp (23.3 KB)

This is the original:
OLYF4140.orf.xmp (12.9 KB)

This is the raw image file that goes along with thisOLYF4140.orf (17.2 MB)

[Edit: added image file]


(Mica) #2

I haven’t noticed any problem in importing images. I assume you have checked the options to check for updated xmp files on start up?

Also I wonder why you felt the need to reimport the library? You can either use the command line option to point to your custom library.db inside the .desktop file, or just symlink the location, ~/.config/darktable/library.db to your custom location.


(Inchoate) #3

Well, this was a full clean installation of Kubuntu, so I didn’t have an existing database to work with. I guess I didn’t have the foresight to put my library.db in my separate partition. I do have a partition where I keep my user files (and photos), but it is not my /home directory.
Looks like I need to put library.db and data.db in a safe place.

What should be darktable’s expected behavior when importing images that have sidecar files? (Given a new installation of darktable and no existing databases).

Thanks


(Mica) #4

I believe the expected behavior is to read up the sidecar file and update it. Different versions of the application have different xmp formats.


#5

To not break anything. I’ll have a look. There were some changes to remove unneeded masks, but I never saw duplicated masks.


#6

This should be fixed now. Thanks for reporting.


(darix) #7

JFYI: our 2.4.x snapshots for linux already have the fix.