Dt 3.8.0 (windows): How exactly should database recovery be performed?

In the event that I need to start again (for example, after pathological disk failure) I think I am covered because I make session (or daily) backups of my images, their XMPs and darktable’s *.db and .db-snp files. But exactly how should I go about recovery? Is the following procedure effective?

  1. Having got to the stage of having usable hardware, and a re-installed/configured OS then download and install the release of darktable in use at the point of failure.

  2. (1st option) Do not, at this time, overwrite the just created *.db files with those from the backup.

  3. (Alternative 1) Outside of dt (which may or may not be running) copy the images and associated XMPS from backup and paste into the original file hierarchy under the $(PICTURES) root.

  4. Open dt (if not already open) and choose Import/Add to Library, judiciously importing images recursively down each branch of the image hierarchy, from the root, ensuing not to import too many images at once.

This immediately raises the question in my mind of what dt will do with the XMPS. My reading of the 3.8.0 manual says the default is that on import dt will create an XMP for each imported image. The manual does not state what will happen if an XMP already exists; it states:

“The “add to library” process simply adds details of those images to darktable’s library database (and creates an XMP sidecar file if applicable) allowing the images to be viewed and developed.”

Am I correct to assume that dt will give precedence to any existing dt-created XMP (and overwrite any that might have been produced by other software which used the same file naming construct as dt)? If this is not correct and dt overwrites the existing, ex-backup, XMP on input, how is the history to be added to the database?

Alternative 2 to step 3, and 4, above does nothing in step 3. In step 4 dt is opened and the Import/Copy & Import option is used, identifying the ‘places & folders’ in the backup media as per the manual. But this again raises the question about how existing XMPs are handled. The manual states:

“This option copies images from another location on your filesystem (including mounted storage devices) and then adds the copied images to the darktable library.”

If I instead I choose Option 2 to step 2 above, which is to overwrite the just-created *.db files with those from the backup (assuming that care has been taken to ensure the just installed dt and the backups are at the same version level), then do I need to worry about the XMP files at all? Or are they totally ignored in the ‘add to library’ or ‘copy and import’ options of Import?

With either import option, trying to reconstruct an image portfolio (with perhaps tens of thousands of images and hundreds to thousands of folders), is the ‘overwrite *.db files from backup’ the recommended way of recovery?

Is there any difference in approach between a Windows install and a Linux install

When importing images into darktable, it will honor existing XMP files and the edits (and metadata/tags) that are captured within them.

I have a desktop with Ubuntu & Windows dual boot as well as one laptop each with Windows and Ubuntu and often re-import “filmrolls“ (using add to library) that habe been edited on another machine and find all my edits.
Only if i were to create a duplicate, I would need to re-import the filmroll or the XMP on my main machine again (using add to library).

As a disaster-recovery or when moving my darktable installation to a new machine, I would go about this way:

  1. Install the desired version of darktable (needs to be the same version as used before or newer)
  2. Close darktable (!)
  3. Copy the image-tree back to the same directory name as you had it on the old machine
  4. Copy the databases back into the right place. Maybe other config files like the darktablerc
  5. Open darktable
  6. (Optional) Import your presets
    Since some recent version (3.6?) you van easily export all presets to a single directory with two clicks and re-import them just as easily.
    The presets should also be within one of the database files, but I like to export them every now and then to be able to import them on my secondary dt installations.

Having backups of the databases and darktablerc is nice to have, but as long as you have the raw files and XMPs as well as the expoted presets, you can skip steps 3&4 and simply re-import (using add to library) the image-tree and the presets.

//EDIT:

  • made it explicit that I mean add to library whenever I mention importing photos (image tree). I usually use recursive import for that.
  • also added that XMP files not only contain the edits but the metadata as well.

I think you need backups of the databases, to be able to restore all your meta data, for example the tags ?

/Per

Thanks. That’s the key principle that I was not clear about. The procedure you have described is as I would have attempted, given that ‘key principle’ - with one exception: your step 2, which I would have forgotten!

Good point. I would have forgotten this.

Tags and descriptions are stored in the sidecar files (if you set up dt to write those). Styles and presets are more delicate, as there’s no way to store those automatically outside the database on change.

1 Like

you can export these: presets from the preferences menu, styles or tag lists from their module ui.

I know, but afaik, that’s a one shot process, so whenever you change or add those, you have to manually export them again? In contrast, tags and descriptions are written automatically to the sidecar files, after you set that up once.

I just checked on the some images I imported yesterday. To my surprise there were no sidecar files.

I checked in the configuration settings, which had “write sidecar file for each image” = “on import”. That indicates to me, that tags have been stored in a database. Also it puzzles me, that the xmp files were not written.

I checked the setting: “look for updated xmp files on startup”, and restarted DT. A dialog came up with a ton of files with updated xmp files - I chose to use the “newest” between the xmp file and database version (I think)

Be careful not to lose data, if you choose not to backup the databases.

I cannot reproduce this; I also have write_sidecar_files=on import in my config, and sidecars were written when I imported images. Your second statement regarding “look for updated xmp files on startup” and the prompt displayed on restart suggest you may have been looking at the wrong directory: why would darktable prompt you when it finds no sidecars?

You are right

The imported files, including xmp’s were in ~/Darktable/20211228_, and not in ~/data/2021/2021-12-28- as I expected.

Did you use add to library… or copy & import…? I never use darktable to copy anything; I copy all images from the memory card to my file system using a file manager, then use add to library… (in-place).

This is what the copy & import… dialogue looks like:

However, as described in darktable 3.8 user manual - import, you can open the naming rules:

This is where you define the naming pattern and the destination directory. As you can see, base directory naming pattern (= base target directory for copy and import) defaults to $(PICTURES_FOLDER)/Darktable. You can override it case by case, or set the defaults in the preferences: darktable 3.8 user manual - import.

1 Like

Thanks a lot @kofa - I think I used the “copy & import”.
I am changing the base name pattern in the preferences.

Thanks for the useful followups.

Indeed I forgot last night to mention the tags and metadata in my post, I also forgot to make it clear that I’m almost exclusively using “add to library” for importing images into darktable.
I’ve edited my post above for future readers.

If you want the easiest and shortest restore/recovery, back up your darktable config folder. This means library.db, data.db, darktablerc ,etc etc.

When you get a new machine, install darktable, then move the config folder and images back into the same place they were before. Start darktable and you should be good to go.

OK; that’s all clear to me now. Thank you.