I need some advice on synchronzation with multiple devices.
I have a shared folder with images which is synchronized with two computers and a smartphone. New images are uploaded outside of darktable on the two computers and on the smartphone. The image folder is managed by darktable on the computers inkl. sub folder structure for film rolls.
Now I like to synchronize the darktable meta information. I learned that it is not feasible to synchronize the sqlite database directly, but rely on the information in the XMP files instead.
To further break down the problem I identified three use cases:
Images have been deleted on one side
XMP files have been changed
New images have been added
For case 1 I could not find any builtin database cleanup, but the dbmaint lua script could do the trick.
For case 2 I could enable the setting “storage - look for updated XMP files on startup”. Although there could be a performance impact when the database keeps growing.
But how could I catch case 3?
First I just re-import the same folder manually. The “add to library” dialog automatically selects the new images. Existing XMP files are not touched. However the flipside is that this is a manual task: I need to go through all filmrolls manually and select the correct folder on disk.
So I wrote a lua script which loops over all film rolls and does a dt.database.import for each film roll path. That basically worked.
But when I added an image which already has an XMP file (because darktable on computer A already imported it) then on computer B the field “darktable:import_timestamp” is updated darktable automatically detects the changed (XMP timestamp vs. database timestamp) an shows the dialog to update the database.
Is there a better workflow when working with multiple computers?
One option would be to skip using the database entirely by starting dt with database=:memory: in darktablerc or --library :memory: on the command line. Then you would be working entirely with the XMP files and not having to worry about database updates.
However wouldn’t that mean that I would have to re-import all film rolls on startup in order to work with the in-memory database? I have several hundreads of them and would guess it takes a significat amount of time. And when I import on one machine the timestamp of all XMP files are probably rewritten (haven’t checked that though).
Thanks for the pointer, this looks exactly what I am looking for!
Its interesting to see, that the author of the feature request had the initial idea to skip XMP files and only relying on the synchronized database instead.
Not all film rolls, but you would have to re-import the images you want to work on.
Also, you can’t do searches over the full image collection.
That said, if you mainly use one machine for work with the whole collection, you could have a database there, and only use the :memory: option on the other machines.
There is a way to synchronize darktable between two (or more) computers, it is described in German here: https://www.bilddateien.de/blog/2017-12-28-darktable-backup-der-configuration.html. It is an old description, but it is still the same today, I do it at least 5 times a year. In case you are not able to understand it I could try to describe it in English.
Thanks for the link (I am a German native speaker)
Indeed backups are essential! I do daily deduplicated encrypted multi-version backups of my images, including the XMP files - both remote and from time to time also physically offsite.
My files are synced with Nextcloud between computer A, computer B and the smartphone. I hesited to include the database in the sync, because 1) I have different base pathes on the computers and 2) the database is a big blob and Nextcloud isn’t good in syncing files large files where only small parts changed.
Apart from the computers where darktable is installed it happens regularily that images are just placed in the folder, e.g. when I upload them manually from the smartphone to the film roll folder. From a darktable point of view they just appear in the folder.
So my idea was to skip caring about database syncing between two darktable instances and solve the general case “update the database when images have been added or removed outside of darktable”.
There is one point which makes me wonder. The author of the article mentions that not all information in the database is also available in the XMP files - mentioning specifically groups:
Datenbank: Enthält zunächst die selben Informationen zu den einzelnen Bildern, wie sie auch in den .xmp-Dateien gespeichert sind, aber zusätzlich auch Informationen, die nicht in einem einzelnen Bild gespeichert sind, z. B. Gruppierungen von Fotos.
My understanding was that XMP files have the full information and the database is only a “fast cache” and can be fully re-created with the information found in the XMP files.
The database does a bit more than that. Besides the grouping information, the database is what allows efficient searches over your full image collection (doing that with only sidecar files would mean that you may have read every sidecar for a search, not funny with 10s of 1000s of images).
It’s not only about the presence of the information, but also about the organisation: getting references to all images in a group from the database can be done with two look-ups:
get the group ID of picked image from one table;
get all images with that group ID from another table.
That’s at worst 2 file operations.
Using sidecars for that would mean you read the sidecars for all images that can possibly be in the same group as the picked image… that’s “a few” more file operations…
That is assuming sufficient information is stored in the sidecars to retrieve groups that way. But there’s no image ID in the sidecar afaik. Which is not surprising, as there’s no good way to guarantee unique IDs through sidecars (unless you want to store a binary blob in there; sidecar files are plain text).
So any manual grouping will be lost. Note that grouping of derived files with the base file is still possible through sidecars (tag attribute “xmpMM:DerivedFrom”).
I do have a script that will do this and uses UUID values for the group identifier.
It works by catching group changes and recording the information in a tag which gets saved in the XMP.
If you lose your database, when you reimport your images with sidecars, the images are retagged with the tags from the sidecars and then the script rebuilds the groups from the tags with the correct image ids.
I just want to do some more testing with the script to ensure it’s as bullet proof as I can make it before releasing it.
Hallo Georg
Ich glaube, es ist besser, dir auf deutsch zu antworten. Also meine Art zu arbeiten ist die folgende:
Ich arbeite mit einem Desktop (iMac) und einem Laptop (MacBook Air). Auf beiden habe ich Linux Mint eingerichtet. Die Fotosammlung ist auf einer externen SSD, die ich von zeit zu Zeit auf einen andere SSD sichere. Der Desktop ist der Hauptrechner, der eigentlich immer auf dem neueseten Stand ist. In den Ferien oder bei Ortsabwesenheiten nehme ich den Laptop mit. Vorgängig update ich diesen, in dem ich die Ordner
/home/{benutzername}/.config/darktable und /home/{benutzername}/.cache/darktable des Desktop mit FreeFileSync kopiere und damit die gleichen Ordner des Laptop ĂĽberschreibe. Am Ende der Abwesenheit geht das gleiche in die umgekehrte Richtung.
Das geht jeweils etwa 5 Minuten. NatĂĽrlich ist es wichtig, auf beiden Rechnern die gleiche Version von darktable zu haben.
Nach meinem DafĂĽrhalten sollte es ja keine Rolle spielen mit einer Cloud oder einer externen SSD zu arbeiten.
Wie das ganze zusammen mit dem Smartphone geht, weiss ich natĂĽrlich nicht.
Hallo Peter. Danke fĂĽr die Darstellung Deines Workflows! Ich vermute mal, die Pfade zu den Bildern auf den beiden Rechner identisch?
Wenn Du bewusst den Datenbank-Sync per FreeFileSync anstößt, gibt es natürlich auch nicht das Problem mit den offenen Datenbank-Dateien oder dem ständigen Update/Upload der Datenbank im Hintergrund. Ich werde die Methode mal im Hinterkopf behalten.
Stimmt. Es macht für Darktable eigentlich keinen Unterschied, ob die Bilder per SSD oder Cloud eingebunden sind. Die liegen halt irgendwo im Dateisystem. Die Besonderheit des Smartphones ist, dass dort kein Darktable läuft und es auch keine Datenbank gibt, die man synchronsieren könnte. Für Darktable sieht es dann so aus, wie wenn in einem Verzeichnis / einer Filmroll neue Bilderdateien liegen. Ich muß die dann irgendwie Darktable möglichst automatisiert bekannt machen - durch Rescan des Ordners z.B.
An alternative approach is to use digiKam as your DAM and Darktable as your editor. digiKam can work across multiple computers, and has more advanced tagging and search options than Darktable.
This is what I’ve been doing for years. I even wrote a script that allows digiKam to display the thumbnail showing the Darktable edits for a more integrated experience.
