I’m getting tired of waiting 20 minutes for Darktable to start, but last time I checked, if you open an image from a location which has an updated XMP file compared to what your machine’s Darktable thinks it should have, that gets erased.
Is there an option I’m missing? Surely this cannot be the most optimal solution.
(Note: I have my RAW + XMP on a network drive I access with two different machines, sometimes a laptop and sometimes a stationary PC. Since one is Linux and the other Windows, the paths are not the same so they currently cannot share a database.)
I assume that Windows also has something like symlink. That should allow you to synchronise the database. I use Syncthing for this. I only added one exception for darktablerc.
That means that any check at startup will have to compare at least all image folders (not necessarily all files) to detect updated XML files. As you use a network drive, that will always be slower than local drives… (I have about 30 000 images in my collection, and checking XMPs on startup takes less than a minute).
An alternative (to be implemented!) could be to check for a changed sidecar whenever it is needed. That would add extra processing time each time you open a file in the darkroom, with possibly a read and update of the thumbnail.
This seems a relatively common setup. I am wondering whether it wouldn’t be possible to add some command line flags (or config options) to specify prefixes or regular expressions to rewrite paths stored in the db on the fly. That would nake it possible to use the same db from different systems.
Semantically, it would be something like: if X/y does not exist, try Z/y instead.
For a time I had a similar setup and my temporary solution was a python script to make a copy of the db, change all the paths in the sql and swap the db file. It kind of worked, IIRC.
I would prefer that, because even if the delay is 10 seconds when opening a photo in Darktable to check for a sidecar update, I could at least spend the remaining 19 minutes and 50 seconds editing that photo.
A samba share over Wifi 5ghz but with a Wifi 6 backhaul. However, there surely must be people with larger collections than mine facing the same issue who are not posting here.
I must admit, that’s not my only beef with “checking for updated sidecar files” method. Does it read-access tens of thousands of photos causing unnecessary wear on the hardware as well, every time I open Darktable, for example?
You started this thread complaining about speed and 20min. Now you are complaining about wear. I don’t think dt reads the images. I think it just scans the folder. If I recall correctly, samba over windows is very slow.
Checking for changes should only require reading the directories, applying the changes to the database requires reading the modified sidecar files. Reading the image files is never necessary when checking for changes (as dt never writes to the original images).
But with current darktable versions, there is no alternative to checking on startup. Not using sidecars means your edits will only be visible on the machine where you made them (with the possibility to get different edits on the two machines for the same file).
And checking each file on opening in the darkroom may actually cause more read operations, if the directory has to be reloaded: check on startup means you read each directory once and each sidecar at most once (if it’s changed). Check when opening in darkroom may require rereading the directory multiple times…
And there’s another issue if you don’t grab the changes on startup: any changes to ratings, tags, color labels, etc., won’t be available in the light table. That means searches will give incomplete results… Even worse when you do the tagging etc. on both machines: you can end up with a situation where you have three different versions floating around: each of the databases and the sidecar.
My thought is to do the check when the collection changes. One directory read, then compare the file timestamps to the image change timestamps. Apply the sidecar for each changed image that is discovered. The tags, color labels, etc should all update as the XMP files are applied. Keep track of each collection that has been checked, so that we only check once per darktable instance.
The only drawback to this is that you can create collections that span multiple film rolls. I could handle this, but it might be nasty.
(Just a nit: I suppose you mean “folders” and not “film rolls” ? )
You may also have issues when you use the search function. Startup would update the current collection, but if you then do a search, it would only look in the database for corresponding files (of course). So any images outside the collection where you started the search could be missed, if they had a changed sidecar…
Would work in shortening startup times, but may require a good think on how to break up the collection: searches over the whole collection become impossible.
I think that would be the main issue with any alternative solution: your database would not be up to date immediately after startup (and possibly never…), which makes searching unreliable (in that you may miss images or include extra images).
Nothing catastrophique, but may make it hard to recover that 4-star image you spend a few hours on last month on the other machine…
