Better solution to "checking for updated sidecar files"?

I’m getting tired of waiting 20 minutes for Darktable to start, but last time I checked, if you open an image from a location which has an updated XMP file compared to what your machine’s Darktable thinks it should have, that gets erased.

Is there an option I’m missing? Surely this cannot be the most optimal solution.

(Note: I have my RAW + XMP on a network drive I access with two different machines, sometimes a laptop and sometimes a stationary PC. Since one is Linux and the other Windows, the paths are not the same so they currently cannot share a database.)

I assume that Windows also has something like symlink. That should allow you to synchronise the database. I use Syncthing for this. I only added one exception for darktablerc.

That means that any check at startup will have to compare at least all image folders (not necessarily all files) to detect updated XML files. As you use a network drive, that will always be slower than local drives… (I have about 30 000 images in my collection, and checking XMPs on startup takes less than a minute).

An alternative (to be implemented!) could be to check for a changed sidecar whenever it is needed. That would add extra processing time each time you open a file in the darkroom, with possibly a read and update of the thumbnail.

This seems a relatively common setup. I am wondering whether it wouldn’t be possible to add some command line flags (or config options) to specify prefixes or regular expressions to rewrite paths stored in the db on the fly. That would nake it possible to use the same db from different systems.

Semantically, it would be something like: if X/y does not exist, try Z/y instead.

For a time I had a similar setup and my temporary solution was a python script to make a copy of the db, change all the paths in the sql and swap the db file. It kind of worked, IIRC.

It does have some kind of symlink but it’s useless in practice.

There are a couple of possible solutions in development.

1 Like

Could you sync the images folder to both machines with syncthing etc.? That might speed up the scanning.

My “cold storage” of finished photos lives exclusively on the NAS and I sync a folder of unfinished photos to my PC to edit them with dt.

What’s the communication protocol to the remote storage? What’s the actual transfer speed?

I would prefer that, because even if the delay is 10 seconds when opening a photo in Darktable to check for a sidecar update, I could at least spend the remaining 19 minutes and 50 seconds editing that photo. :wink:

A samba share over Wifi 5ghz but with a Wifi 6 backhaul. However, there surely must be people with larger collections than mine facing the same issue who are not posting here.

The database contains the full path to the image, so the path will never be the same between Linux and Windows; syncing the db will not do any good.

I wonder if anyone has played with litestream. Of course it won’t work in this case, but it is replication for sqlite databases.

If you had two Unix systems and the nas, you could use the nas as the replication inbetween, since i assume its always on.

I can’t say how it works in Windows.

In Linux, the symlink is saved in DB on my systems, otherwise I wouldn’t be able to synchronise the database between my tablet and PC.

What’s the actual transfer speed?

I must admit, that’s not my only beef with “checking for updated sidecar files” method. Does it read-access tens of thousands of photos causing unnecessary wear on the hardware as well, every time I open Darktable, for example?

You started this thread complaining about speed and 20min. Now you are complaining about wear. I don’t think dt reads the images. I think it just scans the folder. If I recall correctly, samba over windows is very slow.