For some reason darktable has started some odd behaviour. It keeps losing my images from 2023. It’s ok for a week or two, then I’ll open dT and click on an image and get the message “file unavailable”. When I check the folder tree, the folders have lines through them. I solve the problem by re-importing the whole year’s images but sure enough a while later, it happens again.
Any ideas?
Could you please give some details about your environment? Operating system, DT version, etc? This happened to me in the past on Ubuntu (Linux) and with my files mounted on removable media (external SSD).
Sure…
Ubuntu 22.04.3 LTS
64-bit
Gnome 42.9
Wayland
dT 4.6.0
Files store on motherboard mounted m.2 NVME SSD
Your user has read/write permission to the folder where the images are stored?
How did you install darktable? Snap, flatapk, OBS?
I’m not sure about permissions. I had a look at the folder permissions and just for the hell of it changed everything to: create and delete, for all users. I think it must have been ok before, though, or I wouldn’t able to add, edit and delete files, which I can. This losing the files issue is an intermittent problem, not a continuous problem. Mostly, it’s fine until suddenly it isn’t. No idea what triggers the problem. This morning I was editing files, then after lunch I came back to it and all my folders had lines through them. I re-imported nearly 2000 files (which took about 5 secs) and all was back to normal. The database doesn’t seem to have lost the edits, just the links to the files.
Bizarre.
How do I tell what kind of software it is? To update I just type:
sudo apt update
sudo apt upgrade darktable
Do you run any other software that sync’s or backs things up…just wondering if it is messing with your database?? Just trying to think outside of DT because that sound really strange…
Can you install sqlite and check the database?
No. Nothings syncs or backs up
How do I do that?
The two DB files, library.db and data.db are in ~/.config/darktable.
You can install sqlite with sudo apt install sqlite3 (I already have it, so it won’t get installed, but you’d see something similar):
kofa@eagle:~$ sudo apt install sqlite3
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
sqlite3 is already the newest version (3.42.0-1ubuntu0.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
kofa@eagle:~$
Then you can check the DBs (quit darktable before you do this):
kofa@eagle:~$ cd .config/darktable
kofa@eagle:~/.config/darktable$ sqlite3 data.db
SQLite version 3.42.0 2023-05-16 12:36:15
Enter ".help" for usage hints.
sqlite> pragma integrity_check;
ok
sqlite> .open library.db
sqlite> pragma integrity_check;
ok
sqlite> .quit
kofa@eagle:~/.config/darktable$
If the DB files are fine, start darktable from the command-line by typing darktable -d common.
Hopefully you’ll get some useful reports.
Are the images on a local disk (as opposed to network)? Do they always appear on the same path?
Ran the sqlite checks with same results as yours.
Command line yielded:
darktable 4.6.0
Copyright (C) 2012-2023 Johannes Hanika and other contributors.
Compile options:
Bit depth → 64 bit
Debug → DISABLED
SSE2 optimizations → ENABLED
OpenMP → ENABLED
OpenCL → ENABLED
Lua → ENABLED - API version 9.2.0
Colord → ENABLED
gPhoto2 → ENABLED
GMIC → ENABLED - Compressed LUTs are supported
GraphicsMagick → ENABLED
ImageMagick → DISABLED
libavif → ENABLED
libheif → ENABLED
libjxl → DISABLED
OpenJPEG → ENABLED
OpenEXR → ENABLED
WebP → ENABLED
See resources | darktable for detailed documentation.
See Sign in to GitHub · GitHub to report bugs.
0.0001 application_directory: /usr/bin
0.0001 darktable.datadir: /usr/share/darktable
0.0001 darktable.plugindir: /usr/lib/x86_64-linux-gnu/darktable
0.0001 darktable.localedir: /usr/share/locale
0.0001 darktable.configdir: /home/dave/.config/darktable
0.0001 darktable.cachedir: /home/dave/.cache/darktable
0.0001 darktable.sharedir: /usr/share
0.0001 darktable.tmpdir: /tmp
0.0001 new_xdg_data_dirs: /usr/share:/usr/share/ubuntu:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop
0.0298 output profile `APJ_OEMSCP900_MattPlus_EAM.icc' color space `RGB ' not supported for work or histogram profile
0.0298 output profile `APJ_OEMSCP900_PhotoArtSilk_WCRW.icc' color space `RGB ' not supported for work or histogram profile
0.1304 [dt_worker_threads] using 6 worker threads
0.1314 [dt_get_sysresource_level] switched to 3 as `unrestricted'
0.1314 total mem: 64062MB
0.1314 mipmap cache: 8007MB
0.1314 available mem: 1024999MB
0.1314 singlebuff: 64062MB
0.1337 [dt_dlopencl_init] could not find default opencl runtime library 'libOpenCL'
0.1337 [dt_dlopencl_init] could not find default opencl runtime library 'libOpenCL.so'
0.1338 [opencl_init] opencl library 'libOpenCL.so.1' found on your system and loaded, preference 'default path'
0.1433 [opencl_init] found 1 platform
[opencl_init] found 1 device
[dt_opencl_device_init]
DEVICE: 0: ‘Intel(R) UHD Graphics 770’
PLATFORM, VENDOR & ID: Intel(R) OpenCL Graphics, Intel(R) Corporation, ID=32902
CANONICAL NAME: intelropenclgraphicsintelruhdgraphics770
DRIVER VERSION: 23.22.26516.34
DEVICE VERSION: OpenCL 3.0 NEO
DEVICE_TYPE: GPU, unified mem
GLOBAL MEM SIZE: 51250 MB
MAX MEM ALLOC: 4096 MB
MAX IMAGE SIZE: 16384 x 16384
MAX WORK GROUP SIZE: 512
MAX WORK ITEM DIMENSIONS: 3
MAX WORK ITEM SIZES: [ 512 512 512 ]
ASYNC PIXELPIPE: NO
PINNED MEMORY TRANSFER: NO
AVOID ATOMICS: NO
MICRO NAP: 250
ROUNDUP WIDTH & HEIGHT 16x16
CHECK EVENT HANDLES: 128
TILING ADVANTAGE: 0.000
DEFAULT DEVICE: NO
KERNEL BUILD DIRECTORY: /usr/share/darktable/kernels
KERNEL DIRECTORY: /home/dave/.cache/darktable/cached_v3_kernels_for_IntelROpenCLGraphicsIntelRUHDGraphics770_23222651634
CL COMPILER OPTION:
CL COMPILER COMMAND: -w -DINTEL=1 -I"/usr/share/darktable/kernels"
KERNEL LOADING TIME: 0.0105 sec
[opencl_init] OpenCL successfully initialized. internal numbers and names of available devices:
[opencl_init] 0 ‘Intel(R) OpenCL Graphics Intel(R) UHD Graphics 770’
0.1540 [opencl_init] FINALLY: opencl is AVAILABLE and ENABLED.
[opencl_init] opencl_scheduling_profile: ‘very fast GPU’
[opencl_init] opencl_device_priority: ‘/!0,///!0,*’
[opencl_init] opencl_mandatory_timeout: 400
[dt_opencl_update_priorities] these are your device priorities:
[dt_opencl_update_priorities] image preview export thumbs preview2
[dt_opencl_update_priorities] 0 0 0 0 0
[dt_opencl_update_priorities] show if opencl use is mandatory for a given pixelpipe:
[dt_opencl_update_priorities] image preview export thumbs preview2
[dt_opencl_update_priorities] 1 1 1 1 1
[opencl_synchronization_timeout] synchronization timeout set to 0
UNIFIED MEM SIZE: 16016 MB reserved for ‘intelropenclgraphicsintelruhdgraphics770’
[dt_opencl_update_priorities] these are your device priorities:
[dt_opencl_update_priorities] image preview export thumbs preview2
[dt_opencl_update_priorities] 0 0 0 0 0
[dt_opencl_update_priorities] show if opencl use is mandatory for a given pixelpipe:
[dt_opencl_update_priorities] image preview export thumbs preview2
[dt_opencl_update_priorities] 1 1 1 1 1
[opencl_synchronization_timeout] synchronization timeout set to 0
0.2217 [dt_worker_threads] using 6 worker threads
sh: 1: git: not found
sh: 1: git: not found
3 Files are stored on a motherboard mounted m.2 NVME SSD drive. I’m so used to the DOS/Windows way of disk handling that I find it hard to get my head around how Linux does it. What is the Linux equivalent to C: ? Looking at the Disks tool it says the mount path for my darktable files is: /media/dave/datadisk Is that what you would expect?
I have a second SSD also on the motherboard which has a path of /boot/efi
That sounds on the face of it as if the OS is on one drive, with the data files on the other. I don’t remember doing it this way; in fact my recollection is that I forgot I had a second drive for a long time and only recently remembered it. The data drive is 60% full, while they other drive is 1% full. They are both 2TB SSDs.
However, these appeared to be greyed out by default and it normally looks like:
Should I switch off the “Default session” toggle and use the grey out options?
It’s good that your DB looks healthy.
For running with -d common, I meant that you use that all the time and see if there’s a message when the problem occurs.
The mounting options you show look weird to me. According to the File System Hierarchy Standard, /mnt is for temporary mounts. I think that is the root of your problem.
If the generated IDs are not constant (and, given that those are for temporary mounts, they probably change all the time), then it appears files will appear at different paths at different times.
My mount points are not /mnt/some-generated-id, but rather concrete paths (as reported by the mount command):
/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro) – the root of my file system (/), from the first partition of my SSD. System directories like /root, /etc, /bin, /sbin, /usr and /opt are stored on that partition./dev/sda2 on /home type ext4 (rw,relatime) – user directories, like /home/kofa, from the 2nd partition of my SSD/dev/sdb2 on /oldhome type ext4 (rw,relatime) – this is a large HDD where I mostly keep photos and videos/dev/sdc1 on /media/backup type ext4 (rw,relatime) – an external (USB-connected) drive with my backups.What does the mount command print for you?
I’m old-fashioned and don’t know the GUI tools. The mount configuration shown above is the result of manual configuration. Normally, drives are mounted based on /etc/fstab. It’s a text file that you can edit, but you need root (admin) rights.
My fstab looks like this, explicitly specifying which partition appears where in the file system:
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda1 during installation
UUID=b3646d97-a9f1-47e9-bb09-d6cfd8972ddd / ext4 errors=remount-ro 0 1
# /home was on /dev/sda5 during installation
UUID=44fbed5b-641b-43f1-852c-fb89f72247a4 /home ext4 defaults 0 2
# /oldhome was on /dev/sdc2 during installation
UUID=23acdf08-dad1-4666-9e0f-db88c2df0102 /oldhome ext4 defaults 0 2
# /dev/sda6 swap SSD
UUID=e235e39e-3465-45d0-b332-4dcd9f9a9b34 none swap pri=100 0 0
# swap was on /dev/sdc1 during installation
#UUID=b87bd478-ea55-4e3a-83aa-c4951832175b none swap sw 0 0
# /dev/sdg1
UUID=b36c1873-2b3a-4e3d-92f8-89f1f442b5e5 /media/backup ext4 defaults
You can check which cryptic ID (in the Identify As fields of your screenshot) maps to which disk and which partition with ls -l /dev/disk/by-uuid/:
root@eagle:~# ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 10 Jan 12 16:30 23acdf08-dad1-4666-9e0f-db88c2df0102 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Jan 12 16:30 44fbed5b-641b-43f1-852c-fb89f72247a4 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jan 12 16:30 b3646d97-a9f1-47e9-bb09-d6cfd8972ddd -> ../../sda1
lrwxrwxrwx 1 root root 10 Jan 12 16:30 b36c1873-2b3a-4e3d-92f8-89f1f442b5e5 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Jan 12 16:30 b87bd478-ea55-4e3a-83aa-c4951832175b -> ../../sdb1
lrwxrwxrwx 1 root root 10 Jan 12 16:30 e235e39e-3465-45d0-b332-4dcd9f9a9b34 -> ../../sda3
sda/b/c are the disks. sdx1/2/3 are partitions on disk x.
The advantage of the UUIDs is that if I move the physical disk to another connection (another cable), the UUID stays the same, even though sda may become sdc, if I understand correctly.
That listing does not show any of the weird/temporary /mnt/... paths you showed in your Mount Options screenshots. According to the list, your “data” disk is at /media/dave/datadisk, which is at least stable (even though media is intended to be used for external drives, pen drives and the like).
Do you import your photos from /media/dave/datadisk, or from some other, temporary location/path?
I import from SD card via USB and save to the datadisk.
Here’s the output from lsblk
I presume the 4 sd entries are for the sata ports I no longer use since I swapped to nvme ssds. The two entries nvme0n1p1 and nvmeon1p2 must represent my primary SSD which appears to be divided into 2 partitions. I don’t remember doing this, but it looks like there is a boot partition and a data partition. I assume nveme0n1p2 is the drive I call datadisk where my photos reside. nveme1n1 must be my second SSD which currently has almost no data on it.
None of the 3 partitions show any mount points except the supposed datadisk at /var/snap/firefox/common/host-hunspell … which seems a bizarre name!
Having played around with this a bit now, including rebooting, I’m beginning to think that is a reboot that kills the connection to the files and re-importing the files into dT just temporarily fixes the issue until the next reboot. Which points the finger at the mounting arrangements as you suggested.
The way I built this machine was to put the hardware together from parts, then import the files by copying them from my back up SSD. I wonder if the new machine never had its drives properly set up and I need to go through that mounting procedure you mentioned earlier. I’m reading about this as I type this.
I can confirm that rebooting is what kills the link to the files.
nvme0n1p1 is on /boot/efi (not accessible for file storage) - EFI system partition - Wikipedia
/var/snap is for software installed not from .deb packages, but the new snap format – which I hate, and got rid of, but Ubuntu is trying to push on us. I have no idea why your partition is mounted on such a path.
So your partition is mounted on an alternative path. Off now, back later.
blkid to read the IDs of your disks;fstab to make it similar to what I showed you above.My block devices (disks):
kofa@eagle:~$ sudo -i
[sudo] password for kofa:
root@eagle:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 476.9G 0 disk
├─sda1 8:1 0 64G 0 part /
├─sda2 8:2 0 396.9G 0 part /home
└─sda3 8:3 0 16G 0 part [SWAP]
sdb 8:16 0 2.7T 0 disk
├─sdb1 8:17 0 64G 0 part
└─sdb2 8:18 0 2.7T 0 part /oldhome
sdc 8:32 0 2.7T 0 disk
└─sdc1 8:33 0 2.7T 0 part /media/backup
sdd 8:48 1 0B 0 disk
sde 8:64 1 0B 0 disk
sdf 8:80 1 0B 0 disk
sdg 8:96 1 0B 0 disk
root@eagle:~# blkid
/dev/sda3: UUID="e235e39e-3465-45d0-b332-4dcd9f9a9b34" TYPE="swap" PARTUUID="b99a842c-03"
/dev/sda1: UUID="b3646d97-a9f1-47e9-bb09-d6cfd8972ddd" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="b99a842c-01"
/dev/sdb2: UUID="23acdf08-dad1-4666-9e0f-db88c2df0102" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="013d40b2-b2aa-47a9-9ebf-3ff4883006eb"
/dev/sdb1: UUID="b87bd478-ea55-4e3a-83aa-c4951832175b" TYPE="swap" PARTUUID="819f7865-15c6-4081-b9fa-6757f1b46728"
/dev/sdc1: UUID="b36c1873-2b3a-4e3d-92f8-89f1f442b5e5" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="wdbackup" PARTUUID="74d320ac-bb23-40fd-9af6-2d7afd9c642c"
/dev/sda2: UUID="44fbed5b-641b-43f1-852c-fb89f72247a4" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="b99a842c-02"
root@eagle:~#
If you compare that listing with my fstab (repeating the relevant lines here, see the whole file above). The lines beginning with # are comments:
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda1 during installation
UUID=b3646d97-a9f1-47e9-bb09-d6cfd8972ddd / ext4 errors=remount-ro 0 1
# /home was on /dev/sda5 during installation
UUID=44fbed5b-641b-43f1-852c-fb89f72247a4 /home ext4 defaults 0 2
# /oldhome was on /dev/sdc2 during installation
UUID=23acdf08-dad1-4666-9e0f-db88c2df0102 /oldhome ext4 defaults 0 2
# /dev/sda6 swap SSD
UUID=e235e39e-3465-45d0-b332-4dcd9f9a9b34 none swap pri=100 0 0
# swap was on /dev/sdc1 during installation
#UUID=b87bd478-ea55-4e3a-83aa-c4951832175b none swap sw 0 0
# /dev/sdg1
UUID=b36c1873-2b3a-4e3d-92f8-89f1f442b5e5 /media/backup ext4 defaults
You can see that fstab tells my system that b3646d97-a9f1-47e9-bb09-d6cfd8972ddd, also known as /dev/sda1, or the 1st partition of the 1st disk, is my file system root (/).
Then 44fbed5b-641b-43f1-852c-fb89f72247a4 (/dev/sda2, so the 2nd partition of the 1st drive) is my /home
Next, 23acdf08-dad1-4666-9e0f-db88c2df0102 (/dev/sdb2, the 2nd partition on my old hard disk, /dev/sdb) is /oldhome - a directory I created myself. You can also do that: mkdir /whatever (as root user).
e235e39e-3465-45d0-b332-4dcd9f9a9b34 (/dev/sda3, the 3rd partition of my 1st drive) is not visible as a directory, it’s like pagefile.sys on Windows (swap space).
b87bd478-ea55-4e3a-83aa-c4951832175b (/dev/sdb1) is commented out; it was my swap partition before I bought the SSD (which serves as the 1st disk, /dev/sda).
b36c1873-2b3a-4e3d-92f8-89f1f442b5e5 (as the command shows, it used to be /dev/sdg1 when I installed the system, now it’s /dev/sdc1, but the ID stayed the same) is my external drive, mounted as /media/backup, again a directory I created myself, like for /oldhome above.
In darktable, if you select an image on the lighttable, then hover over the image information, you can see the full path of the image:
As long as it’s some /mnt/<random ID>, it is going to be different every time when you reboot. BTW, are you sure you copied the images to the disk? Do you use darktable to do that? add to library does not copy anything, only copy & import does that:
So, you have to get your disk in order. Once you have proper, fixed paths, the problem will be solved. It’s not a darktable issue, but one of Linux system administration.
BTW, I’d suggest you get rid of snap, and use Firefox as a native package. I don’t know about Spotify, but in the worst case, you can listen to it from the browser.
You can follow this guide if you wish to do that. Firefox will load faster, and you won’t have those weird /snap mount points:
Thanks very much for all your assistance (and patience), appreciate it.
I’m not sure what is going on with my drives. I have two SSDs, one with a single partition, the other with two. But I’ve forgotten how I set them up. I checked the BIOS set up and it only shows one drive available for booting, even though there are two drives.
I also tried the blkid command and got:
dave@dave-B760M-DS3H-DDR4:~ blkid /dev/nvme0n1p2: UUID="9adebe28-7207-4e70-b951-7d75ba0590e1" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="5045de25-b53f-44d8-8c4c-28fc83fadf46" dave@dave-B760M-DS3H-DDR4:~
Just the one drive. But if I use nautilus or the disk utility, it shows two drives. Something is messed up somewhere, just not sure what. I shall work through it logically, see if I can figure out how things are configured and use your helpful suggestions.
Many thanks
Dave
