Digikam with large collections on Linux

Software digiKam
1

Hello,

I have been using Digikam for many years for managing my photo collection on Linux. However, the collection has grown large and now occupies nearly a full 4 TB drive and a half-full 14 TB drive. For the last year or so, Digikam has become quite unstable, frequently freezing and crashing.

I need to have machine tags in the metadata of the file, hence I am sticking with Digikam even though it can be frustrating. I have tried splitting it up into multiple collections and switching to MySql for data. It still freezes (and sometimes recovers) or crashes completely.

Has anyone managed to get Digikam to work with large collections? If so, how? If not, is there an alternative that can be used to manage large collections, where the tags need to be included in the metadata?

Thanks,
Derek

2

When I start having issues with a program, it is usually due to hardware rather than software on a linux system. How old are your drives? and what brand are they? How often are they on, how many hours would be a good measurement to see?

3

10+ TB is huge. What does that mean in terms of # of pictures and in your case especially tags? The help menu has a statistics entry for the data base.

I’d make sure the digikam databases (e. g. digikam4.db for SQLite) are on an SSD. Maintaining and cleaning up the DB (Extras menu) should also help.

4

I will disagree with you. According to my crystal ball: it’s bad/poor configuration of the MySQL. Why? Because he has provided:

  1. No hardware information
  2. No networking information
  3. No DK information/settings
  4. No MySQL information/settings

Anybody that deals with Linux would know to provide the basic info…

5

Drives are 1 year old and 3 years old. They are both Seagate.
“How often are they on, how many hours would be a good measurement to see?” - not sure what you mean there.

6

Apologies, I have only been using Linux since 1997. I need to gain some more practice :stuck_out_tongue:

Seriously, I was asking a generic question about whether other people got it working with large collections. I was not requesting technical help.

7

Systenm info -

System Details Report

Report details

  • Date generated: 2025-05-03 19:31:58

Hardware Information:

  • Hardware Model: ASUSTeK COMPUTER INC. PRIME X570-P
  • Memory: 32.0 GiB
  • Processor: AMD Ryzen™ 7 5700X × 16
  • Graphics: NVIDIA RTX A2000 12GB
  • Disk Capacity: 39.0 TB

Software Information:

  • Firmware Version: 3603
  • OS Name: Ubuntu 24.10
  • OS Build: (null)
  • OS Type: 64-bit
  • GNOME Version: 47
  • Windowing System: Wayland
  • Kernel Version: Linux 6.11.0-25-generic

No network is involved as the disks are internal.

There has been no MySQL settings changes, reinstalled Digikam a couple of weeks ago and used MariaDb fresh install. THe database is in /home/derek/Pictures which is on a SSD.

8

Too much info is missing:

  • There’s no hard drive with 39tb. Most likely some kind of nas.

Obviously there is:

  • Is it mMySQL or MariaDB?
  • Which version of DK and where from?
  • Where is the digikamrc/digikam_systemrc?
  • Where is the digikam4.db and the thumbnails-digikam.db?
  • What are the sql settings in digikam
  • What happens when running ‘show status;’ in mysql or mariadb?
  • What happens when running ‘show table status;’ in mysql or mariadb?
  • Have you run mysqlcheck?
9

No, that is the total disk capacity of the system. As noted, they are all internal drives.

I am going to put this on hold for now.
I am kind of stuck in work loops.
When I finish that and come up for air, I will do a fresh install of everything and start over.

I still don’t know if anyone else uses DK with such a large collection, which was what I really wanted to know.

10

Hi @dkeats,
Sorry you’re having trouble. This sounds like an absolutely massive collection. How many images are there? The actual size on disk of all the images is less important than the total number of images.

For VERY large collections (e.g. over 200,000 images), I suggest using an external MariaDB/MySQL database. In other words, not the embedded MariaDB that comes with digiKam. The reason is you want MariaDB in it’s own process space, and you can more easily tune a stand-alone MariaDB server. MariaDB doesn’t have to run on a different machine, just in it’s own isolated process. I also recommend you use fast SSDs to store the database files. digiKam is highly dependent on the database, and the faster the database is, the better the user experience will be.

Once you have MariaDB/MySQL running in an isolated process, you should start researching how to monitor and tune the DB. Look at parameters that include caching and query size.

I wish I had details I could give you on how to tune the DB, but Google and ChatGPT will be very useful.

Cheers,
Mike

11

Thanks Mike. I had a bit of time yesterday morning and that is exactly what I did. It was using its internal, file-based database, which I had not realised in the rush of life. I did a fresh install of MariaDb with nothing else using it, and then pointed DK at it. I then let it rebuild the collections from scratch, which I left it doing all night (tags are in the images so nothing lost in doing that). I then deleted other db files. It is running blazingly fast. There are about a million images. I been taking photos since 1970! The 39TB of disk space is not all photos, I do quite a lot of videos as well (using Kdenlive, they are on their own 14Tb drive). Anyway, now it seems to perform well within acceptable limits, and I am happy!

1 Like
12

And I should add how awesome it is that the DK developers thought about the need for large collections and gave it the ability to use an SQL database. A bazillion thank yous.

1 Like
13

You’re most welcome, @dkeats. We really try to stress digiKam with large libraries, and performance is always one of our top considerations, even on older hardware.

Of the core digiKam team, my collection is probably the smallest at a paltry 25,000 images.

Cheers,
Mike