Migrating multiple Lightroom catalogs to darktable.

Software darktable
1

I want to be able to maintain the grouping - or separation - of my images that the use of multiple catalogs in Lightroom gave me (switching between catalogs was very straightforward). Knowledgeable Lightroom users, like Tim Grey for example, assured us that multiple catalogs were not really necessary and they may well be right - but I preferred to use them and would like to continue that.

I suspect that I will get the same advice here - that there is no practical requirement for, or performance hit resulting from, managing all one’s digital image assets in the one darktable database - but that’s a debate I would prefer to defer.

Edit1: Ooops: I have failed to RTFM. Correcting that now, so please holster your weapons for a while!

Edit 2: Having read, my attempts to create a second database and library are not successful; could I get some advice on where I have gone wrong?

I invoked darktable from the command line with:

(path_to dt_executable)/darktable --library (path_to_new_empty_folder)/library.db --datadir (path_to_new_empty_folder)/data.db

where neither library.db nor data.db existed.

On the first attempt I was able to select images to be imported/add-to-libray but dt became unresponsive. For the second attempt I copied the lua and styles directories , plus darktablerc, keyboardrc, user.css and luarc from my ‘production’ install to the newly created folder for my test install. I tried importing/add-to-library just one image but the process never ended. I did not see any error messages posted.

For the 3rd and 4th attempts I also specified the --configdir and --cachedir directories, but now darktable did not even start.

What have I not done correctly ?

2

The command you have used will take configuration (darktablerc, keyboardrc etc.) from the default directory and will only create library database in your new_empty_folder. If you want to read everything from a different directory (configuration plus databases) you can just invoke with (path_to_dt_executable)/darktable --configdir (path_to_config_dir) but this would require you to maintain duplicate configuration in multiple places (a pain in the ass).

The option --datadir is not used to tell darktable where your data.db is and you shouldn’t use it (see the user manual for further info about what it actually does).

If you wish all of your libraries to use the same configuration but different image databases you just need to run (path_to dt_executable)/darktable --library (path_to_new_empty_folder)/library.db.

You don’t need a separate directory for each library – you could keep them all in the default config directory but with different names, for example (path_to dt_executable)/darktable --library (path_to_config_dir)/library2.db

Note that if your directories contain spaces you will need to use quotation marks to define file and directory names e.g. "(path_to dt_executable)/darktable" --library "(path_to_config_dir)/library2.db"

1 Like
3

Hmm, I don’t know why, but I can’t find anything in the relevant part of the user manual that tells me not to use ‘–datadir’ - but I listen to what you say.

One of my objectives is to create a separate tag dictionary for each ‘catalog’ (using that word in the LightRoom sense) because on the first (and most important, but not the largest) catalog that I have imported into darktable, the dictionary already has 670 entries, and I’m only about 60% of the way through tagging all its images. The next catalog will have a similar number of tags in its dictionary, but almost none of them in common with the current catalog. By keeping the catalogs separate, I avoid having a tag dictionary that may grow, across all catalogs, to be as long as 2500 to 3500 entries. Scrolling such a large tag dictionary, which requires manipulation of two scroll bars, becomes mpractical at that point.

Assuming that the tag dictionary is held only and entirely within the library,db, invoking darktable and specifying a different location for the library - or a different library name - would allow me to meet this objective - correct?

4

Apologies, based on your previous “I don’t understand technical stuff” comments, I was meaning to say that you shouldn’t use it, not that it shouldn’t be used. I thought you probably wouldn’t appreciate a technical description of what it actually does and referred you to the manual in case you wanted to find out.

I’m not sure this assumption is correct. Looking at it, data.db appears to contain a table called “tags” so you might not be able to keep your tag dictionaries separate like this. You might need to use the --configdir parameter and duplicate all your configuration. @phweyland might be able to confirm one way or another.

Let me just reiterate that this is not the recommended way to use darktable, hence why it’s not all that easy to support it without duplicating other stuff.

5

Thanks for this wise counsel: my objective is to avoid getting into a mess with tagging but not at the cost of creating an installation which is so unique that it becomes unreasonable to ask for support on solving problems.

6

Ah! Sacred Blue (or words to the effect, as pronounced by Peter Sellers): my ‘alternative’ database contains only the images I just added, but the tag dictionary is already populated and is identical to that in my ‘primary’ databse. Damn. Back to the drawing board then.

7

I confirm there is no provision in darktable to manage several tags dictionaries.

600 tags are not a lot. Correctly structured you should be able to manage a short list of first level tags (I’ve something like 15 first level entries for more than 2000 tags).

You may even want to limit them to two entries (one for each distinct dictionary you think about)…

8

OK - that’s an incentive for me to try harder: I have about 35 first level entries. But 2? I can’t get there!

For further clarification, the size/scrolling requirement of the tag dictionary was one (and probably the most pressing) reason for me to want to have multiple darktable databases. The elimination of the ‘multiple tag dictionaries’ option still leaves me other ‘issues’:

  • number of film-rolls and the amount of scrolling of the list
  • the number of characters required to show a unique film-roll name and hence the width of the left-hand panel
  • the effort required to rebuild the library in the pathological case where I have to restart by importing images
  • similarly, the effort required to rebuild the tag dictionary in the event that I cannot access an exported tag dictionary file.

On this last point: I assume that the tag dictionary cannot be rebuilt, automatically, by scanning of all tagged images ?

9

That’s the purpose of folders, isn’t it ?

Why not to be able to access your exported tags file ?

If you have the xmp files the dictionary will be rebuilt, but

  • only the tags which are used
  • potentially polluted by the simple tags (iptc keywords or xmp.subjects), I’m not sure any more.
10

Very simple, but credible reason: human error/law of unintended consequences. For example, a hardware failure which wipes out all of my config info. and I also discover that I have ‘misplaced’ the exported tag dictionary (‘misplaced’ being a euphemism for ‘I didn’t export it’).

btw, this is a very real scenario: I spent the last decade of my working life, up to 2000, consulting on preparing for and managing business continuity failure - of which IT failure was a major component. I found that the number of companies/individuals who ran business critical systems without adequate recovery capability always exceeded those who did. 20+ years on I don’t think much has changed.

11

Does the development effort/risk of regression to provide multiple dictionary support mean that this capability (multiple tag dictionaries) is unlikely to ever be provided ?

Thinking (superficially) about the idea of reducing the number of ‘root level’ tags to an absolute minimum - e.g. 2, as you suggest - seems to me to create a different issue: now instead of having a need to scroll the dictionary, at the root level, I have the need to open multiple levels in a hierarchy. This requires more mouse manipulation/clicking than scrolling. The more I reduce the number of tags at a given level in a hierarchy, the more levels I require, surely? With ‘only’ 670 tags, currently, I am finding that I have to do considerable (i.e. too much) scrolling with the two scroll bars (often in opposite directions) and opening of closed hierarchies, which have been closed when their top level has been scrolled out of the tag dictionary window.

On this last point I found that I was having to re-open multiple (>20) closed hierarchies many (>4) tens of times per day for each year folder of images in my current catalog. Often these hierarchies were not visible in the window and opening them required as many as 4 or 5 mouse clicks (to open levels) each time. I resorted to the trick of temporarily renaming a tree of tags to place at the top of the dictionary, and at a ‘root’ level, so minimizing the amount of effort required to see the tags and assign them to images. Those trees will have to be returned to their correct location, in the hierarchy, when tagging is complete. This will result in a large proportion of all my .xml files being updated, resulting in a need for another total backup refresh.

Overall though, I am finding that the task of assigning tags has gradually become slower as the size of the tag dictionary increases - not because of performance reasons but because of the amount of scrolling and clicking that is necessary.

A specific problem has been where I wish to assign multiple tags to a single image:

I have been using the technique of creating a collection on the basis of a named (or dated) folder, within a given year folder, combined with a ‘tag - not tagged’ metadata tag. This has the advantage of showing me only untagged images (a second pass will be required, someday, to make sure the previously assigned tags are the right ones). It has the disadvantage that as soon as I assign the first of multiple tags to an image, it disappears from the collection (because it is no longer ‘not tagged’). Now I have to remember which image it is that requires additional tags and which those additional tags are. With a catalog that is only about 15% of its ‘final’ size, yet contains thousands of images, I can’t manage this memory feat.

My workaround is to assign a ‘staging’ tag (with a name that puts it at the top of the dictionary) to every image in a ‘not tagged’ collection that requires multiple tags . Once that collection is fully tagged I then open a collection based only on the ‘staging’ tag. This is usually quite small (10 to 100 versus 100 to 1000 images of the ‘untagged’ collection) and I can proceed to apply the correct, multiple, tags and detach the ‘staging’ tag from these images.

Because of the power and functionality of the tagging module in dt this all works well - quick and reliable (by the way, the user guide gives a wholly inadequate view of the power of the tagging module). But is there a more efficient method I should be using ?

12

Hi, sorry to be nagging about this, but I do think that the idea of being able to have and use entirely separated (sandbox-like) darktable configurations concurrently has some merit - but requires the solution to issues that make it ‘not easy to support’. I don’t actually think this is a big challenge to all you demonstrably capable developers. It would, for example, overcome the restriction on there being only a single tag dictionary, without having to make any design changes to that module at all.

! Is it worth making this as a suggestion?

13

Entirely sandboxed configuration is already supported. You’re talking about shared configuration but with independent library and data databases, which is not quite the same thing.

It’s not a question of whether or not it is a challenge, but whether or not any of the developers think this is a feature worth working on and subsequently supporting.

By all means raise a feature request.

14

Are you sure you are not just trying to replicate a feature from another program (in this case Adobe Lightroom) within darktable, without actually looking at the problem you are trying to solve?

E.g. your issue with having to deal with 100s of untagged images at a time makes me think all those images are in one directory/film roll. The way the images are stored could actually be one of your “tags”. I (and afaik others here) store images in a directory by capture date. For me, it’s either one day’s worth, on one event’s worth of images per directory. In the case of an event, the name of the directory is the start date, plus a (hopefully informative) short text. Such directories can contain subdirectories if deemed useful.

From there, I’ve never more than a few hundred images to handle in one go.

1 Like
15

+1

And you can use digikam or a similar program to look/search all your images/tags if you ever want to find something specific.

16

There is a search box in the tags module… Why not use it?

17

There are reasons why I value this forum more highly than any of the others I use. Your reply is an excellent example: replies I get here almost always provide guidance but, just as importantly, ask me to think.

In this case I cannot be totally sure that I am not trying to replicate a Lightroom function, but I am sure I have looked at my current problem, based upon my experience with Lightroom (which I would describe as better than ‘average’ and, possibly, as good as ‘expert’).

Specifically, the images I am currently dealing with are from one of the multiple LightRoom catalogs I have which I ‘manage’ entirely within dt now, and have done for the past 2 years. Those images were contained in a directory structure, initially created about 12 years ago, used in Lr and retained in dt. That structure is, at the highest level, by year. Below that it was, initially, by day (so, only 2 levels deep), but then, for some years, was by year, then by subject/topic and then by date (3 layers). Finally, for the last 5 years (i.e. before ever heared about dt) the organisation, below the catalog name has been by year, month, day. The issue with almost all those images - up until 2022 - is that they were untagged or inappropriately/incorrectly tagged in Lr.

That is, I have a major tagging task on my hands that is not a consequence of my image organisation, but is a historic one. Starting from 2022 my images are tagged immediately after my culling step, so there is no problem going forward from here with new images. But this ‘historic’ task is just going to get bigger as I turn to tackle the other Lr catalogs I have, which have been largely untouched since my move from Lr.

And it is this ‘stage’ of my migration that I think will benefit from the idea of having what I describe as ‘sand-boxed’ dt configurations. I’m sure that the developers will give technical or ‘principle’ reasons for rejecting this suggestion (if that is appropriate), but I don’t think there is anything intrinsically incorrect in “trying to replicate a feature from another program” - is there?

On the other hand, your suggestion "The way the images are stored could actually be one of your “tags”. " is most interesting and extends my understanding. Thanks for this

18

I agree - and that is what I do, heavily, so I assume your comment is aimed at the post immediately preceding yours?

19

Any views on a ‘similar program’? I looked at digikam (and iMatch) and decided they were not for me. The non-DAM functions in DigiKam are out-classed by dt. The DAM functions in iMatch are industrial-grade - an order of magitude more powerful than my needs or ability to exploit.

20

I use both digikam and darktable, digikam for initial sorting and tagging, darktable for the editing and further use. Mostly because I don’t like the darktable interface for adding tags, captions, titles (although the later versions have much improved, notably in allowing multiline edit for the description/caption) and find digikam much easier to use for that.
So the workflow is:

  • Digikam
    • import from memory card into the (digikam/darktable) directory tree
    • initial renaming, and accept/reject with deletion of rejected images
    • add tags, captions, titles
  • switch to darktable
    • import images (just"Add to library", not “Copy and import”)
    • edit
    • export when needed

Of course, this requires that you set both digikam and darktable to use sidecar files. As I don’t keep derived files in the same place as the originals, I use sidecars for all file types, as I don’t like writing to my original image files (and for raw, writing to them is not even possible in several cases).

In my experience, tags, captions and titles are then automatically transferred between digikam and darktable (certainly from digikam to darktable). Make sure you attach a tag with all its parents in digikam. If you don’t do that, dt tags become a mess (which is logical, and not any fault of darktable: it can only work with what you give it).

With my settings, later changes in tags, applied through digikam, are picked up automatically in darktable on startup. Such changes can happen when I decide to split a category or group a number of tags within a larger category. Not the most pleasant job if you have many tags to handle, but quite doable.

Note that this means I don’t use (and basically ignore) the tagging module in darktable.
Perhaps not a system for everyone but (shrugs) it works for me.

And if you already have one or more directory trees with your images, you can use those “as is” as a collection in digikam, no need to copy your images to a specific dedicated directory.