Importing 20,000 images

I don’t use film rolls, just the other DAM features. If you always import to the same top-level folder, there is a lua script you can use to rescan your folder when DT starts.

2 Likes

I think imported near 23,000 or 25,000 images when I converted over to Darktable 1.6 or 1.5 back in 2015. I was using Linux at the time though so I can’t attest to the feasibility of that in Windows. It went fine, just took a while. I was running a hexacore Haswell-E at the time so my machine wasn’t a slouch for the era. But the way I organize my stuff on disk helped.

I break things out into four major directories: Clients, Projects, Models, and Photos by Year (where most of my day to day personal photos end up). I have subdirectories for different jobs, clients and subjects under those. IIRC I imported all four of the major directories at once and let it cook overnight.

I don’t use filmrolls either, just the folder feature. Occasionally I narrow down a view by searching with a piece of metadata. Although does anyone know how to get Darktable to not revert back to the filmroll view after an import? That has eluded me since I started using it.

2 Likes

I got this too, but on MacOS. Exactly the same with the ‘darktable not responding’ message, and everything seemed frozen - on a Mac we get the BOD (beachball of death). My import was for 55,000 images, but I tried it on individual year folders to lower the workload, averaging 2-3000 images at a time, and the same ‘freeze’ occurred.

A Force Quit and relaunch clears it, and sometimes it completed the import, and all the folders and files are there. Sometimes it hadn’t completed, but re-running the import then worked fine, with a progress bar, and a completed job.

I do use the recursive folders option, and I do get a film roll listed per lowest level folder. However, as there are a lot of duplicated names there (day/date+Location named folders) this isn’t so useful in this respect - although quite accurate in principle, it’s not my method of filing.

Cheers

1 Like

Film roll/import related. I use YYYY/MMDD_jobcode structure. Unfortunately this means the darktable film rolls en up as MMDD without showing the year. Thought I was clever cutting down path lengths (frequent cli user) as the year was implied by the folder structure…

Got some more info here, I think.

I’ve tried this again, but this time I built darktable using MacPorts. By running from Terminal I think I can see what’s happening, the apparent ‘not responding’ seems to be caused by it importing data from the xml sidecar files that I have associated with Lightroom.

I can see tags being built, but very slowly one at a time - one or two a minute.

Don’t know if that’s relevant, if you also have xmp sidecar files.

1 Like

You can show more than one level on the film roll interface (I don’t remember where is the configuration for that, I’m not on the computer now)

I have +50K photos, but do not use Darktable to manage them. In my humble opinion, that’s not the idea Darktable is created for. I organize and tag my photos with Digikam and is here where I begin my photos processing: Deleting the photos I consider have not quality and assigning flags and tags. After that, I launch Darktable over any photo to process it. Once I have the image open, I can make Dartable to open the entire folder.

Anyway, Digikam took several hours (even 8) to read and process all photos (stored in lot of subfolders) from a local hard disk.

It’s a pity that Digikam can’t see a thumbnail of the photos processed with Darktable, cause that forces user to open each RAW image Darktable processed to see the result. I think Darktable should include (in XMP file or another way) a little thumbnail readable by Digikam, Gwenview … that make the DAM flow less complicated.

1 Like

You may be thinking of the option “number of folder levels to show in lists”, which is under “miscellaneous” in the GUI options.

In any case, once you have imported a tree structure, under “collect images” you will see the lowest level folders represented there, and the option above determines how many folder levels above the lowest are shown.

That will be so much better for the workflow!

Also, prioritizing the “Metadata filters” above the thumbnails scan will be logical. If for example I would like to select only JPG files, the program doesn’t have to scan the rest of the RAW file in the map. In the current version 5.8 I need to wait until all the thumbnails are loaded ONLY THEN I can select the metadata filter. With hundreds of photos in the map it takes more than 10 min only to start the work

Hi, could you elaborate a bit more on this ? dt has made some few progresses on this side. And it would be interesting to know from you what is still missing the most.
Thanks

For me digiKam is appealing for the management, because I have a more intuitive approach to the tagging of images, can very easy edit Tags, rename, reorganize tagging structure. That e.g. comes handy, when you have a structure of tags and want to re-arrange items of the structure. I also am more looking into face-recognition to allow automated tagging of older images. Here digiKam 7 seems to be quite powerful.

Have you checked the tag features in the current dt ? What are your main grievances ?

If I’m not mistaken there is a dt plugin which does that.

I don’t think there is a plugin offering the functionality and usability of the new features for “face management” coming with digiKam 7.

Regarding tag features. The capabilities and usability of digiKam here in my point of view clearly outperform darktable (what is fine. I do not expect darktable to get a full fledged digital asset management. I use it for raw development - with really stunning capabilities.

Just one example. I have a picture database with several 10K of images. Grown over the years … So did my tags structure. While I now work with more hierarchical tags ( sport|trackfield|outdoor|highjump) I had in the past a more flat structure with just “highjump” as tag. In digiKam I can just drag the “old” tags into the structure and let digiKam merge the tags in a way that now all images are tagged with my new hierarchical tag. That is just one drag and drop activity. I don’t think that dt is capable of this level of metadata management. And as said, I don’t expect this. Linux philosophy helps me here to deploy for any task the best suitable tool And for DAM this currently in my view is not darktable. Whereas digiKam for me is not the suitable raw developer… Both are useable side-by-side. So all fine…

1 Like

Thanks for sharing your view.

I read about this approach every now and then but can’t imagine how this should work. For me tagging is essential to find images and select them for later processing and exporting for certain tasks.

I don’t like to select images in one application and then have to open them in another application (one by one?) to export for the given task in a given format etc.

Of course. Digikam is like a Swiss army knife, not only let you tag media (not only photos), manage filesystem (create/move/delete folders not only files) and apply in batch mode anything you can do with ShowPhoto (its destructive photo editor), but you can do search in (even MariaDB) database by tags, filenames, folders names, time, photo size, formats … and do it very well.

Darktable it’s a headache when you make any change to filenames, folder names or delete files (outside it) . User needs to launch scripts to make Darktable notice that there are changes in some folder. In Digikam this can be done automatically or forced by user within Digikam.

I consider Darktable is more RAW operations oriented, but lacks on filesystem management.

I share this standpoint. On my side I feel that now dt is enough easy to use to manage properly all my structured tags, categories or other needed metadata, including what I want to export or not.
I agree that Digikam has more DAM features, but they are not useful for me. And moreover I totally dislike the permanent constraints and duplication of information related to the different xmp idioms. Kind of babel tower.

I agree that dt could be better on this. So I don’t change them outside dt. :slight_smile:

1 Like

I have a doubt about Digikam (or any other DAM), guess this is a good thread to ask. Since digikam cannot show all the edits done to the raw file in dt, I will have to export all as jpegs. Now, does digikam keep a link between the raw and the jpeg? While browsing through the jpegs, if I right click and say “edit”, will it open up the correct raw file in dt?

No: as far as I know, even grouping them will not help, as digikam always opens the active file for editing (in this case, the jpg). Otoh, if you open the raw file for editing with darktable, it will continue where you left off the last time

As @rvietor wrote, when you tell Digikam to open a JPG file will open it, not the related RAW. Remember Digikam is not a RAW editor in the way Darktable is.