Well… Importing 20k images is heavy I/O operation so… It’s problematic. It isn’t a bug per se, but for each image darktable has to open it, parse, add XMP file (depending on settings), add it to database etc… Generally such heavy task should be offloaded to another thread + progressbar should be present but who imports 20k+ images regurarly?
In your case as a workaround I’d leave darkable on importing it overnight
However yeah - theoretically a “proper” solution should be low-priority import in background thread and allowing you to edit 1st imported image while continously adding new images to collection…
Did you really import 20k images in one single step (as one single filmroll) ? Im my opinion this is not a good idea. darktable uses the concept of filmrolls and collections. This allows to structure your data. I rarely import more than 200 images as one filmroll. See https://www.darktable.org/usermanual/en/lighttable_concepts.html and the following page for more info.
I wondered about this when I first tried darktable coming from Lightroom. I had expected to see something like my folder structure or similar to Library view in LR, but dt does it somewhat differently. My first attempts way back resulted in dt crashing with too many imported photos and I’m sure that’s all fixed now but at the time it resulted in my current method. I organize my photos outside of dt, then import a single folder - edit and export, then remove the folder. Likely not how it was intended but works fine for me especially since I don’t use the tagging/search features very much.
OK, I can report that it is only the GUI that freezes (after about 2-3,000 images) and is reported by Windows as “Not responding”. In the background, the import continues. If you wait long enough, you can then close and restart the application and after that all images will have been imported.
I get the point about filmrolls. I had thought that if you did a recursive import of folders within folders, each lowest level folder would become a film roll. Now I see that this is not the case. The user manual does not make this clear.
I guess I was too ambitious but wanted to get some light-touch DAM functionality out of darktable. If this requires you to manually go through all your shoots from the past and import them as a filmroll each, that is too much work. I will simply stick to importing what I intend to process.
I don’t use film rolls, just the other DAM features. If you always import to the same top-level folder, there is a lua script you can use to rescan your folder when DT starts.
I think imported near 23,000 or 25,000 images when I converted over to Darktable 1.6 or 1.5 back in 2015. I was using Linux at the time though so I can’t attest to the feasibility of that in Windows. It went fine, just took a while. I was running a hexacore Haswell-E at the time so my machine wasn’t a slouch for the era. But the way I organize my stuff on disk helped.
I break things out into four major directories: Clients, Projects, Models, and Photos by Year (where most of my day to day personal photos end up). I have subdirectories for different jobs, clients and subjects under those. IIRC I imported all four of the major directories at once and let it cook overnight.
I don’t use filmrolls either, just the folder feature. Occasionally I narrow down a view by searching with a piece of metadata. Although does anyone know how to get Darktable to not revert back to the filmroll view after an import? That has eluded me since I started using it.
I got this too, but on MacOS. Exactly the same with the ‘darktable not responding’ message, and everything seemed frozen - on a Mac we get the BOD (beachball of death). My import was for 55,000 images, but I tried it on individual year folders to lower the workload, averaging 2-3000 images at a time, and the same ‘freeze’ occurred.
A Force Quit and relaunch clears it, and sometimes it completed the import, and all the folders and files are there. Sometimes it hadn’t completed, but re-running the import then worked fine, with a progress bar, and a completed job.
I do use the recursive folders option, and I do get a film roll listed per lowest level folder. However, as there are a lot of duplicated names there (day/date+Location named folders) this isn’t so useful in this respect - although quite accurate in principle, it’s not my method of filing.
Film roll/import related. I use YYYY/MMDD_jobcode structure. Unfortunately this means the darktable film rolls en up as MMDD without showing the year. Thought I was clever cutting down path lengths (frequent cli user) as the year was implied by the folder structure…
I’ve tried this again, but this time I built darktable using MacPorts. By running from Terminal I think I can see what’s happening, the apparent ‘not responding’ seems to be caused by it importing data from the xml sidecar files that I have associated with Lightroom.
I can see tags being built, but very slowly one at a time - one or two a minute.
Don’t know if that’s relevant, if you also have xmp sidecar files.
I have +50K photos, but do not use Darktable to manage them. In my humble opinion, that’s not the idea Darktable is created for. I organize and tag my photos with Digikam and is here where I begin my photos processing: Deleting the photos I consider have not quality and assigning flags and tags. After that, I launch Darktable over any photo to process it. Once I have the image open, I can make Dartable to open the entire folder.
Anyway, Digikam took several hours (even 8) to read and process all photos (stored in lot of subfolders) from a local hard disk.
It’s a pity that Digikam can’t see a thumbnail of the photos processed with Darktable, cause that forces user to open each RAW image Darktable processed to see the result. I think Darktable should include (in XMP file or another way) a little thumbnail readable by Digikam, Gwenview … that make the DAM flow less complicated.
You may be thinking of the option “number of folder levels to show in lists”, which is under “miscellaneous” in the GUI options.
In any case, once you have imported a tree structure, under “collect images” you will see the lowest level folders represented there, and the option above determines how many folder levels above the lowest are shown.
Also, prioritizing the “Metadata filters” above the thumbnails scan will be logical. If for example I would like to select only JPG files, the program doesn’t have to scan the rest of the RAW file in the map. In the current version 5.8 I need to wait until all the thumbnails are loaded ONLY THEN I can select the metadata filter. With hundreds of photos in the map it takes more than 10 min only to start the work
Hi, could you elaborate a bit more on this ? dt has made some few progresses on this side. And it would be interesting to know from you what is still missing the most.
Thanks
For me digiKam is appealing for the management, because I have a more intuitive approach to the tagging of images, can very easy edit Tags, rename, reorganize tagging structure. That e.g. comes handy, when you have a structure of tags and want to re-arrange items of the structure. I also am more looking into face-recognition to allow automated tagging of older images. Here digiKam 7 seems to be quite powerful.
I don’t think there is a plugin offering the functionality and usability of the new features for “face management” coming with digiKam 7.
Regarding tag features. The capabilities and usability of digiKam here in my point of view clearly outperform darktable (what is fine. I do not expect darktable to get a full fledged digital asset management. I use it for raw development - with really stunning capabilities.
Just one example. I have a picture database with several 10K of images. Grown over the years … So did my tags structure. While I now work with more hierarchical tags ( sport|trackfield|outdoor|highjump) I had in the past a more flat structure with just “highjump” as tag. In digiKam I can just drag the “old” tags into the structure and let digiKam merge the tags in a way that now all images are tagged with my new hierarchical tag. That is just one drag and drop activity. I don’t think that dt is capable of this level of metadata management. And as said, I don’t expect this. Linux philosophy helps me here to deploy for any task the best suitable tool And for DAM this currently in my view is not darktable. Whereas digiKam for me is not the suitable raw developer… Both are useable side-by-side. So all fine…
I read about this approach every now and then but can’t imagine how this should work. For me tagging is essential to find images and select them for later processing and exporting for certain tasks.
I don’t like to select images in one application and then have to open them in another application (one by one?) to export for the given task in a given format etc.
Of course. Digikam is like a Swiss army knife, not only let you tag media (not only photos), manage filesystem (create/move/delete folders not only files) and apply in batch mode anything you can do with ShowPhoto (its destructive photo editor), but you can do search in (even MariaDB) database by tags, filenames, folders names, time, photo size, formats … and do it very well.
Darktable it’s a headache when you make any change to filenames, folder names or delete files (outside it) . User needs to launch scripts to make Darktable notice that there are changes in some folder. In Digikam this can be done automatically or forced by user within Digikam.
I consider Darktable is more RAW operations oriented, but lacks on filesystem management.
but wanted to get some light-touch DAM functionality out of darktable. If this requires you to manually go through all your shoots from the past and import them as a filmroll each, that is too much work. I will simply stick to importing what I intend to process.