A newcomer to darktable, I am trying to learn. When I installed darktable 3.0.0 on Windows, I decided to import all the images in my collection - about 20,000. I encountered problems, where darktable would become non-responsive and the progress bar of the import would stop. I then had to repeatedly force-close darktable and start it again. After about 8-10 restarts, the whole collection was imported.
Two questions:
Was I stupid to import all my images? What do others do?
Is this a known bug or should I report it on Github?
Wellā¦ Importing 20k images is heavy I/O operation soā¦ Itās problematic. It isnāt a bug per se, but for each image darktable has to open it, parse, add XMP file (depending on settings), add it to database etcā¦ Generally such heavy task should be offloaded to another thread + progressbar should be present but who imports 20k+ images regurarly?
In your case as a workaround Iād leave darkable on importing it overnight
However yeah - theoretically a āproperā solution should be low-priority import in background thread and allowing you to edit 1st imported image while continously adding new images to collectionā¦
Did you really import 20k images in one single step (as one single filmroll) ? Im my opinion this is not a good idea. darktable uses the concept of filmrolls and collections. This allows to structure your data. I rarely import more than 200 images as one filmroll. See https://www.darktable.org/usermanual/en/lighttable_concepts.html and the following page for more info.
I wondered about this when I first tried darktable coming from Lightroom. I had expected to see something like my folder structure or similar to Library view in LR, but dt does it somewhat differently. My first attempts way back resulted in dt crashing with too many imported photos and Iām sure thatās all fixed now but at the time it resulted in my current method. I organize my photos outside of dt, then import a single folder - edit and export, then remove the folder. Likely not how it was intended but works fine for me especially since I donāt use the tagging/search features very much.
OK, I can report that it is only the GUI that freezes (after about 2-3,000 images) and is reported by Windows as āNot respondingā. In the background, the import continues. If you wait long enough, you can then close and restart the application and after that all images will have been imported.
I get the point about filmrolls. I had thought that if you did a recursive import of folders within folders, each lowest level folder would become a film roll. Now I see that this is not the case. The user manual does not make this clear.
I guess I was too ambitious but wanted to get some light-touch DAM functionality out of darktable. If this requires you to manually go through all your shoots from the past and import them as a filmroll each, that is too much work. I will simply stick to importing what I intend to process.
I donāt use film rolls, just the other DAM features. If you always import to the same top-level folder, there is a lua script you can use to rescan your folder when DT starts.
I think imported near 23,000 or 25,000 images when I converted over to Darktable 1.6 or 1.5 back in 2015. I was using Linux at the time though so I canāt attest to the feasibility of that in Windows. It went fine, just took a while. I was running a hexacore Haswell-E at the time so my machine wasnāt a slouch for the era. But the way I organize my stuff on disk helped.
I break things out into four major directories: Clients, Projects, Models, and Photos by Year (where most of my day to day personal photos end up). I have subdirectories for different jobs, clients and subjects under those. IIRC I imported all four of the major directories at once and let it cook overnight.
I donāt use filmrolls either, just the folder feature. Occasionally I narrow down a view by searching with a piece of metadata. Although does anyone know how to get Darktable to not revert back to the filmroll view after an import? That has eluded me since I started using it.
I got this too, but on MacOS. Exactly the same with the ādarktable not respondingā message, and everything seemed frozen - on a Mac we get the BOD (beachball of death). My import was for 55,000 images, but I tried it on individual year folders to lower the workload, averaging 2-3000 images at a time, and the same āfreezeā occurred.
A Force Quit and relaunch clears it, and sometimes it completed the import, and all the folders and files are there. Sometimes it hadnāt completed, but re-running the import then worked fine, with a progress bar, and a completed job.
I do use the recursive folders option, and I do get a film roll listed per lowest level folder. However, as there are a lot of duplicated names there (day/date+Location named folders) this isnāt so useful in this respect - although quite accurate in principle, itās not my method of filing.
Film roll/import related. I use YYYY/MMDD_jobcode structure. Unfortunately this means the darktable film rolls en up as MMDD without showing the year. Thought I was clever cutting down path lengths (frequent cli user) as the year was implied by the folder structureā¦
Iāve tried this again, but this time I built darktable using MacPorts. By running from Terminal I think I can see whatās happening, the apparent ānot respondingā seems to be caused by it importing data from the xml sidecar files that I have associated with Lightroom.
I can see tags being built, but very slowly one at a time - one or two a minute.
Donāt know if thatās relevant, if you also have xmp sidecar files.
I have +50K photos, but do not use Darktable to manage them. In my humble opinion, thatās not the idea Darktable is created for. I organize and tag my photos with Digikam and is here where I begin my photos processing: Deleting the photos I consider have not quality and assigning flags and tags. After that, I launch Darktable over any photo to process it. Once I have the image open, I can make Dartable to open the entire folder.
Anyway, Digikam took several hours (even 8) to read and process all photos (stored in lot of subfolders) from a local hard disk.
Itās a pity that Digikam canāt see a thumbnail of the photos processed with Darktable, cause that forces user to open each RAW image Darktable processed to see the result. I think Darktable should include (in XMP file or another way) a little thumbnail readable by Digikam, Gwenview ā¦ that make the DAM flow less complicated.
You may be thinking of the option ānumber of folder levels to show in listsā, which is under āmiscellaneousā in the GUI options.
In any case, once you have imported a tree structure, under ācollect imagesā you will see the lowest level folders represented there, and the option above determines how many folder levels above the lowest are shown.
Also, prioritizing the āMetadata filtersā above the thumbnails scan will be logical. If for example I would like to select only JPG files, the program doesnāt have to scan the rest of the RAW file in the map. In the current version 5.8 I need to wait until all the thumbnails are loaded ONLY THEN I can select the metadata filter. With hundreds of photos in the map it takes more than 10 min only to start the work
Hi, could you elaborate a bit more on this ? dt has made some few progresses on this side. And it would be interesting to know from you what is still missing the most.
Thanks
For me digiKam is appealing for the management, because I have a more intuitive approach to the tagging of images, can very easy edit Tags, rename, reorganize tagging structure. That e.g. comes handy, when you have a structure of tags and want to re-arrange items of the structure. I also am more looking into face-recognition to allow automated tagging of older images. Here digiKam 7 seems to be quite powerful.
I donāt think there is a plugin offering the functionality and usability of the new features for āface managementā coming with digiKam 7.
Regarding tag features. The capabilities and usability of digiKam here in my point of view clearly outperform darktable (what is fine. I do not expect darktable to get a full fledged digital asset management. I use it for raw development - with really stunning capabilities.
Just one example. I have a picture database with several 10K of images. Grown over the years ā¦ So did my tags structure. While I now work with more hierarchical tags ( sport|trackfield|outdoor|highjump) I had in the past a more flat structure with just āhighjumpā as tag. In digiKam I can just drag the āoldā tags into the structure and let digiKam merge the tags in a way that now all images are tagged with my new hierarchical tag. That is just one drag and drop activity. I donāt think that dt is capable of this level of metadata management. And as said, I donāt expect this. Linux philosophy helps me here to deploy for any task the best suitable tool And for DAM this currently in my view is not darktable. Whereas digiKam for me is not the suitable raw developerā¦ Both are useable side-by-side. So all fineā¦
I read about this approach every now and then but canāt imagine how this should work. For me tagging is essential to find images and select them for later processing and exporting for certain tasks.
I donāt like to select images in one application and then have to open them in another application (one by one?) to export for the given task in a given format etc.
but wanted to get some light-touch DAM functionality out of darktable. If this requires you to manually go through all your shoots from the past and import them as a filmroll each, that is too much work. I will simply stick to importing what I intend to process.