I really appreciate the improvements to copy and import now available in DT. I am trying to move away from Adobe’s LR to import and catalog my images by date. DT is now able to import and place images into folders by date. That is really great and I do plan to switch to DT for imports from memory cards because of this capability. However, LR can actually tell what is a new image from the source and what has already been imported. This ability of the LR software is really great for synching backup drives and downloading only the new images from your camera’s card. I am hoping one of the developers would see this issue as worthy of their time. I used to looking at DT as an editing only program but its ability to import and manage images has grown so much.
Rapid Photo Downloaded does this very well and a lot more. Its worth checking out.
“Select only new pictures” is already an option on the import dialog box
Hi Mark,
I am running DT 3.8 flatpak on Linux Mint. I do not have the option to select new, although I see in the user guide that I should. When I get back home from travelling I will check out DT 3.8 on my Windows computer and hopefully the option will be there.
that’s only true for ‘add to library’, but not for ‘copy & import’
Indeed. It might be possible to do so for copy & import as well, but it’s a lot harder to get right. add to library doesn’t move files around, so if a file with a given path and timestamp is in the library, there’s no need to import it again. And checking that requires only reading the directory, not each individual file.
This doesn’t work when files are copied from e.g. a memory card, where not even the access path is guaranteed between sessions. So how do you check if a file isn’t in the database yet?
Yes, I see it is only available in add to Library. So my suggestion for a potential improvement still stands. The reason it would be useful is that you could keep synching or updating external backup drives to add only new images or if taking only the latest images from your camera card. It would be useful. I hope a developer agrees that it would be worthwhile.
That sounds like you don’t want to see new files on import, but files not yet saved to backup. In other words, copying files already in darktable.
That’s a completely different task, more suited to a backup program (under linux, rsync comes to mind, or @paperdigits’ suggestion of git-annex for a more complex, and more powerful, solution).
I hate to use the LR word here, but currently I use LR catalogs to backup new images (mainly edits) from my working external drive and copy them into my backup external drive. LR does this really well and DT has nearly got to the point for copying and importing that I hope to the Adobe in the position to stop using LR. This one extra improvement to copy and import is needed for that to be the case.
The issue is that LR catalogs are not really a backup. They can get corrupted and can get very large in size. LR does backups of the catalog to help of there are issues but it also needs to know the location of the files.
In LR you can also request it to create xmp files (or saved within the dang). The xmp file contains the tags and edit info for the image. They are a better backup option of your edit instructions. An export of the processed image would be the most important to do a backup.
DT uses the xmp approach. Your edits are saved as xmp files with your raw files. DT does not have a catalog. It has a library database that helps find images for edits but your edits remain in the xmp.
Therefore to do a backup all you need is to copy the images/xmp from your folder in your HDD to your backup HDD. There are backup software that will do this automatically (monitor for edits).
In my opinion the copy/import in DT should only be used when you want to move your files from your camera card to your harddisk and you want to rename them. Once they are in your harddisk in the folder structure you want, all you need to do is to Add to library.
PS. I keep a local backup but I’ve switched to cloud based backups (eg. Backblaze, pCloud, Sync). The option to constantly have a backup external to my house is really good. The cost keeps coming down and my internet speeds are faster that makes it a great solution.
Thanks G-man. I appreciate your response, but I am not interested in the catalog in LR or even DT. If I do an edit I crystalise the result by exporting it as a new image. What I am suggesting is that LR is really good at identifying new images from one location (card or drive) and making a copy of that file to the second drive. I have a unique catalog for each drive to facilitate this. I really hope DT can add this feature, but I appreciate if I am not able to code it myself I just have to hope a developer can see the merit and value in doing this. BTW, I often trash all my .xmp files because as both DT and myself improve I am not that interested in the editing steps from a previous session. To me DT is like the film darkroom that I use to make my living from. Each time I walk in there I want to do a better job than the time before.
Ok. Maybe I don’t understand what LR is doing and the benefit.
That’s not correct. The library contains the editing instructions. You can (at your own risk, I would never recommend this) even disable writing XMP files. Also, there’s an option to check for updated XMP files (updated compared to what you have in the library database – again, not something I use). See darktable 3.8 user manual - sidecar files :
In addition to the sidecar files, darktable keeps all image-related data in its library database for fast access. […] If an image is subsequently re-imported, the database will be updated from the contents of its XMP file.
Once an image has been imported into darktable, the database entries take precedence over the XMP file. Subsequent changes to the XMP file by any other software are not visible to darktable – such changes will be overwritten the next time darktable synchronizes the file. On request, darktable can be configured to search for updated XMP files at startup, offering a choice to update the database or overwrite the XMP file where changes are identified.
It was my understanding that the database had tags/colors and such but not the edit of each image. I will need to see what the code is actually doing.
You can try this:
- disable writing XMP files in preferences
- import an image
- changes some value (e.g. bump exposure by 5 EV, that’s highly visible :-D)
- exit & restart darktable
Don’t forget to turn writing XMP files back on.
LR is a brilliant digital asset management system (DAM). I bought a perpetual licence for LR6 and love the DAM capabilities. I never thought of DT as anything more than a great editing program, but now it is really improving in the DAM capabilities. I do not want DT to become a free LR, but I can just see how a small improvement would really improve the copy and import capabilities,
For example, I am currently scuba diving in Thailand. I downloaded the images from one days diving to my computer for safety and backup. I then went out the next day and returned to the hotel and wanted to download only the images taken that day. I was able to do this, but the task would be so much more efficient and powerful if I had the option to just copy and import new images from the camera.
Thanks for your response Kofa. I do not have any issues with DT database or xmp files. I have set my preferences to only write an xmp file when an image is edited. This cuts back on many unnecessary xmp files. I have never ask the developers of DT to replicate the DAM capabilities of LR, but DT has become so good at DAM now that I throw up this one (hopefully) little improvement which will let me say good bye to Adobe for good. I am even switching some of my computers to Linux to try to break Microsoft’s grip on my life.
Keep in mind that traditionally, Linux uses specialised tools for each task. Extreme examples of this are programs like find, sed, awk, grep, which are meant to be used (also) as components to do one sub-task within a larger task.
In this case, that would mean using a different program to download files, and another tool to keep track of new files to be written to a backup location. MS-Windows software is much more oriented towards “one program does all”.
And darktable is originally a Linux program, with MacOS and MS-Windows builds provided when/if there are volunteers to do so. That’s why you are likely to recommendations for different programs to do what you want.
A big advantage of that approach is that each team only deals with what they know best. Disadvantage is that the user has to learn more interfaces/GUIs…
Now I understand the request and I think it is useful. Your example of traveling is a good one since I dont like to delete my images from the SD cards when I travel in case the laptop gets stolen/lost.
I think it would be best for you to type it into Guthub under issues with a Feature Request tag. I would avoid saying: like LR.
Request:
DT to track previously imported images to avoid importing them again. This should be a similar functionality to the Add Library Select only new images.
Rational:
When traveling, images from the SD card can be imported each day without deleting the images from previous days from the card. The ability to only import new images would make it easier to track and avoid duplicates or missing importing some.
While this doesnt mean the feature would be added, it provides a clear message on the functionality you are suggesting.