There is this fairly old repository providing lua code for git annex integration in darktable and I really like the idea of it since it fits my workflow perfectly.
However - because of previous breaking changes in darktable’s LUA API the code needs some love.
Even though my LUA skills are basically non-existent I took the initiative and implemented some changes.
Development is currently within my local repository only but will be pushed soon.
What I have achived so far:
adapt code to work with current LUA API
replace sequential get/drop actions with bulk get/drop actions making use of git annex -J parameter (speeds up git annex get/drop significantly due to parallelism in multiple threads)
What I am planning to do:
lighttable module with action control buttons (as addition to keyboard shortcuts) and configuration inputs.
automatically add pictures to annex on import
Looking for further ideas! I know, there are some users in this forum that make use of git-annex as well.
Wow, very ambitious…good luck and thanks for sharing this. I look forward to monitoring your progress. It has always sounded like a good Idea to me but I have never gathered up the initiative to get it working…
Hmm…It is early morning while I read this and my brain is a bit/lot foggy. I don’t really understand what this means. I have a feeling that it is something very good and thanks for working on it, but please explain for dunderheads like me. Thanks.
git-annex allows managing large files with git, without storing the file contents in git. It can sync, backup, and archive your data, offline and online. Checksums and encryption keep your data safe and secure. Bring the power and distributed nature of git to bear on your large files with git-annex.
The integration of git-annex within darktable basically provides an interface for these git-annex specific functions, that are usually only accessible via command line interfaces or for instance within the system file browser.
Also big thanks from me for picking this up! I am managing my collection since years with git-annex, and having some control from inside darktable would be such a workflow boost.
During the years I’ve been thinking about possible git-annex integrations a lot, and here are some comments that came into my mind which might be of interest:
A reasonable workflow with annex is to not have all images available for darktable all the time. To efficiently work with annex from inside darktable, I always thought that there needed to be a mechanism for persistent preview images cache, such that for annexed files, there’s never the skull shown, and you know what you are fetching from another medium.
Automatic translation of image rating into annex metadata would be handy, such that for images with a reject rating, the annex.numcopies could be set to a lower than default value, and for 3+ stars rating, the numcopies could e.g. be set to a higher than default value. Also, images of particular importance (e.g. with 3+ stars or with a special tag) could be additionally sent to a special remote that backups only the files of particular importance.
I agree and I was able to tackle the issue regarding missing previews by enabling “use disk cache for preview images” in settings → lighttable. You may want to check if that suits your needs
That sounds as a useful feature. To be honest I haven’t been aware about the metadata feature in git-annex up until now nor that you can actually manipulate annex.numcopies for a subset of a repository that way. I’ll have to play around with it. But I like the idea. This could actually replace/improve my very conservative approach of a global annex.numcopies=2 for my whole collection.
I am using the disc cache at least for the thumbnails, but it feels a little unreliable to me. At least, I have skulls from time to time, which means that either the cache is deleted externally (not likely, I am not messing with the .cache folder) or there’s some invalidation happening that I don’t understand. Maybe that’s worth another bug report if it happens again …
Here’s a short screencast showcasing the current state of development.
The most noteable though not obvious change is the ability to handle git annex actions over a set of images spread over multiple distinct repositories.
This might become handy for folks that don’t necessarily manage their collection within one single repository.
Feel free to check out the metadata branch to test the metadata feature in it’s current state.
This short screencast show’s it in action.
The idea is, that the user can define it’s own set of rules (like rating = 4 stars , tag attached = Venice) and store these in settings.
Then, for each rule user can set the desired metadata.
For example:
set annex.numcopies = 2 to all 4 star images
set annex.cumpopies = 3 to all images tagged with ‘Venice’
I’d like to invite you to test out a few things i just added. As usual, any feedback is highly appreciated.
The main functional changes up to this point are:
git annex actions(add/get/drop): added “include sidecars” toogle to include all sidecar/xmp files. It is adviseable to check in xmp files into git rather than git-annex. Git annex will do this automatically if you configure annex.largefiles accordingly:
Metadata: added author + tag to the list of git annex metadata fields
Metadata: added copy + move options: when selected, no metadata will be written. Instead git annex move/copy /path/to/file --from-anywhere --to=remote will be executed. Note, that ‘remote’ is not the path of the repository but the the “name” of the remote repository as set in
git remote add 'name' <URL>
added preference “Pre/Post Import Hook add” as Opt-In: enable it, to automatically add files to annex on import. Note, that this only works, if you choose “add to library” from import module but not by “copy & import” - which is a pitty, but i couldn’t figure out, how to solve it (yet)
added experimental feature Sidecar History as Opt-In: enable it in preferences and restart dt.
Afterwards, you should be able to go back and forth in history of your git-tracked sidecar files.
There is a drawback though: You’ll have to manually reload the sidecar file from history stack module - yet. Here is a short screencast showcasing the feature in action