AI Masking on Darktable

I’ve been working on adding the FAST-SAM model to Darktable. It’s currently being executed in the CPU but once the mask of an image is created in theory there should not be need to be executed again.

My fork is the following: GitHub - MikoMikarro/darktable: darktable is an open source photography workflow application and raw developer

It is unstable as the system doesn’t allow me to continue editing afterward. It seems some kind of thread gets blocked.

This is more of a proof of concept and a small call to the developers to reach out if they want to follow this feature. I’m slightly lost in the documentation and reverse-engineering of the code, some insight on how to correctly approach this feature would be awesome.

If this is executed correctly it would open the tools for other AI models to be imported in DarkTable. I’ve seen the comments on the release of DarkTable 5.0 Many people are missing this kind of AI tool and are forced to move out to LightRoom, Adobe, Capture One, etc…

I’m open ears!

You can see the small demo I prepared on the original reddit post: https://www.reddit.com/r/DarkTable/comments/1icp9az/ai_masking_in_darktable/

20 Likes

13 posts were split to a new topic: salty hn post (split from Ai making post)

Cool, I’m definitely going to take a look!

2 Likes

This is a great feature, hope it gets more people on board to implement it.

Not everyone has the same pointer dexterity and for some users being able to mask something with a single click is a godsend, even if the mask will need slight adjustments afterward.

Also, at the end of the day darktable will be put in crude feature for feature comparisons with other developers, and not being “behind” is a big bonus for it. I know it’s a drop in the bucket when it comes to a raw developer’s features, but many people will go for first impressions.

3 Likes

Definitely something I am interested. Darktable should not be left behind compared to other AI supported apps.

Hi guys! New post updating some of the progress being done with the AI Masking!

https://www.reddit.com/r/DarkTable/comments/1idlm0h/ai_masking_update_ii/

I’m trying to keep the dark table coding philosophy of keeping the code executable on “budget” or old PCs. The inference is currently being executed in CPU but I’ll try to get GPU execution added so it would take much less than a second to generate the masks.

I still can’t execute the pipeline completely and doesn’t work when exporting the image. If someone who is more versed on the Darktable codebase could give some insight, it would be awesome.

Regards.

8 Likes

I think the thing that’ll make this come to darktable is making a generic module that can shell out image, mask, and whatever other data needs to get output to make this happen,and then receive back the altered image stream, or in this case, mask.

I know AP had something like this going for his Python deconvolution testing years ago.

While all the AI.might be useful, it’ll be a hard sell to get python as a dependency in DT. And as someone who packages DT and other things, Python sucks to package… At least its not node.js I guess.

Yes, this makes complete sense. Maybe the real point of these guys for manufacturers is to normalise the idea of having the latest gear all the time.

1 Like

I hope the OP gets some help from other developers to complete this project.

1 Like

I looked at your github fork and so far could only spot a single commit for your interface ??? I would for sure help out where i can but if you want a chance for us devs to check, try ourselves, review your code or discuss how things would have to be done to be integrated into darktable, darktable github would be the place.

If you have code to test you should open a pr or if you need help to find out how things can be done “in real code” you could also open an issue with request for help. See you there :slight_smile:

Some impressive work being done here, congrats.
You’ve probably worked it out by now but Reddit might not be the first place you want to go for help and to post updates. The crowd on that subreddit are quite an impatient bunch and will probably just assume the feature is all ready to go! Github is where you want to go to work with the main developers. And here is a good place for feedback from long-time users who are committed to the project.

I never really expected AI to come to Darktable, and I’m fine with that. But I’ll always cheer new features on, so good luck with your efforts!

3 Likes

Thank you for your work @MikoMikarro! This would be extremely valuable for professionals that work with large volumes of image where editing time per image adds up.

Hi! Just wanted to update the videos there and get the hype a little bit up. This is more a proof of concept and all the diffusion I can get to find a lovely developer that wants to help is always welcome.

I also revived the “AI Masking” Issue on Github and keep updating the demos there. Let’s see who hops on the train!

Hi! For now I have 9 commits that highlight some of the milestones I perceive as I continue the development.

I don’t know what you mean by the interface. If you mean the GUI, that’s a “simple” part as I’ve only added a new cursor to indicate what mask are you interested in. I still need to handle a lot of interactions and have a better mask creation procedure as I’m currently generating it too far down the pipeline. I lack knowledge of the codebase and I’m self-reverse engineering it to understand even where my code should be or what “built-in” functions that dt developers use to properly integrate the system.

If you mean the interface as the software that allows to execute the ML model, it is the ONNX runtime. I just created a ONNX model of the FAST SAM model and translated their python code to C so it could be executed without adding the python dependency to DT. It is not easily replicable as I literally modified the CMakeLists.txt by hand but I hope to integrate it on the build script correctly as the code is more propely integrated in the codebase.

Hope it gives some light on what you see on the fork! All help is welcome, we can keep the discussion on the “AI Masking” issue on github : AI Masks · Issue #12295 · darktable-org/darktable · GitHub.

2 Likes

I split all the salty hacker news stuff to a separate thread so this can get its own attention.

8 Likes

Well, you asked a darktable dev to “step in” and here i am doing so :slight_smile:

As said i had a look at your fork, may be i was blind but it seemed to be, that all “relevant stuff for your code” is in master branch, so not good to be spotted or tested by us devs.

Please keep master “as is” and put all your stuff into another branch and do a pull request for that branch so we can discuss and try on our machines . That’s “how the darktable cookie crumbles”, basically most other larger projects work this way.

BTW, i fully appreciate your results and would love to have progress integrating into darktable code base.

I personally don’t read reddit at all, did so some years ago but it seemed to be just a waste of time.

17 Likes

Thanks for jumping in! I hope I don’t need to take much of your time with this.

I’ve created the new branch and the pull request. Some of the code is not necessarily doing anything right now as I first created a new section for the AI masks but then I understood that being in the brush section made more sense.

To recreate it you will need the fast sam onnx model, it can be easily created using this repo: GitHub - ChuRuaNh0/FastSam_Awsome_TensorRT.

Also I downloaded the 1.20.1 release version of onnx Release ONNX Runtime v1.20.1 · microsoft/onnxruntime · GitHub and modified “by hand” the API version on the included submodule (probably I should check out the submodule to the correct version instead of being hacky with it)

I hope you can reconstruct and try it by yourself without mayor problems.

2 Likes

Hi! Created the pull request: Object mask by MikoMikarro · Pull Request #18331 · darktable-org/darktable · GitHub

I changed the commit of the onnxruntime submodule to be the correct version and deleted some of the code that was from an old attempt of creating the GUI. It is still necessary that you get your onnx model from the FASTSAM Google Colab.

Finally, it is also necessary to download the ONNX RUNTIME from the releases: Releases · microsoft/onnxruntime · GitHub (I use v 1.20.1)

I also revived the AI Mask topic on GitHub: AI Masks · Issue #12295 · darktable-org/darktable · GitHub

1 Like

I think you should add some content to that PR. At least describe the goal and dependencies for this to work.

3 Likes

Created new “cleaner” pull request: Object mask clean by MikoMikarro · Pull Request #18356 · darktable-org/darktable · GitHub

1 Like