[feedback needed] integrating nind-denoise with darktable

Software darktable
1

On FOSS side, I think nind-denoise (created by Benoit Brummer) is the only practical AI denoise available that I tested. I’ve been using it personally and professionally, totally happy with its performance, ISO up to 10,000 can look clean as ISO200 (even with artifacts, reduction of false colors from noise is a huge plus). Search for dt-nind-denoise to see samples.

It works on RAW images that have not been denoised, and it’s slow (about 9 seconds on RTX3060, or 63 seconds on Ryzen 5 7640HS, for a 24MP image). Thus, it should not be part of darktable interactive darkroom, but applied only at export.


Injecting nind-denoise into darktable export process
In DT, an image’s history stack is saved to the XMP file. darktable-cli can process the image with a specified XMP. The idea is to copy the original XMP into two XMPs:

  • 1st-stage has only RAW-related operations (demosaic, hotpixel, highlight reconstruction …)
  • 2nd-stage has the remaining operations of the history stack

We apply nind-denoise in between the two stages.

The flow would be:
(darktable-cli) apply 1st-stage XMP and export to TIFF32
==> (nind-denoise) output to a denoised TIFF32
==> (darktable-cli) apply 2nd-stage XMP and export to JPG (optionally apply RL-deblur)

I’ve been working on a Python script to streamline the process. It’s still a WIP, tailored to my needs, but I think it’s stable enough. Here’s an attempt to document the tool, hopefully to get some feedback/testing on making the tool more accessible. I’ll keep this top post updated.


Installing nind-denoise
nind-denoise is the main component, so you’ll need to get that working first. Try to follow the instructions on the repo, basically:

  • install dependencies (using pip3, package manager, …)

  • clone and checkout my darktable-cli branch to run locally.
    NOTE: check out the branch darktable-cli directly to avoid problem with models’ folder name having invalid char (for Windows)
    git clone https://github.com/hqhoang/nind-denoise --branch darktable-cli

  • download the available models ( I found this one to be the best).
    If you modify the folder name of the model, note whether it has unet or utnet , and specify --network UNet or --network UtNet later when using the command.

  • optional: install CUDA for faster denoise on nVidia GPUs (look up tutorials/guides for your distro/OS)

Once you get nind-denoise working, you can test it manually by feeding it a JPG exported from darktable (without any operation that affect the noise patterns such as profiled denoise, sharpen, lens correction, …).

Example command on mine, with DSC0123.jpg being the JPG exported from RAW in DT, nind-denoise will output to DSC0123_denoised.jpg (testing manually with JPG to avoid complication with TIFF).

/home/noname/python_envs/nind_denoise/bin/python3 /home/noname/tinker/git/nind-denoise/src/nind_denoise/denoise_image.py --network UtNet --model_path "/home/noname/tinker/git/nind-denoise/models/2021-06-14/generator_650.pt" --input "DSC0123.jpg" --output "DSC0123_denoised.jpg"


Streamlining the workflow

I wrote a Python script for the above workflow.

In the src/darktable folder of the checked out github repo:

  • copy the file dt_nind_denoise.example.ini into dt_nind_denoise.ini
  • update the dt_nind_denoise.ini file according to your local paths (using the same paths that have been tested manually above).
  • run dt_nind_denoise.py on your RAW images. You can put those two files (together) anywhere, they don’t have to stay in that folder.

You might need to install a few required packages for the script (e.g. pip3 install exiv2 beautifulsoup4 lxml)

Example usage: dt_nind_denoise.py DSCF0123.RAF DSC0234.NEF
or, for all RAW files in a folder: dt_nind_denoise.py *
or, for only images rated 3- or 4-star: dt_nind_denoise.py -r34 *
or, use bash range: dt_nind_denoise.py DSC0{121..129}.*

(run dt_nind_denoise.py -h to list all parameters)

When run in debug mode (with -d flag), the script will print more details, and keep the intermediate TIFF32 files for inspection.

My workflow: edit RAW files normally in darktable, but don’t apply denoise (and no sharpening, since I use RL-deblur instead). Once I’m done editing, instead of exporting from darktable, I’d drop to the terminal, and run the dt_nind_denoise.py script on the whole album/folder.

IMPORTANT: the dt_nind_denoise.py script doesn’t handle multiple instances of a module (e.g. if you adjust exposure mutltiple times in between other operations). To avoid unpredictable behavior, always compress history in darktable before running the script.


GMIC RL-deblur

Richardson-Lucy deconvolution is a reconstruction process (unlike sharpening), really great to make the most of your lenses, bringing back details without halo. Its only downside (beside being slow): noise is also accentuated. But that’s why nind-denoise is its perfect companion.

The script optionally uses GMIC’s RL-deblur on the exported JPG (I don’t use sharpening in DT). If you don’t have gmic installed, just comment it out in the dt_nind_denoise.ini file.


Other notes

Demosaic makes huge difference to nind-denoise, as each algorithms alter the noise pattern slightly. I’m not sure which demosaic algo was used for the training data, but I found RCD is best for Bayer, and Markesteijn 3-pass is best for X-Trans. color smoothing and match greens should be disabled.

I’m not familiar with C++ and darktable’s code structure, only use Python once in a while, so this is just a really dirty hack, while waiting for a proper solution from the darktable team. All feedback and help are very welcomed and appreciated.

15 Likes
2

Wow, I’ve seen some results now and I’m really impressed!
Would love to try and install. So will try to do so.
Maybe a problem maybe not is that I’ve failed so far to get OpenCL running. Can’t tell.

Other weakness - I’m neather tech savvy, nor a Linux greatness…

Kind regards, Jetze

3

Thanks for sharing this…I am just wondering at the beginning you look like you are using the denoise on TIFF and then you switch to talking about applying it to Jpg…so I am likely missing something and I will go back and take a more careful read but can you clarify why you switched to talking about the jpg??

1 Like
4

I’d use JPG to test nind-denoise manually, it avoid complications with TIFF, especially TIFF32 requires advanced software like GIMP to view correctly. But when integrating with darktable, it needs to be TIFF32 in order to retain the data being passed between DT and nind-denoise.

(If you look at the commits on the branch, you can see that I had to try twice to get TIFF32 out of nind-denoise correctly :smile: )

If you run the script with -d flag (debug mode), it’ll keep all the intermediate TIFF32 and XMP files for you to inspect.

Thanks for the good question, I’ve updated the top post with clarification.

5

Thanks I thought this is maybe where you were going with that but I wasn’t sure…thanks for taking the time to set me strait….

1 Like
6

Hi, I tried to clone your repo, but failed with this log:

PS C:\Users\---\Documents\Programy\Darktable_NIND> git clone https://github.com/hqhoang/nind-denoise.git
Cloning into 'nind-denoise'...
remote: Enumerating objects: 308, done.
remote: Counting objects: 100% (95/95), done.
remote: Compressing objects: 100% (55/55), done.
remote: Total 308 (delta 34), reused 84 (delta 33), pack-reused 213 (from 1)
Receiving objects: 100% (308/308), 108.11 MiB | 30.73 MiB/s, done.
Resolving deltas: 100% (122/122), done.
error: invalid path 'models/nind_denoise/2019-02-18T20:10_run_nn.py_--time_limit_259200_--batch_size_94_--test_reserve_ursulines-red_stefantiek_ursulines-building_MuseeL-Bobo_CourtineDeVillersDebris_MuseeL-Bobo-C500D_--skip_sizecheck_--lr_3e-4/model_257.pth'
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

PS C:\Users\gardi\Documents\Programy\Darktable_NIND> git status
fatal: not a git repository (or any of the parent directories): .git

I don’t have much experience cloning git repos, so I have no idea if I’ve made just some silly error… if so, please tell me :sweat_smile:

7

My best guess would be something to do with that super-long folder name of the models. I cherry-picked a commit from a different branch that shortened the long folder paths, perhaps that person was having the same problem.

Try deleting the cloned folder (Darktable_NIND), and try again with this command (that clones and checks out the branch with shortened folder paths).

git clone https://github.com/hqhoang/nind-denoise --branch darktable-cli

If that still doesn’t work, try cloning into a folder at the root of C: (e.g. C:\DT_NIND), so that we don’t have a long folder path being added to that long model name to hit the OS’ limit.

Let me know if that works, I’ll update the documentation accordingly.

8

Both alternatives fail with the following message related to the same path:

error: invalid path 'models/nind_denoise/2019-02-18T20:10_unet_original/model_257.pth'
9

Ah, could be that colon in the folder name?

We won’t be using those models anyway. You can download the better model separately, and can put the models anywhere, so I deleted the model folders altogether from the branch.

Please delete the cloned repo and try again.

git clone https://github.com/hqhoang/nind-denoise --branch darktable-cli

1 Like
10

Alright, success! Now I’m a bit confused what to do next, since I don’t have a python_env folder as mentioned here:

What can I try to test what I’ve got so far?

11

If you didn’t install the dependency packages into a specific Python environment, then you can just try running the “global” python (or python3), the same way you would run Python interactively or a Python script (sorry, not familiar with Windows).

Assume you cloned the repo into:
C:\Users\---\Documents\Programy\Darktable_NIND

Assume you downloaded the model and put it as:
C:\Users\---\Documents\Programy\Darktable_NIND\models\generator_650.pt

Assume that you exported a RAW file as JPG and put it at:
C:\mytest.jpg

Then the command could be:
python C:\Users\---\Documents\Programy\Darktable_NIND\src\nind_denoise\denoise_image.py --network UtNet --model_path "C:\Users\---\Documents\Programy\Darktable_NIND\models\generator_650.pt" --input "C:\mytest.jpg" --output "C:\mytest_denoised.jpg"

12

Isn’t this what the feature “apply style on export” is for?

Thus this workaround would not be needed, once it is a normal module, right?

2 Likes
13

Is there an option to add nind-denoise to a style? I haven’t followed DT development lately.

That feature works with profiled denoise, for example, as profiled denoise execution order has been determined by darktable to be before and after certain operations (e.g. after demosaic, but before lens correction). IIRC, in current version, darktable displays the modules in the same order they will be executed.

Similarly, nind-denoise has to be injected at a specific spot on the execution order, but we can’t control that yet.

Thus, the 1st-stage has only operations that don’t change the noise patterns (demosaic, hotpixels, …). All other operations that changes the noise pattern such as retouch, lens correction, sharpen, rotate/perspective, haze removal, … have to be applied after nind-denoise (basically nind-denoise needs to be at the exact spot where profiled denoise is)

Yes! If nind-denoise is a module in darktable, it can easily be executed in a specific order during the export process. However, due to its time-intensive nature, it should not be executed during darkroom development (still clunky even with caching the denoised image).

Until then, I only have this work-around, which could be considered as a proof-of–concept to test out the details for that “nind-denoise module” :smile:

I think a broader/more generic concept is being worked on:

1 Like
14 
C:\Users\-\AppData\Local\Microsoft\WindowsApps\python.exe: can't open file 'C:\\Users\\-\\Documents\\Programy\\Darktable_NIND\\src\\nind_denoise\\denoise_image.py': [Errno 2] No such file or directory

I think we’ve got a problem with slashes

15

I think that it would be even better if it was possible to denoise the image before black/white point. One problem then will be to deal with each camera model and the order of the Bayer pattern.

16

for the purpose of just testing it out, it’s probably easiest to just move the Darktable_NIND folder out to the root of C:\ . That way, your path will be simpler with no spaces to complicate it. Also make sure that there’s only one backslash in between the folders on the path (e.g. \\src\\nind_denoise\\denoise_image.py should be \src\nind_denoise\denoise_image.py instead)

After moving the folder to the top level C:\, the command would be:

python C:\Darktable_NIND\src\nind_denoise\denoise_image.py --network UtNet --model_path "C:\Darktable_NIND\models\generator_650.pt" --input "C:\mytest.jpg" --output "C:\mytest_denoised.jpg"

17

Still the same problem, this is what I run:

PS C:\Darktable_NIND> python C:\Darktable_NIND\src\nind_denoise\denoise_image.py --network UtNet --model_path "C:\Darktable_NIND\models\generator_650.pt" --input "DSC_2898.jpg" --output "DSC_2898_nind.jpg"                                                               
C:\Users\gardi\AppData\Local\Microsoft\WindowsApps\python.exe: can't open file 'C:\\Darktable_NIND\\src\\nind_denoise\\denoise_image.py': [Errno 2] No such file or directory
18

From what I tested, nind-denoise can denoise well for very underexposed RAW (I cap AutoISO at 800 for my X-T2 and often push +4EV or even +5EV in darktable). Thus, I keep the “exposure” module on the 2nd-stage by default.

The NIND dataset was taken with an X-Trans X-T1, but the train data are demosaiced, so it’s more about the selected demosaic algorithm. As mentioned in the top post, I found RCD best for Bayer, and Markesteijn 3-pass best for X-Trans (are we talking about different CFA or something else?)

19

Sorry, I forgot to quote around the first path, it should be:

python "C:\Darktable_NIND\src\nind_denoise\denoise_image.py" --network UtNet --model_path "C:\Darktable_NIND\models\generator_650.pt" --input "C:\mytest.jpg" --output "C:\mytest_denoised.jpg"

(dang, now you remind me how annoying Windows is after I left it 15 years ago :frowning: )

20

Yep, that’s exactly THE reason I’d like to move to Linux, just have to figure out which distro…

The annoying thing is – quotes make no difference :confused: