[feedback needed] integrating nind-denoise with darktable

Software darktable
205

It is impressively good at spitting out code that looks good, but is actually so awful… It got me when I was too tired to tell the difference.

You have my sincerest apologies, and as pennance I did actually optimize it 90% of the way, but I got stuck on x-trans and won’t finish until tomorrow. Probably too late for you.

Some gory tech details: I replaced the neighborhood search with independent channel FFT operating directly on the RAWs, partly cause it should be faster but mostly to avoid an expensive demosaic. This was hard. I have a newfound appreciation for the elegance of the original design. Eventually, I did manage to get it to meet or exceed the original accuracy, and its ~5-6x faster. Getting it to work in X-Trans kept on giving me 2-4px residuals; no matter what crazy stuff I tried (I actually implemented something akin to this which I think is cool and useful on its own, but only reduced residuals to 1px. Whatever. Stupid Fujicams. I’ll figure it out tomorrow maybe.

I could batch on gpu as well, but I’m not sure its worth the effort.

I did make a prototype index/memcache though, which might prevent the do-over issue. If you haven’t finished downloading by tomorrow I can try to get it usable quickly.

4 Likes
206

i was distracted with other things and didn’t attempt to restart yet. a restartable script that recovers quickly from interrupts would be very nice.

1 Like
207

just FYI, the bug on importing TIFF has been fixed in master branch

1 Like
208

It (*the rawnind download-preprocess script) is in mostly working draft form now. Just needs one or two more connections near the end and some bugsquashing.

I got a bit carried away, but I’ve always wanted to build this architecture soooo I may have gotten carried away.

Edit: It will depoend on your uplink speed, but it runs in about ~2-3 hours for me. And is rather fault-tolerant.

*edit2 for clarity

2 Likes
209

Pre-release release of a functional Lua Plugin is live and needs testing.

4 Likes
210

Will this be available in next DT version (5.4)?

211

yes

1 Like
212

Failed to load script:

     6.0917 LUA ERROR: dtutils.lua: prequire: 223: Error loading tools/nind_denoise_rl 
     6.0918 LUA ERROR: dtutils.lua: prequire: 224: Error returned is bad argument #3 to '?' (number expected, got nil) 
     6.0918 LUA ERROR: luarc: activate: 541: error loading tools/nind_denoise_rl 
     6.0918 LUA ERROR: luarc: activate: 542: error message: bad argument #3 to '?' (number expected, got nil)

:thinking:

213 
83c83
<   if success then
---
>   if success and value ~= nil and value ~= "" then

These lines change in nind_denoise_rl.lua will help in fixing the loading error

214

Though in macos I am getting tiff errors

     2.8574 [tiff_open] error: TIFFFillStrip: Read error on strip 0; got 18446744072894649248 bytes, expected 31516
     2.9009 [rawspeed] Canon7dii_s1_denoised.tiff corrupt: rawspeed::RawImage rawspeed::RawDecoder::decodeRaw(), line 334: const TiffIFD *rawspeed::TiffIFD::getIFDWithTag(TiffTag, uint32_t) const, line 242: failed to find 1 ifd with tag 0x828e
Magick: Read error at scanline 4294967295; got 0 bytes, expected 31516. (TIFFFillStrip).
     2.9027 [GraphicsMagick_open] ReadImage failed for 'Canon7dii_s1_denoised.tiff'
     2.9067 [imageio_storage_disk] could not export to file: `Canon7dii.jpg'!
215

Thank you.
But now there’s another error (linux).

  1258.9625 LUA new_name: /media/a/photos/foto/0out/20240919__D8C4477.jpg
  1258.9625 LUA RL deblur enabled: true
  1258.9658 LUA Denoise command: source "/home/a/nind_denoise/.venv/bin/activate" && python3 "/home/a/nind_denoise/src/denoise.py" --tiff-input -o '/media/a/photos/foto/0out/20240919__D8C4477.jpg' --sidecar '/media/a/photos/foto/2024/09/2024_09_19/20240919__D8C4477.NEF.xmp' --extension jpg --quality 97 --sigma=1 --iterations=20 '/tmp/20240919__D8C4477.tif'
sh: 1: source: not found
  1258.9664 LUA ERROR: [NDERR-1001] Denoise/deblur processing failed
216

I had to change the activate command from
activate_cmd = "source \"" .. extra.denoise_dir .. "/.venv/bin/activate\" && "
to
activate_cmd = ". \"" .. extra.denoise_dir .. "/.venv/bin/activate\" && "
to get the export starting.

217

Please refer @dirksagwitz reply below.
‘source’ is not available on all shells.
You can use ‘.’ instead of ‘source’

219

Are you trying to process an image right out of SD card?
May be the SD card doesn’t allow writes.
Try copying image to a different folder in your hard disk (/home/a/photos)

221

No, all paths are on HDD. And it work fine in the CLI. But without --tiff-input.

222

I have the same issue … the s1 tiff is generated, the denoised tiff and the jpg are not.
Nevertheless calling the denoise script from the command line is working for me, too.

223

I reloaded nind_denoise_rl.lua and changed only lines 83
if success and value ~= nil and value ~= "" then
and 855
activate_cmd = ". \"" .. extra.denoise_dir .. "/.venv/bin/activate\" && "

It actually works.

1 Like
224

And now I see a new problem: cropped images. They are exported to TIF already cropped (and scaled(!) if you specify a scale), and then cropped again after denoising.

225

Another problem: TIF export takes into account the entire editing history, whereas in the console version without the --tiff-input option, the list of modules is severely limited.

226

Tags from darktable are lost.