How to create camera noise profiles for darktable

When I run the denoise profile generator tool with darktable 4.0 I get the same terminal output as @FalCT60

===> Check for required tools
--> Check for images handling tools availability
--> Check for images export tools availability
--> Check for profiling tools availability
--> Check for tethering tools availability
--> Check for pdf tools availability

===> Check profiling directory

===> List profiling input RAW images
--> Found ISO 100 image: /home/user/Pictures/IMG_0568.CR3
--> Found ISO 200 image: /home/user/Pictures/IMG_0569.CR3
--> Found ISO 400 image: /home/user/Pictures/IMG_0570.CR3
--> Found ISO 800 image: /home/user/Pictures/IMG_0571.CR3
--> Found ISO 1600 image: /home/user/Pictures/IMG_0572.CR3
--> Found ISO 3200 image: /home/user/Pictures/IMG_0573.CR3
--> Found ISO 6400 image: /home/user/Pictures/IMG_0574.CR3
--> Found ISO 12800 image: /home/user/Pictures/IMG_0575.CR3
--> Found ISO 25600 image: /home/user/Pictures/IMG_0576.CR3

===> Checking profiling RAW images correctness + Jpeg export
--> ISO 100:
    /home/user/Pictures/IMG_0568.CR3
usage: darktable-cli [options] [IMG_1234.{RAW,..}|image_folder/]

options:

  --cachedir <user cache directory>
  --conf <key>=<value>
  --configdir <user config directory>
  -d {all,cache,camctl,camsupport,control,dev,fswatch,imageio,input,
      ioporder,lighttable,lua,masks,memory,nan,opencl,params,perf,demosaic
      pwstorage,print,signal,sql,undo,act_on,tiling,verbose}
  --d-signal <signal> 
  --d-signal-act <all,raise,connect,disconnect,print-trace>
  --datadir <data directory>
  --disable-opencl
  -h, --help
  --library <library file>
  --localedir <locale directory>
  --luacmd <lua command>
  --moduledir <module directory>
  --noiseprofiles <noiseprofiles json file>
  -t <num openmp threads>
  --tmpdir <tmp directory>
  --version

I’ve run the tool in Fedora, Manjaro, and Debian-testing and get the same result everytime. Does anyone have any suggestions on how to resolve this? Is it possible the denoise tool is feeding commands into darktable-cli with incorrect syntax?

The raw data should be the same if I convert them from CR3 to DNG and then create a denoise profile from them. I don’t have CR3 support in my VirtualBox darktable version. If you want you can send me the files and I can try generate a profile. You can try the profile first before sending it in to Github.

Thanks Peter, I will take you up on that offer, but want to reshoot the pictures first to make sure they’re clipping on all three channels. I will send a link tonight.

Are you sure your dt supports CR3, i.e. is built against exiv2 0.27.4 or later w/ BMFF support enabled?

I’m fairly certain it does. Darktable opens all of my CR3 files without issue and displays correct exif data in the “image information” module. To rule the file format out I also tried converting the CR3’s to DNG files first before running the tool, but the terminal output is the same.

@Peter do you mind sharing what your setup is that has the noise profiling tool working for you?

-Which specific linux distro?
-Darktable version number?
-Did you build darktable from source or did you install through distro package manager?

Any details appreciated.

I built it last year I think on Ubuntu. Think the version is 3.6. Many trials and errors, as you also can see in this post, and that’s why I have it in a VirtualBox. I can check the Terminal history to see if there are any clues.

On this line, does moving --apply-custom-presets false before --core perhaps solve the issue? And the same here.

1 Like

That fixed the tool for me in Debian-testing using darktable 3.8.1, thanks!

Do you still need my help then?

No, but thanks for offering though!

1 Like

I’m trying to get a noise profile for my Nikon Z 6ii as darktable doesn’t seem to one for it yet (only the Z 6) and while I can get the noise profiler script working, I’ve noticed that some images have the wrong ISO in the output of the script.

Specifically with any images that use the Lo or Hi ISO settings of the camera.

Lo 1.0, Lo 0.7, and Lo 0.3 should have ISO values of 50, 63, and 79 respectively (verified in darktable itself and by running exiv2 manually and seeing what it outputs), but the output pdfs and presents.json from the scipt list my images at these settings with ISO values of 48, 52, and 56 instead.

Similarily, Hi 0.3, Hi 0.7, Hi 1.0, and Hi 2.0 should have ISO values of 64508, 81275, 102400, and 204800 respecively, but the json file and PDFs list them at ISO values 172, 176, 180, and 192 instead.

Any suggestions as to what may be causing this? Aside from those images everything else seems to compute and output correctly…

Some old issue at Github https://github.com/darktable-org/darktable/pull/1612

Correct me if I am wrong, but It seems those extended ISO nr below ISO 100 just are the same as ISO 100. If that is the case, then the NR will be able to interpolate from ISO 100 anyway meaning you don’t need lower than ISO 100 in json. But it still seems like a bug to report.

Most likely not, I can’t even see any noise at ISO 50 that warrants a denoise pass anyways, I’m just trying to get as full of a dataset as possible to send for inclusion into darktable. Could probably let interpolation handle the Lo extensions if I needed to…

The Hi extended ISOs would definitely be an issue in providing said dataset, as they are being detected as very incorrect values than what they should be…

@rawfiner, any clues?

At minimum, I’ll probably fire off a bug report for this too like how you mentioned, so that it can be fixed in a future release.

I’m sure there will be at least one other person that runs into this eventually :sweat_smile:.

ETA: https://github.com/darktable-org/darktable/issues/12264

Unfortunately the script sometimes badly reads the ISO values. It is maybe a similar issue as this one: https://github.com/darktable-org/darktable/issues/8992
The good news is that the ISO values can be manually corrected in the noiseprofile json file that the script outputs in order to match the values showed by darktable (in darkroom or lighttables). That way, darktable will correctly detect the noise profile.

Ya, I’ve done that for my local copy of darktable and it works fine there, but I want to be able to submit the profile for inclusion too and I’m not sure if having to modify the json file to correct these ISOs would be accepted for that since it would no longer match up with the generated pdfs…

Yes it will be accepted even if some ISO values have been corrected manually, you can submit it. All that matters is that the curves looks good, and that darktable correctly sets the profile in denoise profile module.

Ah ok. I’ll submit mine later today with the hopes if it being accepted in that case, just with a note stating the ISO changes to try and prevent confusion.