How to create camera noise profiles for darktable

@ikfotografeer hello & welcome!

  1. What version of ubuntu? Where did you install the darktable package? PPA, snap, something else?

  2. Does the file open correctly in darktable? Can you see the image without issue?

  3. This is a problem with a lot of them, as we just dump the camera name from exit data, so it may not match what exiv2 tells us.

Linux Mint version 19 based on Ubuntu 18 LTS.
I used the Ubunbuhandbook1/darktable ppa. But to make sure I added the Opensuse repo and disabled the Ubuntuhandbook1 ppa.
It makes no difference for the package-issue, look: This is the output from sudo apt-get install darktable-tools-noise (when imagemagick is installed):

De volgende extra pakketten zullen geĆÆnstalleerd worden: <- This will be installed
  graphicsmagick-imagemagick-compat
De volgende pakketten zullen VERWIJDERD worden:  <- these packages will be REMOVED
  imagemagick imagemagick-6.q16
De volgende NIEUWE pakketten zullen geĆÆnstalleerd worden: <- These will be installed
  darktable-tools-noise graphicsmagick-imagemagick-compat

So, as it removes imagemagick and installs imagemagic-compat, the check on the package Imagemagic fails.
Preferred solution: Use the latest imagemagic.
Quick&working solution: Fix the script to accept both imagemagic AND imagemagic-compat as prerequisits.

Now I know that the camera model is incorrect I correct it manually in the json file.

The software runs and creates noise-profiles, though it does not work out-of-the-box. The errors do not impact the actual creation of noise profiles. However many people may just give up.
I will send results shortly, I would like to know if my noise-images are good enough.

You could tell apt not to install dependencies.

Yes, I could. And we could argue about whether that makes my own system less stable.
Right now I worked around the problem by disabling the check for imagemagic. The noise-generator is a bunch of clever scripts which are flexible enough to run from a local directory.

Frankly it does not matter how I make it run on my laptop.
The script checks for package A to be installed but the package containing the script will remove package A. The solution is not any local fix, but removal of the root cause.

My appoligies for trying to be helpful.

My apologies for being blunt.
I have it functional enough to generate noise profiles.
But fixing it on my laptop does not help others. I know I am put off if I follow a websites instructions and they donā€™t work - others might simply stop.

I hava a question:
When I create a noise profile using two images with the same ISO, the system generates two entries for this ISO.

      "profiles": [
        {"name": "E-M5 Mark III iso 200", "iso": 200, "a": [1.06932705674516e-05, 5.37294730243459e-06, 9.78057733430079e-06], "b": [-3.21350856218692e-08, -2.8041084638846e-08, -3.03659946457895e-09]},
        {"name": "E-M5 Mark III iso 200", "iso": 200, "a": [1.07871939587923e-05, 4.32148444789824e-06, 7.51797846844814e-06], "b": [9.72228373322003e-09, 2.61078613451342e-09, 4.26048385102063e-09]}
      ]

Each set of values is based on one image.
My question is: Would it improve the noise profile if I were to average these values?

Possibly, given that the current entry for the Canon PowerShot G10 says:

duplicate measurements kept for statistical validation (maybe we should average them at some point)

It seems to make sense. But look closer at my sample values. The first 3 are fairly close together. Averaging might make sense.
But the other three are more different. The all have positive and negative reversed. This might be a coincidence. I do not know. I do not know the meaning of them well enough to say that averaging is good.

This may seem like a silly question. Do the noise profiles vary from camera to camera of the same brand and same model? Meaning if Iā€™m using a refurbished Nikon D7200, can it have a different noise profile than the brand new Nikon D7200?

Thank you,

Technically speaking, probably yes. But practically speaking, no. :smiley:

Darktable profiled noise reduction is per camera model.

Thanks for your kind offer. I have created profile shots for the Samsung NX1100 and Samsung NX3000. I tried to process them myself, but failed after seriously trying (distribution Mint 20.1). I have 7zipped and CC0ā€™ed them here: Filebin | gyri7ie9taayarph. If there is anything I can do, let me know.

My link posted above is listed as empty when I visit it, though the week for filebin storage is not over. Therefore, I have stored the files again at Filebin | j6g9z6gic4782zr4

Hey sorry, been too busy to process these, Iā€™ll try and get to it soon

Dear Mica,

no problem, actually I feel kind of guilty for not being able to process them myself.
Let me just say that I am impressed with all your contributions, and Darktable rocks!

Both cameras are underestimated and deserve a decent noise profile.
You will make me happy if the profiles should really appear in 3.4 :wink:

Have fun,
Volker

@Colourimeter are all the zips files one set of noise shots each?

Dear Mica,

yes. You see 2 files, one for each camera, that I segmented into 3 parts each to reduce upload errors with filebin. You will have to use an Archive-Program that can read segmented 7zip files - the default for an ubuntu distribution should handle that automatically. Un7zipped, you will find individual shots for each ISO, called ISO100.SRW, ISO125.SRW, ISO160.SRW ā€¦ and so on. They are accompanied by sidecar files that hold the CC0 license notice (generated with Darktable).

If you prefer another format, just let me know. For me, anything seems easier than processing the noise profiles (there is a clash between my ImageMagick and the profile generation package, and I am not enough of a Linux man to forcefully restore peace between these two. I had a few ideas, but failed in the end).

Have a nice weekend, Volker

Thanks a lot @asn for your great how-to!
I used it to get a denoising profile for my drone (DJI Mavic 2) on Ubuntu 20.04. I succeeded, but wanted to share the bumpy ride:

  1. If you want to install the package darktable-tools-noise on Ubuntu 20.04 you are confronted with the following dependency issue:
The following packages have unmet dependencies:
 graphicsmagick-imagemagick-compat : Conflicts: imagemagick but 8:6.9.10.23+dfsg-2.1ubuntu11.1 is installed
                                     Conflicts: imagemagick:i386 but it is not going to be installed
The following actions will resolve these dependencies:

     Remove the following packages:                                                           
1)     imagemagick [8:6.9.10.23+dfsg-2.1ubuntu11.1 (focal-security, focal-updates, now)]      
2)     imagemagick-6.q16 [8:6.9.10.23+dfsg-2.1ubuntu11.1 (focal-security, focal-updates, now)]

This means that we need to remove imagemagick in favor of the forked graphicsmagick which also provides a compatibilty layer for imagemagick.

  1. Once you installed darktable-tools-noise, you need to remove the check if ! convert --version >/dev/null 2>&1; on line 137 within /usr/lib/darktable/tools/subr.sh. This check somehow seems to only succeed using imagemagick but not with graphicsmagick, which is weird, because we were forced to remove imagemagick in favor of graphicsmagick to begin with :smiley:

  2. While running /usr/lib/darktable/tools/darktable-gen-noiseprofile -d $PWD in the denoising image folder the following warnings popped up:

convert convert: Unrecognized channel type (-channel).
/usr/lib/darktable/tools/subr.sh: line 488: [: : integer expression expected
convert convert: Unrecognized channel type (-channel).
/usr/lib/darktable/tools/subr.sh: line 495: [: : integer expression expected

Nevertheless the profiling succeeded and the results with (wavelet-auto) look convincing :+1:
The noise_result.pdf (400.0 KB) looks quite interesting though.
Here is the final presets.json (1.8 KB) file.

Here is an example: DJI_0751-thumb|690x459, 50%.
I also uploaded my RAW files and the generated profile (whole tar.gz file):
https://filebin.net/j6g9z6gic4782zr4

I also created a PR to get the profile into darktable master: Add Hasselblad L1D-20c (a.k.a. DJI Mavic 2) to noiseprofiles by da-phil Ā· Pull Request #7310 Ā· darktable-org/darktable Ā· GitHub

Please open a darktable bug and contribute the profile so it will be added upstream! Thanks.

Thanks, the profile has been already merged to master: Add Hasselblad L1D-20c (a.k.a. DJI Mavic 2) to noiseprofiles by da-phil Ā· Pull Request #7310 Ā· darktable-org/darktable Ā· GitHub

1 Like