How to create camera noise profiles for darktable


(Schenlap) #21

I am the one how does most of the darktable noise profile tickets in redmine. Accepting noise profiles is always a subjective process. Always attach the generated tar.gz file (without any modifications), it contains all the information that is required. I first look at noise_result.pdf mainly the “noise_levels” (for good fit) and “flat noise levels” (should be near the ideal line). For noise profiling only the green channel is used. Afterwards I look at the images if they are out of focus. For the images in the initial post i was not sure if I they are good enough. In my view there are better images like the one from elstoc as they have more pixels between over- and underexposed ones. Also profiling images could be more out of focus when using a tele lens, which have a smaller depth of field.

Only profiles from camera with original firmware will be added, also please do not modify presets.json file!


(Schenlap) #22

Raw linear style is applied in the script, no need to do anything in darktable before.


(Tore Busch) #23

I have compiled and installed darktable-2.4.3.tar.xz

I tried to follow the tutorial last night, and everything seemed to run smoothly, but I got an error message after darktable-gen-noiseprofile had processed the raw files.

From the directory I had my raw files in:

$ /opt/darktable/libexec/darktable/tools/darktable-gen-noiseprofile -d .

...
===> Prepare profiling job
--> Remove previous presets
--> Ready to profile images

NOTE: This process takes some time and a lot of memory and disc space
(up-to several gigabytes, depending on the number of ISO settings and
the size of the RAW files.
ERROR: Could not find darktable-noiseprofile tool in /opt/darktable/libexec/darktable/tools

After a while, I grepped after the error message and found it in line 138 in darktable-gen-profile, where there was a check for a file called ‘dt-noiseprofile’, and indeed, no such file existed.

So I made a softlink in /opt/darktable/libexec/darktable/tools :

ln -s darktable-gen-profile dt-noiseprofile

and FINALLY it ran as it should and generated my new noiseprofile.

But, have I missed something in the build-process, or is the check for ‘dt-noiseprofile’ a check from a time when the script actually was called that?


(Andreas Schneider) #24

Ouch.

Yes, there is a typo in darktable-gen-noiseprofile in line 137:

- if [ ! -x "$scriptdir/dt-noiseprofile" ]; then
+ if [ ! -x "$scriptdir/darktable-noiseprofile" ]; then

I will create a patch tomorrow.


(Tore Busch) #25

I suspected as much. Thank you for the answer :slight_smile:


(Eduardo Battaglia) #26

Hi! Did you have a chance to look at my redmine post for the Nikon D5600? (https://redmine.darktable.org/issues/12116)
Any feedback will be useful

Also due to the lossy after split issue I can’t profile high ISOs files, if I convert the NEF to DNG using Adobe’s DNG Converter, can I profile those DNGs?

PS: I love how FOSS allows us to get in touch with devs and contribute if possible.


(darix) #27

can you stuff the patch into the graphics:darktable package?


(ron) #28

Thanks for the info! I am in the process of picking up a pretty noisy camera (it has other benefits that outweigh the noise) - so when I use DT (hopefully will start with that soon) - I’ll be all set to go thanks to your explanations.


(Peter) #29

I’m struggling with de-noising. I’m using darktable for Windows so can’t go through “Creating noise profiles” steps. Don’t have denoise profile for Panasonic GX7 for ISO 320, 500, 640, 1000, 1250, 1600, 2000, 2500. The 1600 is a basic setting, other are set by camera. Is it possible to upload (or share via cloud) samples and ask somebody to do the job? If this request is acceptable please let me know and I’ll prepare files. Does it make any difference in lens I use for preparing such samples? I have Panasonic lens 14-42/3.5-5.6, 100-300/4.0-5.6, 20/1.7 and Helios 58/2


(Mica) #30

If you can post the necessary shots, I’d be happy to run the noise profile generation script on them.


(Trougnouf (Benoit Brummer)) #31

Could it be beneficial to redo this work if my camera is already supported? The tutorial I saw previously didn’t go to such great lengths (no punch card) and my camera support came from an older firmware (there has been at least a change to the white balance, which I’ve been trying to merge)


(Andreas Schneider) #32

Maybe, maybe not. However it would be great to find something which is better than punchcards. I haven’t had the time to experiment again, as I’m working on other tutorials.


#33

Hi,
I made some pictures for noise reduction profiling with the RICOH GR 2 which is as far as I know not supported yet in darktable. I’m not yet a user of darktable and don’t have time at the moment for compiling the profiles myself. Is somebody willing to complete this task if I send the pictures? Would be very nice.
kind regards


(Mica) #34

Yes, please release the images under a license that is free like creative commons zero, and I’d be happy to process them for you.


#35

Allright, I’ll have to figure out how to do this first and then they’re yours. Thank you very much.


(Mica) #36

You can either include the license in the files metadata or include a text file in the zip stating the license. Then post them to filebin.net


(Walt French) #37

Hi. Brand new to dt etc so apologies for n00b ask…

I recently developed my OWN noise profiles by shooting a Gretag MacBeth chart (out of focus), then writing my own s/w that looked at the variance of each color panel. Shot some 200+ photos using 5 different bodies. Usually -2, -1, +0, +1 and +2 EV for each bod/iso.

Time consuming!

But I was able to get a model for each body/ISO combo…about 360 points for each (3 colors, 24 patches X 5 exposures).

These fitted VERY well to Poisson distributions, and looking at the small sensors’ low-brightness values I could even see granularity of individual values times a multiplier—e.g., values of 13, 26, 39… suggesting 1, 2, 3 photons captured—before other artifacts made it hard to ID the true Poisson info.

I didn’t look TOO hard, but also didn’t see any color-specific interaction. Things like bleed across adjacent pixels, or the blue brightness affecting measured red average/noise.

Because of the strong reason to expect Poisson distribution, I treated deviations from it as non-linearity of the sensor amps—I tweaked the brightness so that it fit Variance = k * brightness.

All in the service of my work on a RAW de-noiser system that could perhaps complement it’s.

Anybody caring to comment on weaknesses in my approach, or sloppy interpretation I made, I’d really appreciate hearing.


(Andreas Schneider) #38

I’ve updated to article with some easier way to create pictures.


(Chris E) #39

Have you had any new noise profiles accepted by the darktable project using this method?


(Andreas Schneider) #40

Today we looked at the plots today and they look great. If you want to take a look: