Finding D max in Negadoctor with distortions in RAW shadows

Software darktable
1

I scanned about 5,800 negatives in Vuescan with my Minolta Scan Elite II, using a fixed exposure for each film roll.
Just to explain my workflow: My intention was to convert the film rolls in Negadoctor by picking individual features where they apply (see * below) in order to create a general profile of the film roll’s features. Then I would merely adapt the “print properties” (apart from “paper grade”) of the individual images.
So far, so good.

Unfortunately, in the DNG files, the scanner produces some very dark pixels in the green channel, sometimes even a kind of weak white-noise pattern in the RAW shadows. In many cases, I wouldn’t mind, because these pixels would disappear in the highlights range of the converted image. However, when sampling D max with the color picker, these extreme shadows are counted as the “densest” areas of the film, thus creating a higher D max and pushing the rest of the image into the shadows.
If such a “faulty” D max is kept in place, you can counteract it with the print properties sliders on paper black and print exposure (including paper gloss) - but only within the “distorted” spectrum of the nonlinear curve that D max imposes on the overall picture - or via the “paper grade (gamma)” slider - which may impose even more nonlinear distortion.

I feel really insecure about whether a D max is really based on a “correct” value or just a distortion or “outlier” - particularly since this value is very important for the entire character of the image. Also, this potentially affects a lot of scans with high loading times per image.

Of course, there is a default value of D max in Negadoctor, but there is no real standard range of D max, since it depends on many factors (e.g. exposure of the scan; film stock). In one forum discussion, Aurélien highlights an aggressively low D max of one user, while he uses a low one himself in a Youtube video.

  • I could try to find these faulty pixels and exclude them from the color picker’s sampling area - exhausting.
  • I could also work with the histogram - with all corrections in the later tabs disabled - could be imprecise
  • I tried using a blurring filter - but this also “pushes down” the correct highlights in the RAW
  • De-noising (on the RAW, before Negadoctor) does not seem to affect those extremely dark pixel values, but rather, the “correct” ones above them.

1 - With Negadoctor deactivated, the color picker module shows some pixels with green at pure black
1 - With Negadoctor deactivated, the color picker module shows some pixels with green at pure black1919×941 1.3 MB

2 - If sampled from the entire image, D Max is extremely high
2 - If sampled from the entire image, D Max is extremely high1919×945 330 KB

3 - If I sample D Max from the sun, it also contains some black in the green channel, forcing me to use the histogram only
3 - If I sample D Max from the sun, it also contains some black in the green channel, forcing me to use the histogram only1919×941 1.01 MB

My question:
Can you see any better solution to this?
Is there some kind of low pass filter that specifically excludes (brightens) such outliers in the RAW (in the pipeline before Negadoctor) without affecting the “correct” values too (as a tone curve would)?

I could add sample DNG files, but each is 232MB in size.

*footnote on sampling sources: film base from test strip, D max [and paper grade] from the brightest photograph, scan exposure bias by comparing the values of several images containing “real” black, color corrections from images with a neutral light source - if possible

2 Likes
2

I always click the D Max picker first, but it often fails, especially when there are dust or scratches on your negative. So I then just try picking other light areas to get a good result. But I often revert to just eyeballing it.

Increasingly, I have just tried to get “good enough” with Negadoctor, and then tweaked in other modules like Color Balance RGB. I sometimes even use a neutral Sigmoid instance to fine tune the black and white points.

In Darktable, there is often a “technically best” way of working, but in practice I have often found that there is very little visual difference if you use another non-recommended method. So, depending on your specific needs, you might find it’s just easier to use different tools than Negadoctor after the basic conversion.

2 Likes
3

I really share your frustration with the color picker failing.
In my case, thanks to the digital ICE, function of my scanner, dust only becomes a problem when this function has to correct too large patches, still leaving “extreme” spots. I guess that using a camera-based method leaves you with more dust problems. After some “research”, I believe that my problem is based on some noise issue of my scanner sensor in the very low end of the spectrum (hence, only highlights [after conversion] are strongly affected). Or it could be some problem with how this is processed in Vuescan, producing grain aliasing. The result (after conversion) is either dotted/strip-like paterns of “too” white pixels in highlights - or elsewhere, pixels where at least one RGB channels randomly hits the maximum value.

No matter how this is produced, I’ve opted for a mix of modules to counter this effect. Of course, whether this works for others, depends on many factors.

  • Denoise (profiled): Wavelets; Linear [default] diagram, strength 3, preserve shadows 0, bias correction 0; masking (I want to retain the grain in the rest of the picture): gray value 0 - 0 - 0 - 4.3 [applied on the negative, since the module comes before negadoctor]
    This evens out most visible stripes in the (inverted) highlights which mess up the look of extreme highlights like reflections, lamps or the sun.
  • Diffuse or sharpen: Iterations 3; central radius 8px, radius span 3px; speeds: all at 100%; anisotropies: all at 0%; masking: gray value 0 - 0 - 0 - 1.6 [also applied on the negative before negadoctor]; blurring radius 4.0px
    Using this module without the mask, the settings appear pretty random, but with the mask, it attempts to eliminate (dampen) only those final outliers which Denoise didn’t catch - and which tend to give you such extreme D Max values in Negadoctor. Of course, you could also use a similar module like Blurs, but there you don’t get such varied options for “playing around” with different diffusion parameters. Since I only wanted to apply this to the extreme highlights and quickly let it “fall off” below that, I had to test the settings on various images in order not to turn extreme highlights into washed out darker dots surrounded by unaffected lighter rings where the mask hat fizzled out (imagine a candle with a dark spot in the middle). Also, you have to constantly check the histogram and repeat applying the color picker in D Max.

As a result, the color picker in Negadoctor’s D Max now gives me much more normal values in all images - and thus, a more acceptable range in which to place the entire picture. At this point, I can more easily rely on the automated step of letting my Autohotkey script grab D Max from the selected “brightest” images of my 285 film rolls.

I guess you could still improve the individual settings (particularly in diffuse or sharpen), but I’m happy for now. It was only the “solving” of this problem that allowed me to realize that some analog photographs were simply underexposed and needed further corrections - as discussed in my other recent post (Fixing negative underexposure in Negadoctor) which you also reacted to.