darktable 4.0 default "highlight reconstruction" RAF problem

Hi,

I just upgraded my good old darktable 3.old to 4.0 on Linux and the shiny new highlight reconstruction module immediately threw an unexpected problem at me. Description:

  • clean darktable 4.0 fresh install,
  • stock out of the box default settings, nothing was altered, module order is “v3.0 RAW” as by default,
  • a correctly exposed RAF file with some tiny spots of overexposure on the clouds, white fabrics and sun glares (the “raw overexposure” indicator shows just a few red dots in each, nothing serious - even “clipping indicator” is silent, no clipping detected) is being opened.
    In darkroom, these (supposedly) overexposed areas are covered by flat solid patches of a faded bluish color. No, this is not a neutral grey. After some research, it turned to be a problem with the “highlight reconstruction” module painting highlights this ugly way - instead of a neutral white.

Some fiddling with it revealed that

  1. yes, the “show mask” confirms overexposed areas locations,
  2. the “reconstruct in LCh” method only makes the flattened patches bigger in size,
  3. the “reconstruct color” method removes the flat ugly patches, but loosely injects the areas with magenta pixels instead; but Ok - at least no solid patches here,
  4. the “guided laplacians” method is not available (non-applicable?) with X-Trans RAF.
    More experimentation with “clip highlights” method and the “clipping threshold” slider (the one equipped with the warning “you shouldn’t ever need to touch this”, ha-ha) revealed, that moving the slider all the way right (to 2.000) retains solid colored patches but changes their color to magenta.

Moving the slider the left direction from the default value of 1.000, predictably increases the size of the areas covered with that solid faded bluish flat color, and adds some burnout to the surrounding areas, too.

So, my idea of getting rid of the patches just with the slider, immediately failed miserably.

darktable applied the exposure correction +0.5 EV on import by default. I tried decreasing it, the histogram obediently moved to the left and ceased to touch the rightside clipping border, but the colored patched did not change.

My question is: how do I make my (slightly) overexposed spots at least neutral white? And, if possible, how to make all available highlight reconstruction methods equally non-destructive with regard to colors.

Right now I need the urgent solution and a few RAF frames only, so the retouch module will save my life this time, but for future I’d prefer some more robust and unified solution.

Please help me solving this puzzle! Thanks in advance. Regards, Andreas

UPD. Here you can look at a sample RAF. Not the one I was speaking initially but suffers the same problem. The patches are visible at the nose bridge, at the shawl, at the forehead, and on the hair of the lady.

if there’re just a few overexposed spots you might try o ditch highlight reconstruction module an do the stuff in filmic.
Have a look at [Darktable] Broken 'Pink' highlights and how to fix them - YouTube

3 Likes

Most likely it’s not worth it, you’d rather dial it up, as Fujifilm cameras underexpose more than other manufacturers by default.

And since the contrast is built around (properly chosen with exposure module) middle grey, it’s better not to decrease the exposure.

Without seeing your image, I imagine your problem will be solved if you:

  1. Turn Highlight reconstruction OFF
  2. Add +1 or +1,25 EV
  3. Set Filmic RGB as follows

I’m not sure the new HLR mode is for non Bayer sensors… not sure if that could be some of what you see… also it is very sensitive the the threshold. I have found the best results are to raise the threshold to whatever produces magenta highlight and then slide or scroll down slowly… at some point the magenta usually just disappears and then you are very close to the best solution… this could be a value greater than one but generally I find has been between 0.84 and 1… limited testing though I might add

Edit. any chance you can share an image

Thank you I’ll try this. The link to the sample RAF image is added at the bottom of the original post, do you see the patches, too?

Thank you for sharing the hint! I’ve added the link to a sample RAF at the bottom of the original post.

Ditching the highlight reconstruction resulted in magenta spots in highlights instead, which are somewhat smaller in size. Once again, the “raw overexposure” shows only a few random red dots there so I don’t think that the shot is really overexposured.

You will need to see the video recommended…based on the norm used you can see exaggerated issues like this in the highlights… esp with the default maxrgb. Did you try to use filmic reconstruct from that tab…does it help?? Can you shoot some random image with the same issue if you are not able to share these ones with the forum

This video is a must watch for anyone who is not happy with how DT handles highlights. It explains the problem so clearly and provides solutions.

4 Likes

His small series on filmic is also very good…

1 Like

Hmmm… I don’t think so - perhaps we’re looking at the same thing, but I don’t see any problem :wink:

Anyway, that’s my quick edit:

TTTT6752.RAF.xmp (11.9 KB)

There are some small troublesome patches, like the nose and the top of the scarf. Even though those areas don’t show up as clipped, there’s a lack of detail in them that makes me suspect the default raw white point is set a tad too high. But I can’t find an embedded value in the .RAF.

What bothers me more is the moiré effect in the blouse (look at the 100% image, full size aggravates the problem by adding yet another grid…). Applying some operations that result in a faint blur help (denoise, chromatic abberation, …) but removing the moiré completely is hard (and the result depends on the viewing scale, so not sure what happens on export).

1 Like

Thank you @rvietor and everyone (I am unable to mention more than 2 people in 1 post), for your replies and suggestions! What I plan to conduct later today/tomorrow:

  1. Experiment with the raw b/w point setting - I didn’t ever touched this before, will try.
  2. Analyze the edit kindly shared by @Leniwiec - thank you! and try to acquire some hints from there.
  3. As for the moire yes, I’ve noticed this, too - so I’ll begin with setting the demosaic module to the “Markesteijn 3-pass + VNG” mode first and add 5-pass color smoothing. If this won’t help, I’ll try the other hints and ideas from around.
  4. Also I have Win10 as a “double boot” option, with its filesystem accessible from Linux, and which is already equipped with X-Raw Studio, C1 for Fuji, and SilkyPix for Fuji. So I’ll perform a comparative mini-research to see what should be done to darktable 4.0 to provide the best results among three others.

The idea of tweaking the filmic’s reconstruction part doesn’t seem too attractive to me as of now. The “highlight reconstruction” module is a nice and useful, carefully crafted piece of software which occupies its precise position in the pipeline. I don’t want to ditch it, I want it to work for me as intended, leaving the filmic step to its own part of work later in the pipeline.

So thanks once again, I’m going to my work now and I’ll share my findings back ASAP. Regards, Andreas

I think so. It comes up as 16383, the 14 bit max. RT’s camconst has it at 16100, in which case it clips like this -

camconst -
"make_model": [ "FUJIFILM X-T20", "FUJIFILM X-E3", "FUJIFILM X100F", "FUJIFILM X-H1" ],

"dcraw_matrix": [ 11434,-4948,-1210,-3746,12042,1903,-666,1479,5235 ], // DNG_v9.4 D65

"raw_crop": [ 0, 5, 6032, 4026 ], // full raw 6160,4032, Usable 6032,4026 - for uncompressed and lossless compressed files (but reduces height by 6 pixels)

"ranges": { "white": 16100 }

1 Like