Darktable/Rawspeed clipping more highlights than other Software

But I did that. Disabling filmic, using base curve, not reconstructing highlights etc. I never get any more details in the sky than on the screenshot on top. And look at the results from the other two programs. That’s not looking like any clever trickery reconstruction, but they seem to be actually “seeing” more data than Darktable.

Also, it’s only Darktable in this case. Rawtherapee gets the same results as On1.

What I suspect is that the white point for the Pentax K-5 raw (in-camera DNG, btw) is wrong.

Good morning, @firefrorefiddle, and welcome!

a) Here you will find more info on how to “add a license”: [PlayRaw] Waiting for the Big Boy - #4 by pphoto

b) @rvietor’s advice (above) is a very good start.

Of course dt as well as RT can squeeze much more info from the sky, here are three very rough, swift examples:
RT:


dt:

dt 2 with more oompf (and by using clever masking, you can restrict the oompf to the sky):

Have fun!
Claes in Lund, Sweden

1 Like

Thanks, Claes!

Hm… I stand corrected. The details are actually there. But how did you do it?

I assume, that you have Scene referred workflow activated. Then it’s most likely filmic’s behaviour, that’s causing the clipping. That’s not a problem through, because you can always change that by adjusting white relative exposure. If uncertain, then you can test for clipping in RAW data in darktable.

Hoping, this is of any help.

Daniel

Start with rvietor’s advice.
To achieve the oompf, use
the local contrast module.
Play around with the controls.
In my case, I used
Screenshot_2022-04-05_10-53-43

Note: This will make all flowers and trees go crazy,
of course, so to get a more “natural” look, you have
to mask for the sky.

Play around & have fun!

Claes in Lund, Sweden

PXK55990.DNG.xmp (16.2 KB)


I tried to extract texture, then used graduated density to copy colours into the blown region.
darktable 3.9 from darktable windows insider program 4/3

2 Likes

It looks reasonable, at least for the RPU K-5 DNG sample:

Exif.SubImage1.WhiteLevel                     16316

Could increase it all the way to 16383 (14-bit max) in the rawprepare module, but doubt it’ll make a significant difference…

Thanks Istvan,

you did great! I am happy to look at your XMP file, it’s very instructive. I see now that my original suspicion was totally wrong and the issue was actually caused by the highlight reconstruction module (which I wasn’t aware that it was on by default). Also, I learned about the grey vs colored highlight reconstruction in filmic which is very useful.

Overall I have to say that I am really happy with darktable, the logic it uses and the results it can produce.

And thanks kmilos for looking that one up!

2 Likes

I said to disable filmic to check what’s in the file. Not to enable base curve.

Having nothing activated but white balance and exposure , you should be able to move the exposure slider freely to check where your data stops and starts.

A white point misread from the raw file might actually be a possibility, but I doubt it.

The problem was not caused filmic, but by the highlights reconstruction module. I didn’t know that it was on, and turning it off or changing the method instantly showed that the details are there.

Note that in many cases you can use the highlight reconstruction module in reconstruct in LCh mode; it gives much better results than the default clip highlights mode; on the other hand, reconstruct color is prone to artefacts.

The way I shoot now is to underexpose in order to protect the highlights, then let the exposure module undo my in-camera exposure correction. Of course, I’ll let the Sun, lamps and specular highlights be overexposed. :slight_smile:

Maybe check out this thread for many approaches to fixing highlights:

2 Likes

You will also notice if you use filmic that the chosen norm can really impact the highlights…so you have to take that into consideration.

@jorismak is telling you what you need to know - the sky was clipped in the camera.

I opened your raw in my hack software, where I can show the raw data without any processing:

Look at the histogram, middle-left pane. The sky pixels are all piled up at 16313, apparently the largest 16-bit value your camera can record. The RGB values for these pixels is 16313,16313,16313, or plain white when scaled for display, no detail.

None of the programs showing you detail is recovering it, they’re making it up. A better approach is to pull your camera exposure down to get more of the sky in recordable range, and then to lift the shadows to suit in post-processing. The filmic curve is good at that, but you have to watch that its upper-end shoulder doesn’t push the highlights right back to desaturation…

I will just add my 2 cents worth to this discussion without comparing software. I have found when working with partly clipped highlights (not all channels clipped) that experimenting with the three methods available in the highlight reconstruction module can be helpful. See which gives best result. Also with filmic I don’t push the white relative exposure slider too far left as that makes the problem worst. In filmic V5 (which I presume you are using) try moving the latitude slider found in the look tab towards the right to put saturation back into the highlights. Filmic V6 plays differently. I am still getting my head around filmic V6, which I expect will be in the mid-year release of DT. If after these steps I still have problems with highlights I might then and only then go to the reconstruction tab and apply a threshold for the masking of highlights and adjust the sliders as required. I will not offer an explanation of how I approach those sliders here.

I am very pleased with the way DT handles Highlights and the control you have for the look.

I see Filmic v6 quite often mentioned. Since I like only official stable releases installed, may I ask what is coming in v6? Differences? New functions?

1 Like

I suspect you might be on Linux but if not I have posted a portable version for Windows that has Filmic v6, sigmoid (experimental) , Laplachian HLR, and the new exposure matching and color matching features. You don’t need to install anything. It uses a config folder in the program folder and is set not to write xmp files so it should not mess up anything. Just delete the directory when you are done. If you run a minimal Win dual boot it should also be easy to use… But it may also not be something you can or want to run if you are strictly running Linux…

1 Like

Hmm… interesting. I was aware of custom builds, but somehow missed they’re portable and “mess-free”. So far I had some brief episodes with Linux, but recently I’ve bought a second SSD and I can safely keep Windows on one, and play around with Linux on the other - I hoped especially for best possible experience with darktable, and Arch (well… EndeavourOS to be specific) looks promising. Extraordinary performance :clap:

I just made a couple of tweaks to make it portable as explained above…The little batch file runs DT locally from what ever folder/location you unzip so for example could just be downloads…and I add and empty config folder in the program folder which is not the normal location and then add that command switch to the batch file…I set it not to write xmp files so that nothing inadvertently gets messed up… Its still just for testing not daily use but it does include access to some of the latest features added or about to be added to the master branch and a proposed tone mapping module that has had considerable discussion… It sound like it should be no issue for you to take it for a spin…

1 Like

Right. Thank you for this :slight_smile: