Processing skies with clipped channels in Darktable filmic/sigmoid

[ariznaf (Fernando)]

Thank you for your time and contibution. A quite natural look, I like it.

Yes probably you are right, we tend to put too much attention to the sky.

But I do not tend to dramatize them too much.

The problem is that I don’t remember that day with such a dull sky.
It was a luminous day, with some clouds but a blue sky, quite deep blue indeed,

This image would have beter results with less exposure, and probably the shadows won’t have suffered that much.

But I selected it to see what can be done in DT to recover and expand lights in that situations.

With capture one of LR (and may bit a bit of PS later to use some layer and playing with fusion modes) it was not difficult to get a pleasent result.

In DT it is giving me a lot of work and many times I don’t get pleasent results.

[davidvj (David Vincent-Jones)]

On my display this provides a well detailed sky
DSCF0175.xcf (16.8 MB)

[gaaned92]

ART tone curve, tone equalize, HR/color propagation, local contrast

[ariznaf (Fernando)]

Thank for all responses.
I will have to study the provided answers and try to implement them myshelf.

I have another photo with only some part of green clipped that I would like to test with what was provided here.

@gaaned92

ART tone curve, tone equalize, HR/color propagation, local contrast

That is the kind of natural look with no artifacts in recoverd zones and color preservation I was seeking.
It would be easier to force it a bit to get more deep blue if you like it in those images.

Does RT and ART use linear scene refered mode or they use the traditional workflow working in non linear space?

As long as I know, LR uses as working space a kind of ProPhoto space but with a linear or at least no standard gamma.
But it uses the camera tone curve in the first demosicing steps, if I am not wrong, so the image is not really linear in the procesing steps.

[priort (Todd Prior)]

@ariznaf

And what about the threshold and transition and other sliders in lights reconstruction in filmic?
Have you adjusted them?

If you don’t adjust the threshold filmic is not doing any reconstruction…its takes some cpu so by default the threshold is high essentially disabling it… you need to turn on the mask to see what you are working on…

[g-man]

@ariznaf

With capture one of LR (and may bit a bit of PS later to use some layer and playing with fusion modes) it was not difficult to get a pleasent result.

Can you show this image with pleasant results using any other software?

[davidvj (David Vincent-Jones)]

I really found that dt did a great job with very little effort. Mainly filmic-reconstruct and masked local contrast … what’s all the fuss?

[Jose_Figueres (jofial)]

my proposal…

DSCF0175.RAF.xmp (12,4 KB)

Cheers

[Jose_Figueres (jofial)]

@ariznaf

I am not sure of channels being clipped as much as it seems, may be an effect of color balance, I would like to see the real raw histogram.

I don’t think so either

white balance according to shooting

Cheers

[ariznaf (Fernando)]

Yes, that is true, in most images channels are not as blown out as they seem to be, color balance many times make them get more clipped.

But it has blown channels, if you deactivate color balance and any module, you see it.

In Dt the raw channel indicator is supposed to show blown channels.
But am not sure if it shows them from the raw with any transformation or after demosaicing or color matrix calculations.

[elstoc (Chris E)]

@ariznaf

In Dt the raw channel indicator is supposed to show blown channels.
But am not sure if it shows them from the raw with any transformation or after demosaicing or color matrix calculations.

It takes white balance into account so it’s not “real raw data” from the sensor pixels. See discussions on Feature Request: Toggle in Raw Over Exposure Indicator to show "real" clipped pixels · Issue #6596 · darktable-org/darktable · GitHub for more background.

[ariznaf (Fernando)]

That was my impression seeing some photos and deactivating color balance.

A pity, I would like to see somewhere real data about clipping.
I understand that many times it would be more useful to see it after color balance, as not being the channels blown, it may get blown after color balance.

But in an unbounded workflow having values above 1 should not be a problem.
The problem may come with colors as the clipped channel would have a value lower than it should.

There is where a good recovering algorithm is handy, to restore the value from surroundings.

A raw histogramm would be great too.

[apostel338 (HP)]

DT 3.9.0-116

DSCF0175.RAF.xmp (13.9 KB)

[ariznaf (Fernando)]

Let as see how badly exposed was that image.

Most images that seem badly burned are not, they are only burned in the green channel usually (depends on light, but it is the more common case).

This is the case of this image.
I have seek for a too to show raw data and histogram and I found it here (just for reference it is not an opensource tool): RawDigger | Everything You Always Wanted to Know About Raw(But Were Afraid to Ask)

This is the histogram, shown some clipping just in green channel:

And this are the clipped zones in the green channel (1.7% of the image) in a zone quite uniforme with not too much gradations.

Red an blue are far from clipping.

So not that bad the job from the camera meter.

The clipping comes from the color balance and demosicing process then.

It surprises me in an unbounded workflow.

Ir channel green can be recovered to some extent to provide a similar color to surroundings, there should be no problem later, and no problem displacing and expanded that highlights to the midtones.

[priort (Todd Prior)]

This is the raw histogram from Art

Art and so likely RT have a raw sensor gain setting…not sure exactly how that acts or if its a simple multiplier…dropping it a bit yields this…

[kofa (István Kovács)]

Important edit: the charts that were originally here are mostly meaningless. I’m keeping the post so you can see what @ariznaf responded to, but these histograms clearly do not match the raw histograms from RawDigger. Even if I set all profiles (input, display, histogram) to Rec2020, the plots don’t line up.

In darktable, by disabling most modules, then setting demosaic to photosite color and input color profile to Rec2020 linear (same as working space), I think the histogram shows the raw data:

.

Or, in rgb parade mode:

No multipliers due to white balance or due to colour space conversions.

But there is something I don’t understand. Look at the green channel in rgb parade mode. It reaches much deeper than the others. If I change the raw black point in the raw black/white point module, more details emerge:

It looks almost as if we had a somewhat squashed version of the original green channel, repeated in the region below the original. One difference is that it does not show the flattened top part, characteristic of clipping, exhibited by the original green region.

Can anyone make sense of that?

Edit: it may actually be a bug, as I see it in other files, from other cameras, too:

Setting the display profile to sRGB removes the ‘ghost’. Maybe my display profile (from DispCal) is broken?

[ariznaf (Fernando)]

You can see that it is not the raw data by seeing that blue and red are very high at some parts of the image.
So it seems that the histograms shown are color balanced.
The shown histogram is just pure data, linear in x, pure raw and linear data.

Linear too in Y axis. Your histogramm seems to be lov in x axis at least, that comprises data to the right.

The DT histogramms are always shown in the selected output profile, till I know, so it is not raw data, there is allways color cooking even when you disable color balance.

That is the data DT receives. As it is linear and unbounded for most of the pipeline till filmic, there shoild be no clipping problem or artifacts at all.
It only has to recover a small part of the green channel.

If there are problems with the result it has to be in the recovery steps or the tone mapping.

Even when there is no clipping it is not easy to expand that highlight zones to the midtones to compensate the strong compression filmic or any other tone mapping has to do in that part of ghe data in order to keep midtones with good contrast and the general grey point.

May be it is just a matter of the available tools not letting you beaggresive enough, like in color balance rgb and the available range of darkening for highlights.

I get better results with color balance and playing with bleu and red, and masks, exposire also helps, but it displaces the white point too, and you wante to keep the white and expand the zone at the last one or two EVs shere there is a lot of data in linear mode.

In other tools using a traditional bounded non linear workflow it is easier to get that expansion, when they should have more problems than DT with this kind of data, as they clip a lot of info in the first step of color balancing and generate spikes in data, being that data integers.

So I don’t think there is anything inherently wrong in DT workflow, I think it is in advantage respect the traditional approach.
You just have to be more aggressive playing with highlights in linear mode than you use to, and need that tools to be more aggressive.

[anon41087856]

First of all, there is something fishy in the 3rd edit. If I try to contrast-match it and color-match it for the yellow-ish building on the right:

RT output JPEG

DT output matching 3rd OP picture

you see that nailing colors for the building at 47% L produces a completely different rendition in highlights, with a large blue skew, so there is something non-linear going on here, perhaps in the profiling. And this is with a modified filmic version with disabled desaturation, so the curve should preserve hue and saturation 1:1 because it’s essentially an exposure change. (NB: saturation = chroma / lightness, so it’s normal to see chroma increase with exposure correction).

Then, if I simply lower the exposure to frame the sky around 50% L (no filmic, no contrast curve, no chroma boost, just exposure + WB + input profile), this is what I get:

Notice that the sky registers at L = 50%, C = 0.48%, h = 6.58°. The filmic match (with norm = max(RGB) and desaturation disabled) above registers it at L = 85%, C = 0.72%, h = 6.45° so, it’s the same color ± the average side-effects.

So:

1. your RAW file does not have a lot of details, beside being partly clipped,
2. the filmic rendition is pretty damn close to the reality,
3. what you call “natural” is actually already enhanced.

In conclusion, you need to work manually on this enhancement because darktable has no formula to make pictures look good since we have no numerical model of “good looking”. Other software skewing colors in a way that looks better to you is a different matter, as far as darktable design goals go, it’s mission accomplished.

You best luck here is:

1. darken the sky with tone EQ
2. mask the sky in color balance RGB, set the white fulcrum properly, darken on the power channel and increase the linear global chroma by a lot.

DSCF0175.RAF.xmp (11,9 Ko)

[ariznaf (Fernando)]

Thank you for your guidance.
I will try to follow your instructions to see if I can improve the edition.

I will try to answer other comments you make.

I don’t need the software to tell me what is “good looking”.

What is good looking for me only knows me, for other may be other things (different edits of these same boring photo hints you that everybody has its own conception).

What I need from the software is the ability to let me modify the image as much as I want and expand lights or compress them.

After all the talk here and in the sigmoid thread I am coming to the conclusion that indeed there is nothing wrong with DT workflow or even the tone mapping made at final steps (well you can do it in different ways, but as you point your at the end taling about the last one or two EVs).

The problem many of us find is that the tools (like color balance RGB for example) don’t let you be as agressive as needed to expand al that info you have in the las two EVs to midtones and counteract the final compression that is going to happen in the tone mapping (be it sigmoid, filmic or whatever).

It seems that being in the linear edition workflow seems to need more range of liberty.

I am sure that using DT a linear, nonboundedn and flotinggg point calculations, it potentially can get much more accurate and better results than any other traditional approach.

But with old tools you could these easily (or with more ease). It were not perfect, you have demonstrated there are flaws in it that are shown when contrasting.
But that is the final image, it does not bother me (or not too much) if there are imperfections as long as the eye does not detect them.
The eye is a silly thing, easy to distract and cheat.

Using mask and the color balance trick and reducing exposure (this is a bit problemathic as the white point is changed) I could get better results thaat I was previously gettinng with color balance RGB.
But it keeps being a bit difficult to get “dramatic” results or even natural looks with current tools.

Let appart the problem with color reconstruction of clipped channels even in areas not so big or with too much transitions, more or less uniform (it creates artifacts) but that does not bother me that much, as it seems a recognized problem and there is work in progress to solve it.

I only need a way to mask it from the eye inspection.

Many have addressed me to underexpose the image (wel this is a bit overexposed with a bit of green channel blown, but not that much blown).
Well todays cameras let you underexpose and then compensate in development, as they have a lot of DR usually (except when you are in situations of low light and you need to make your ISO higher).

But most of the info and least noisy one is in the higher EVs.

If you underexpose one point (in the ETTR sense) you are throwing away 1/2 of the infor your camera can capture, 3/4 if you use 2 and 7/8 if you use 3 EV (like when you measure in spot mode in the highest light and do not compensate).

[Jade_NL (Jacques)]

@anon41087856

First of all, there is something fishy in the Rawtherapee edit.

Which RawTherapee edit would that be? It’s not the one posted by nosle.

So what did you do? Where’s the pp3? Why do you pull the RawTherapee output (a JPG, PNG, TIFF? 8, 16 or 32 bit??) into darktable and change the exposure?

For someone that’s sipposed to be scientific you are really vague here. At least post the pp3, xmp and JPGs (or PNGs or TIFFs) you used and tell us how you (tried to) match them. I’m sure others would like to repeat this and see for themselves how this holds up.

Don’t just take a swipe at another RAW editor like this.

Also: Use another program to compare the two?

There’s a nice Dutch advertising slogan from back when: Wij van Wc-eend adviseren Wc-eend. This would translate into something like: We from darktable advise using darktable. Of-course you would