Trying to figure out RGB (filmic) workflow

[jorismak]

@herbert-50

The problem with png is that many modules (e.g. exposure) work differently with RAW files. So it’s not really comparable.

I don’t know of anything that exposure does differently depending on input file type.

In fact, the only module that Ï know of that behaves differently depending on input data, is the highlight reconstruction module (because sometimes it requires bayer data for example).

after that and the demosaicing step, it’s just all floating point rgb data. Scale, squash, tone-map, do with it what you want.

Your raw sensor data is always between values of 0% and 100%. Because the file can’t represent more. After highlight reconstruction, you can get values above 100%. And if you apply an exposure boost (as the default 0.7ev for example) you get values above 100%.

But if you start with a PNG (even 8bit), you get values between 0% and 100%. Boost that with exposure, and you get values above 100%, no problem.

But… I think a normal photograph often contains values that are between 0% and (give or take) 150% to 200%… and then you have some bright highlights that reach up to 800%. So most of the data sits at the < 200% end, but you can have peaks.

If you start with a nice, perfect gradient ramp, and you scale that up to 1000%, you still have data that doesn’t seem to ‘emulate’ the lightness values you encounter in most photographs.
It can help in showing that happens with the original values, since it’s easier to see.

But all the gamut mapping and hue matching will also respond different to a gray gradient vs real colour data. So if it’s a test to take conclusions from, I’m not so sure.

The whole idea of the Darktable pipeline is that the pipeline can’t clip. So if you load a PNG with values between 0% and 100% and apply 1EV exposure (to get values between 0% and 200%), or if you load an EXR file that already has values between 0% and 200%… no difference.

[jorismak]

Saw the picture, and had to give it a go. Just because I think it’s a very nice picture .

DSCF8284.RAF.xmp (9.5 KB)

First of, enable lens correction because the vignetting is strong in this one .

I white-balanced on the snow. Both legacy-color and modern-color seem to give similar results here.

Raised exposure enough to have the face brightness where I want it, enable filmic and use the white-picker to set it’s auto mode.

Then, directly after that: Time to get the details out, so I enabled ‘local contrast’, switched it to bilateral mode, set the contrast to 3 and cranked the details up to 400%.

Now, I can tweak the filmic white slider again to get a nice mix between details and brightness in the snow.

After that, I went back to local contrast to enable a parametric-mask on it, to only make it work on the highlights (snow). Now I get a nice, bright picture and subject, while still having details in the snow.

From there, it’s pretty much ‘standard workflow’. I enabled color-balance rgb and try the presets. Vibrant works well here, I think. Enable denoise if you think it needs it, enable diffuse to sharpen, and add a touch of ‘local contrast’ in the stock default settings for a small, small amount of ‘punch’

No highlight reconstruction, because I saw nothing being clipped. Maybe I missed a small part somewhere.

[herbert-50]

Yes, I think you are right.

The PNG represents a linear color space because the distances between the individual gray blocks are equal: 0, 28, 56, 85, … always increasing by 28.

It is the same as a RAW, except for the difference that 255 is the end.

So the solution is to use a conversion from linear to non-linear, i.e. Filmic RGB, as with a RAW.

In this case, the Exposure module (and probably all the others that work in linear color space) work in the same way as with a RAW.

Yes, I know the modules don’t make a difference, but I’m referring to what I see on the screen and in the histogram. And when I load a JPG and set +1EV, it looks different than when I do it with a RAW.

[Roger_Kerr]

Great edit, well done.

[herbert-50]

@kofa

Then, increasing exposure by 1 EV will multiply linear data by 2 (check the color picker values in the screenshot):

Thanks for the explanations.

I tried to reproduce it with darktable. I have succeeded so far. The point is to set the color space for the histogram to linear Rec2020RGB.
In this case I get a doubling of the color values ​​at +1EV.

But what absolutely confuses me is that the gradations between the gray blocks are no longer linear: 0 3 10 23 42 …

With sRGB i get , i get 0 28 56 … → linear

Adding 1EV results in

This is not 100% linear and the values ​​were not multiplied by two.

[alpinist]

@jandren

A better approach is to generate openexr images and open them if you are testing things after the demosiacing step. These are true scene-referred files in float. I have generated them both using Python and as renders in Blender.

Are your files available somewhere? If not, would you mind sharing them?

[jandren (Jakob Andrén)]

Yes, they are shared in the sigmoid mega thread!

Maybe a bit hard to find

[priort (Todd Prior)]

openexr-images/TestImages at master · AcademySoftwareFoundation/openexr-images

part of a repo of exr images…

[kofa (István Kovács)]

There is a somewhat confusing interaction between the histogram profile, the display profile and the colour picker. If you want to see linear values, set everything to linear. Setting the display profile to a linear space will produce the wrong screen output, of course, but you can do that if you want to study the algorithms of darktable. Just don’t forget to turn it back to your display’s profile; otherwise your subsequent edits will look wrong to everyone (been there, done that…).

If you open the image in the Gimp, it’ll give you readings:
0, 28, 56, 85, 113, 141, 170, 198, 226, 255.
Those are in non-linear sRGB space. You always have increments of 255/9 = 28 1/3 (so 28 or 29, for every 3rd step).
Set Gimp to use linear light. With 8-bit values (not well suited for linear light, but let’s use that now for the sake of comparison with darktable’s colour picker), the readings are
0, 3, 10, 23, 42, 68, 103, 144, 194, 255.
You can clearly see that the steps are not 1 EV apart (that would mean doubling the value in linear representation).

Darktable’s picker gives you the same (linear) readings (you need to set the histogram profile to a linear space)

Adding 1 EV:

You can see that the numbers doubled, up to 255, which is the maximum. Set your display profile to linear rec 709 or 2020, and the picker becomes unbounded, with values:
0, 6, 20, 46, 84, 136, 205, 288, 388, 510.

[priort (Todd Prior)]

@herbert-50

Another spin on what @Kofa was explaining…

[RFC/WIP] Move overexposed before colorout

Lots of info in this thread as well above and below this post but basically this is what @kofa is demonstrating

Choice of display profile affects histogram, color picker values and overexposure indicators

[jandren (Jakob Andrén)]

Follow up with actual files and pointers.

I can recommend this one from the ACES community, has pretty much every gradient you might think about:
syntheticChart.01.exr (16.9 MB)

I have made some using Blender as well. Both Eevee and Cycles output scene-referred data in rec-709 if you save as .exr
Its possible to construct pretty advanced charts using the “Emission shader” and some other nodes!
Example blend-file:
color slice.blend (667.7 KB)
Example exrs that I have generated:
inside cube.exr (5.5 MB)
glare triangle bloom.exr (21.7 MB)

And finally with Python:
I have used imageio to save numpy arrays as exr:
imageio.imwrite('chart.exr', chart_np_array)

Example exr:
color_wheel.exr (983.8 KB)

Hope it helps!

[alpinist]

Thanks, very interesting indeed. When increasing exposure, I noticed that filmic (color science v6) desaturates green noticeably before red and blue. Sigmoid and filmic v5 don’t show this effect. I assume that this is supposed to be a feature of filmic. On the other hand, I noticed visible Mach bands with filmic v6.

[priort (Todd Prior)]

Are you trying a specific chroma pres mode…I suspect that will impact what you see…

[Donatzsky (Nis Donatzsky)]

@jandren

Uploading: glare triangle bloom.exr…

Looks like one that didn’t get uploaded properly.

[micha]

@heckflosse

My try (though RT, not dt)

With RT, as is so often the case, you quickly get very good results. But, Ingo, tell me, did I just miss the .pp3 or did you forget to upload it? That would be interesting, also for all enthusiastic dt-users.

[darky (Rudy)]

Motivated through the excellent explanations of Boris on www.youtu.be/hFIWUo0Dauc about using two “tone equalizer” modules for highlight contrast improvements, I tried again the photo of the girl in the snow, and the miracle occurs:

20200126-1233_dscf8284.raf.xmp (79.2 KB)

I like the photo, because getting highlight details back is a usual challenge. With this image, there is now no more vignetting! I used only exposure, tone equalizer 1+2, color balance rgb and local contrast. I tried filmic rgb and as alternative sigmoid, but did not use them.

[darky (Rudy)]

On top the contrast equalizer:

20200126-1233_dscf8284.raf.xmp (81.3 KB)

The issue with output sharpening is always, that you have to know about the intended image size respectively viewing distance. Therefor checking the picture with high zoom and sharpening without knowlege of intended use does not make sense.