Another tutorial – Color appearance – Truck under a tunnel

I offer you another tutorial to try the branch « lacam16n » and PR 6861
https://github.com/Beep6581/RawTherapee/pull/6861

https://rawpedia.rawtherapee.com/Local_Adjustments#Cam16_with_HDR_pre-processing.Development_branch%E2%80%9Clacam16n%E2%80%9D

The aim of this tutorial is not to demonstrate the ‘best’ way of processing this image, but to show that by integrating a pre-processing module upstream of Ciecam, you can process images with a very high dynamic range so that they can then be easily processed by Ciecam (Cam16).

Here I chose in Source Data Adjustments to only use Tone Response Curve – TRC.
image

The 2 NEF and pp3 files can be downloaded on the first page of the tutorial.
https://rawpedia.rawtherapee.com/Local_Adjustments#A_second_example_-Using_the_TRC-_Truck_under_a_tunnel

If you wish you can make comparisons with other RT tools, or other software such as “Sigmoid tone mapper”

Executables (lacam16n)
https://github.com/Beep6581/RawTherapee/releases/tag/pre-dev-github-actions

Jacques

2 Likes

I would like to thank @nosle for his perseverance and forcing me to look into why there was in certain cases a bad behavior of LA with the highlights - either by the nature of the image, or for adjustments.

The PR is modified and of course the executables (lacam16n).
https://github.com/Beep6581/RawTherapee/releases/tag/pre-dev-github-actions

I also modified the 2 attached pp3 files in Rawpedia.

This modification only changes the default value in “Setting” / “Specific cases” / “Avoid Color Shift”, now to “XYZ absolute”.

Another piece of information. I think you saw that in “Cam16 Image Adjustments” the presence of a Sigmoid Q module. This module is not intended to be high dynamic image processing, but a complement in the processing chain, perhaps easier to use than a curve… It’s more of a personal challenge.

jacques

3 Likes

I added a Checkbutton “Smooth highlights” in Source Data Adjustments to slightly lower the luminance when it is close to 100 (Lab).

Completes the processing carried out by gamma, slope and midtones by causing a slight lowering of highlights. Please note this does not replace Highlight reconstruction.

5 Likes

An example of using “Tone Response Curve & Midtones” in LA Cam16 - “Source Data Adjustments”

Note.
No use of tools such:

  • Log encoding
  • Sigmoid tone mapping
  • Dynamic Range Compression
  • Or other Tone Mapping tools

Only :

  • gamma with a high value to recover highlights
  • slope with a high value to recover shadows
    We connect a linear part (slope) and a parabolic part (gamma) without any discontinuity.
  • Smooth highlights restores Dynamic Range beyond L=70

All the Dynamic Range of the image is restored, approximately 21Ev
mid.tif (3.4 MB)

And the pp3
mid.tif-trc.pp3 (19.9 KB)

I think it’s simple and quite intuitive.

Jacques

5 Likes

thank you @jdc. I’ve tried this on several images that have always caused problems and so far it seems to be the simplest and most effective way of dealing with high dynamic range compared to the other possibilities that you outlined above. It preserves the colorimetry and keeps good detail in the highlights. The slope slider seems to be quite sensitive at the lower end of the range and finding the right combination of slope and gamma to preserve the contrast can be a bit tricky but these are minor issues.

3 Likes

@Wayne_Sutton

Thank you for this positive review. I take your remark into account.

It is essentially a mathematical problem which has no simple solution: the connection of a straight line and a parabolic curve of degree n (n = real = gamma)

But nothing is perfect :wink:

Jacques

I agree 100% with @Wayne_Sutton on this tool: @jdc is creating an excellent tool.

Maybe it’s just me, but I do have an odd problem with the Truck Under a Bridge sample image. If I start with a Neutral profile, add a full-frame spot to a very dark location, then apply an appropriate TRC gamma and slope, it all works well. This is at zoom level “Fit whole image to screen”. If I then zoom in to 100% or beyond, the image loses the LA adjustment entirely on-screen, but the LA adjustments remain in place in the tool. If I export while zoomed in, RT produces an output file that also lacks the LA adjustment. Strangely, I do not see this same behaviour when I use the Alexandre III Bridge image or the latest 21-EV exposure strip.

RT Version: 5.9-731-g603d02106 on Windows 10

Terry

@tbransco

I can reconstruct what you describe with this image.

This is a curious phenomenon, because with similar images in terms of DR (13 to 14 Ev, or more) and deep blacks, everything goes well.

I’ll have to investigate, it’s intriguing

Thank you

Jacques

@tbransco

I think I found and fixed the problem in this case of the black screen (I just push a change), with 100% zoom and TIF/JPG output.

For rare images there may be RGB values equal to 0. in this case the TRC/primaries algorithm cannot perform certain operations.

This is little or not at all effective in the preview, because we take one pixel out of 10. So little risk.
I added a control which makes the minimum value taken into account

I replace 0 by 0.000001 - Normal interval: 0, 65535.

Of course, to test, to see if there are any side effects.

Jacques

1 Like

this seems to have fixed the problem with the grey-scale test ramp also.

1 Like

@Wayne_Sutton

I change the GUI for slider “Slope” …with a “Log” scale.
It changes the appearance and ease of use, but not the math behind it.

Thank you for testing.

Jacques

1 Like

Works like a charm now, @jdc.

1 Like

With the latest compilable version or with the executables (lacam16n), I made some modifications
https://github.com/Beep6581/RawTherapee/releases/tag/pre-dev-github-actions

  • simplification of the GUI interface in “Basic” complexity mode, the primaries have disappeared

image

I completed the basic primaries available (Complexity mode : Standard, Advanced) - they are now similar to the “Working profiles” choice:

  • Prophoto (default)
  • Beta RGB (near Pointer’s gamut) - corresponding to the reflected colors
  • WideGamut
  • ACes p1
  • Rec2020
  • AdobeRGB
  • sRGB
  • JDCMax
  • JDCMax StdA
  • ACES P0 - for storing datas
  • Best RGB
  • Bruce RGB

And of course : free

You can have an overview of these profiles with the CIExy diagram incorporated into the primary module
I attach as an example JDCMax, which encompasses pointer’s gamut:
image

Jacques

5 Likes

Hello

I added a “Graduated Filter” (on luminance)

Only available in “Standard” or “Advanced” mode.
Disabled in “Basic” mode (default).

This makes this module more autonomous and makes it possible to improve images which have large differences in luminance in their geographical distribution.

image

Executables (lacam16n)
https://github.com/Beep6581/RawTherapee/releases/tag/pre-dev-github-actions

2 Likes

Hello

I added some illuminants to make them identical to those of “Abstract profile (main - Color tab)” (and vice versa) - Illuminants : D41, D50, D55, D60, D65, D80, D120, stdA, Tungsten 2000K, Tungsten 1500K, E.

For the sake of consistency, I added the “shift x” and “shift y” sliders to “Abstract profile” to act on the dominant color.

Jacques

2 Likes

Hello (Excuse my bad english).

I continued the harmonization between “LA cam16” and “Abstract profile”.
After making it possible for illuminants and primaries to have an identical selection, I added to “Abstract profile - Color Tab”, 2 components present in “LA cam16”:

  • “Midtones”: adds midtone control to the TRC - Tone Response Curve;
  • “Smooth highlights”: gives you the option of lowering (softening) highlights. Note that this has nothing to do with the Exposure Tab - Highlight reconstruction (Inpaint Opposed, Color Propagation, etc.), which attempts to “reconstruct” colors that have disappeared on the sensor. The idea here is to lower highlights when they are high, to make the image more legible.

The (overall) differences between “LA cam16” and “Abstract Profiles”:

  • very little in terms of algorithms: the “Source data adjustments (Local tab)” and “Abstract profiles (Color Tab)” modules are very similar.
  • their position in the pipeline is different:
    ** LA cam16" is almost at the beginning of the process, just behind “White balance”.
    ** Abstract profile" is at the end of the process, just before “Color Appearance & Lighting”.
  • their integration with Ciecam is different:
    ** integrated into the same tool for “LA Cam16” with simpler tools in Ciecam.
    ** separated between “Abstract profile” and “Ciecam”, with Ciecam’s many (too many) possibilities.
  • but what they have in common is that the starting point is after the White-balance (I recommend in most cases “Temperature correlation”).

There’s nothing to stop you using both … :wink:

“La Cam 16” and “Abstract profile + Ciecam”: an alternative to other tools such as “Log encoding” or similar, “Sigmoid”, etc. :

Why do I say this?

  • It is necessary both to be able to “reduce” the apparent DR (dynamic range), and to maintain a colorimetry that is not too degraded.
  • In many writings, it is said that a linear approach should be favored as much as possible:
    ** Logarithmic coding (unless mathematics has evolved?) or Sigmoid (which uses exponentials) are anything but linear transformations. I’m not saying they don’t work (you’ll find both in “LA Cam16”), but they do induce transformations in contrast and colorimetry.
    ** The TRC (Tone Response Curve), one of the 3 parts of the most common ICC profiles, contains a linear part (slope) and a parabolic part (gamma). The linear part brightens shadows, while the parabolic part simulates human vision. Almost all ICC profiles come with the 2 usual TRCs, sRGB (slope= 12.93 gamma=2.4) and BT709 (slope=4.5 gamma=2.22). Try it and you’ll see the difference.
    ** I’ve modified this TRC to make it suitable for processing images with (fairly) high dynamic range, especially those with blocked shadows. “Slope” can reach very high values (300 or more) and “gamma” high values (15 …). This enables absolutely linear retouching of shadows, and progressive retouching of highlights. This does not mean that the noise will not rise, or any other malfunction.
    ** As nothing is perfect, I added 2 components: a) one to act on the midtones; b) one to soften the highlights.
    ** Of course, this approach is not without its difficulties. Images with (very) high dynamic range (DR > 13) or highly differentiated luminances can be poorly processed (as can Log or Sigmoid coding). Nevertheless, this TRC combined with Ciecam is capable of rendering DRs in excess of 20 Ev.
    ** The possibility of using Ciecam, which simulates (at best) human vision at the phisiological level, is a more than interesting option.
    ** But getting down to basics with 4 simple settings solves most of the problems of difficult images.

What is Abstract Profile?
When I initiated this module, in late 2020, early 2021, there was a lot of criticism (to put it mildly). I was asked or told: 1) to withdraw this module (isolated request); 2) what’s this bullshit; 3) and so on.

The basic idea is to apply a virtual ICC profile to the data inside the pipeline. You know the input profiles (Input - ICC or DCP) to calibrate the camera. You know the output profiles for driving screens/printers. “Abstract profile” is a virtual profile that applies to data and modifies neither the “Working profile” nor the output profiles.

It is made up of 3 components:

  • the TRC, which takes into account human vision for luminance restitution.
  • the illuminant (D50, Tungsten, etc.) to take account of shooting conditions (sun, shade, etc.)
  • primaries (AdobeRGB, sRGB, Prophoto, Rec2020, JDCmax, etc.) to select the range of colors taken into account (gamut).

An article on this subject:
https://www.johnpaulcaponigro.com/blog/17781/using-color-management-color-adjustment-synthetic-profiles/

You can act on these 3 components.

In the basic version, you have access only to the luminance components: “gamma”, “slope”, then “midtones” and “smooth highlights”.

You can act on the primaries. The difference with other approaches is that, here, you can modify them directly, modifying saturation (distance between the white point and one of the 3 vertices of the CIExy diagram) or hue by moving one of the vertices perpendicularly (or not) to the saturation vertex. The primaries (unless I’m mistaken) are the standardized ones shown on the CIExy diagram.

I’m not going to repeat the presentation of this concept and the tools described in Rawpedia - even if, obviously, the latest modifications are incomplete, and tutorials are pending.
https://rawpedia.rawtherapee.com/Color_Management#Abstract_Profiles

Yes, but Ciecam is what, and it’s complex.
I’m referencing an answer I gave in “Is Rawtherapee complex?”
https://discuss.pixls.us/t/is-rawtherapee-complex/42183/39

Try LAcam16 and/or Abstract Profile (with or without Ciecam) and you’ll see.
Executables :
https://github.com/Beep6581/RawTherapee/releases/tag/pre-dev-github-actions
This PR is quite voluminous in terms of modifications, but I don’t see any other way of doing it. Cutting it into several pieces wouldn’t make sense. Of course, it will complicate the review (@Lawrence37 , …)

Jacques

3 Likes

Hello
@nosle @jonathanBieler @Lawrence37 @Wayne_Sutton

I did a lot of research into why, in certain cases, the system created artifacts in the highlights and elsewhere with a remedy in “Settings” - Avoid color shift = “Munsell” instead of “XYZ absolute” and vice versa.

After checking the algos of Munsell, Gamut Lab, Gamut XYZ… nothing all is fine.

I looked at the code (it’s at least 5 years old, or more…) and in one or two places I “Clip” the Highlights and “a” and “b” (chroma)… That’s what was causing the mess.

So I kept the code, but “Clip L”, no longer does anything, and Clip C (for “a” and “b”) instead of cutting -40000 and +40000 (the normal absolute limit is 32768), now cuts to -100000 and +100000 (huge values… to avoid too much imaginary colors).

So I removed “XYZ absolute” from Default Settings and (re)set “Munsell only”… and everything is fine.
On the other hand, for Settings to take effect, you must “Reset” to Neutral… and of course not reload an “old” pp3…

To check, if everything works, especially with Cam16.

Executables (lacam16n)
https://github.com/Beep6581/RawTherapee/releases/tag/pre-dev-github-actions

Thank you

Jacques

1 Like

Great that you found the issue. It always “felt” like it was unnecessarily clipping to me. Will check it out!

1 Like

@jdc wow, this function now looks great, thank you very much. Cam16 is working fine for me.
btw: the auto tone-curve in exposure tab is not working in your version

@marter
Thank you for this test, and this validation of the modification (it had to be found :wink:)

Thanks for the overall assessment of this branch, which seems to work well. Of course to be confirmed by other evaluations.

Can other people:

  • validate this last modification: behavior in highlights associated with Avoid color shift : "XYZ absolute (default) ", which I replace with “Munsell only”
  • evaluate the real novelty which is the “Source Data Adjustments” module, in particular for its capacity and the ease (or not) of processing difficult images (high DR, very blocked shadows), or evaluate the “Abstract profile (Color Tab)” module which presents the same algorithm.

Thank you.

You say “the auto tone-curve in exposure tab is not working”… I think you speak of “Auto-Matched Tone Curve” in Exposure. ?
I haven’t changed anything in this part of the code.
I checked and compared the latest “dev” and “lacam16n” (compiled versions) on a CR2 file and a NEF. In both cases: 1) neutral; 2) pressed “Auto-Matched Tone Curve”.

Preview seem identical, and I made 2 x 2 TIF outputs compared in Photoshop (differences): results, they are identical. But perhaps something else ?

Thank you again. :grinning:

Jacques

1 Like