Magenta highlights vs raw clipping indicator vs filmic white level

[kofa (István Kovács)]

If you check camera parameter databases, you’ll see that sensor saturation levels don’t necessarily match the theoretical maximum (4095 for 12 bits, 16383 for 14 bits, 65535 for 16 bits).

RawTherapee camconst.json:
https://github.com/Beep6581/RawTherapee/blob/dev/rtengine/camconst.json
Look for ‘ranges’:

"ranges": {
            // measured at ISO 100. ISO differences not measured, but known to exist
            "white": [ 16300, 15700, 16300 ], // typical R 16383, G 15778, B 16383
            "white_max": 16383
            // aperture scaling not measured, but known to exist, at f/1.8 the G channels hits white_max
        }

or for ‘levels’:

    { // Quality B
        "make_model": "NIKON COOLPIX A",
        "dcraw_matrix": [ 8198,-2239,-725,-4871,12388,2798,-1043,2050,7181 ], // DNG v13.2
        "ranges": {
            "white": [
                { "iso": [ 100, 125, 160, 200, 250, 320, 400, 500, 640, 800 ], "levels": [ 16300, 15700, 16300 ] }, // typical G1,G2 15760-15800 R,B 16383
                { "iso": [ 1000, 1250, 1600, 2000, 2500, 3200 ], "levels": 16300 }, // typical G1,G2, R,B 16383
                { "iso": [ 4000, 5000, 6400 ], "levels": 16200 }, // typical G1,G2, R,B 16383
                { "iso": [ 8000, 10000, 12800 ], "levels": 16000 }, // typical G1,G2,R,B 16383
                { "iso": [ 16000, 20000, 25600 ], "levels": 15700 } // typical G1,G2, R,B 16383
            ],
            "white_max": 16383
        }
    },

white_max is the theoretical maximum.

I don’t know where the same kind of data for darktable is stored.

[priort (Todd Prior)]

Check this out though…so if I double click the raw white slider in any raw file right now it will go to 3971. If I reset the module it goes to what I can find in the exif data…for that LX-7 file both actions yeild 3971. I feel like this is a coincidence and that it should be 4095 as it is with the other Lumix file…

[priort (Todd Prior)]

@kofa

I don’t know where the same kind of data for darktable is stored.

Coming from the rawspeed directory…

[afre (Alan)]

@kofa

@afre

Although maxes may not always be at 4095, they are usually close for this kind of camera.

Perhaps, I was making a generalization that may not necessarily apply to this model.

@priort Interesting. I see the white="3971"! Unfortunately, there is no note to say why. That is what I appreciate about camconst.

[anon41087856]

I have been comparing C and OpenCL side to side for about an hour, I can’t find what doesn’t match.

The joys and wonders of duplicating everything for 2 codepaths…

[anon41087856]

If more people want to help (reading your own code is kinda reading what you meant instead of what you wrote), we have 2 steps of reconstruction that are equally faulty here:

1. guided laplacian (copy-paste details between channels),
2. diffusion over RGB ratios (propagate “color” from the neighbourhood).

Each iteration interleaves both in a multi-scale setup.

To disable the guided laplacian temporarily:

1. for the C version, in src/iop/highlights.c, replace lines 1298-1299 with:

          high_frequency[c] = (alpha * (a_HF[c] * high_frequency[guiding_channel_HF] + b_HF[c])
                            + alpha_comp) * 0.f + high_frequency[c];

2. for the OpenCL version, in data/kernels/basic.cl, replace lines 878-879 with:

    high_frequency = 0.f * (alpha * (a_HF * ((float *)&high_frequency)[guiding_channel_HF] + b_HF)
                   + alpha_comp) + high_frequency;

To disable the diffusion temporarily:

1. for the C version, in src/iop/highlights.c, replace line 1446 with:

          high_frequency[c] += 0.f * alpha[c] * multipliers_HF[c] * laplacian_HF[c];

2. for the OpenCL version, in data/kernels/basic.cl, replace line 987 with:

    high_frequency += 0.f * alpha * multipliers_HF * laplacian_HF;

If you disable both, then the whole module becomes a no-operation that essentially turns your computer into a heater. The no-op at least is ok on both pathes.

But the mistakes are in those functions, above the lines mentionned here.

The guided laplacian is smoother (or at least closer to what I expect) for the C path. The color diffusion is smoother for the OpenCL path.

[g-man]

Th e 3 or 4 looks different, but I will be honest, CL is not my forte:

highlights.c 1422
const size_t neighbor = 4 * (i_neighbours[ii] + j_neighbours[jj]);

basic.cl 968
neighbour_pixel_HF[3 * ii + jj] = read_imagef(HF, samplerA, (int2)(j_neighbours[ii], i_neighbours[jj]));

[anon41087856]

OpenCL accesses float4 pixels by their (x, y) coordinates directly while C accesses them by their index in the float array ((y * width + x) * 4).

[kofa (István Kovács)]

@kofa

But well, there are many things I’ll never understand, so I can live with that.

So, I’m still stuck here.
I’ve set the output and display profiles to linear Rec2020, believing it’d give me access to the pipeline values via the colour picker.

Here are readings from the clipped Sun disc, and the unclipped sky, with filmic on (default params + autotune white and black):

And here with filmic off:

So, filmic maps the blown area from 4166, 1687, 5304 to 200,  81, 254 (compression ratio ~20.8)
And the non-blown sky          from 1572, 1188,  707 to 239, 183, 111 (compression ratio ~6.5).

It seems the blown highlights are extremely over-compressed; the R and G components end up being mapped to values lower than for the non-blown areas.

I added a masked channel mixer (via color calibration) forcing the blown area to grey (showing the grey = blue channel on the screenshots):

When using the R channel to produce the monochromatic Sun, 4166 is mapped to 252  for all channels.
               G                                           1687              240
               B                                           5305              254

How is it possible that, with unchanged filmic parameters,

the least-bright grey-from-green 1687, 1687, 1687 triplet is mapped to 240, 240, 240, and
the original                     4166, 1687, 5304                      200,  81, 254

despite the original input clearly carrying way more energy? The blown part gets mapped to R and G values that are significantly (R) and much (G) below their grey-from-green counterparts, only B being somewhat higher?

If you want to experiment, don’t forget to set the display profile to linear Rec2020. Here’s my XMP:
2022-05-01_20-04-21_P1070278_08.RW2.xmp (33.1 KB)

Note: I’ve just realised that the colour picker was in max mode, instead of mean. However, since the areas are quite homogeneous, I don’t think this introduced a huge error. I’ll check again tomorrow (the machine is already off, In typing on my phone).

[g-man]

AP, I’m of little help to you at the moment. I don’t know cl enough to help. At worst i would be a distraction.

[priort (Todd Prior)]

@kofa

Here are readings from the clipped Sun disc, and the unclipped sky, with filmic on (default params + autotune white and black):

So Max RGB??

[anon41087856]

@kofa

despite the original input clearly carrying way more energy? The blown part gets mapped to R and G values that are significantly (R) and much (G) below their grey-from-green counterparts, only B being somewhat higher?

WTF ??? The original input has clearly wrong energy compared to the neighbourhood because it is CLIPPED.

So, filmic maps the blown area from 4166, 1687, 5304 to 200, 81, 254 (compression ratio ~20.8)
And the non-blown sky from 1572, 1188, 707 to 239, 183, 111 (compression ratio ~6.5).

Will you please listen ???

First, the color picker picks after display conversion, so it’s not suitable to measure anything > 100%. You need to do all tests with exposure set to avoid clipping if you plan on using it, because there is no traceability on the display output converted back to histogram space.

Second, this is input:

This is filmic’s output with default and auto-tuned white/black:

This is the same with extreme luminance desat all the way:

Relative lightnesses are preserved as they should. Now, because your clipped magenta has a chroma = 93, you bet it’s not going to be treated as white…

So the max RGB norm goes bonkers in this region because the max of each pixel in this neighbourhood are not in sync. You can’t apply max RGB on decorrelated patches because it extracts a single channel, and precisely this what failed here to make a white sun in the first place.

A tone curve is not an highlights reconstruction. Even though the luminance “norm” as well as the euclidean hide it better. The reason is simple: all channels weigh in those norms, so they smoothen the problem by sort-of averaging it.

[paolod (Paolo)]

There is one potential issue in the CPU version of highlights.c - there are a lot of for_each_channel loops with the aligned SIMD directive indicating that some variables are on 64 byte boundaries. But not all of those variables are guaranteed to be aligned that way - the dt_aligned_pixel_t type is only 16 byte alignment.

I don’t think that is actually causing any problem right now though - I don’t see any change in the output if I define DT_NO_SIMD_HINTS to make it ignore those directives.

[kofa (István Kovács)]

Aurélien, thank you very much for putting up with me.

@anon41087856

the color picker picks after display conversion

That’s why I set the display profile to linear Rec2020, and that’s how the > 100% readings were produced. That is why I wrote:

@kofa

believing it’d give me access to the pipeline values via the colour picker

Maybe that assumption was wrong. It seemed to be working, but it was just an assumption, and if you say it was wrong, I accept that.

I have the feeling filmic v6 with maxRGB or power norm is ‘trying too hard’ to keep colours, and sometimes produces flattened highlights (even when there’s no clipping). Making the very bright (clipped, magenta, but still very bright) sun (all channels brighter than the non-clipped sky) darker than the sky is an extreme example of that. I think it may do the same, even if no clipping is involved, if the photo contains bright, coloured light sources, such as traffic lamps or ‘neon’ signs.

What I did not, and do not understand is that the monochrome version derived from the lowest-valued channel (green, for that clipped magenta) is brighter after the filmic mapping than the original RGB:

I’ll just shut up and accept that it is the nature of the saturation-preserving norms. There is no norm optimal for all cases, so I can pick another; and you have given us plenty of tools to deal with colours before we hit filmic.

[RawConvert (Andrew)]

I was wrong about the new version being faster, probably forgetting about cropping.

@kofa , it’s tricky isn’t it. But there is an alternative method for highlight reconstruction - go into gimp and paint what you want!

[kofa (István Kovács)]

I don’t want to reconstruct those highlights – it’s the surface of the Sun, or specular reflections of the water. The luminance Y norm does what I want, and I can also fall back to filmic v5 and its desaturation.

[anon41087856]

@kofa

I have the feeling filmic v6 with maxRGB or power norm is ‘trying too hard’ to keep colours, and sometimes produces flattened highlights (even when there’s no clipping). Making the very bright (clipped, magenta, but still very bright) sun (all channels brighter than the non-clipped sky) darker than the sky is an extreme example of that. I think it may do the same, even if no clipping is involved, if the photo contains bright, coloured light sources, such as traffic lamps or ‘neon’ signs.

  const float norm_min = exp_tonemapping_v2(0.f, data->grey_source, data->black_source, data->dynamic_range);
  const float norm_max = exp_tonemapping_v2(1.f, data->grey_source, data->black_source, data->dynamic_range);

  // Compute the norm using the selected variant
  float norm = get_pixel_norm(i, variant, profile_info, lut, use_work_profile);

  // Save the ratios
  float4 ratios = i / (float4)norm;

  // Norm must be clamped early to the valid input range, otherwise it will be clamped
  // later in log_tonemapping_v2 and the ratios will be then incorrect.
  // This would result in colorful patches darker than their surrounding in places
  // where the raw data is clipped.
  norm = clamp(norm, norm_min, norm_max);

  // Log tonemapping
  norm = log_tonemapping_v2(norm, grey_value, black_exposure, dynamic_range);

  // Filmic S curve on the max RGB
  // Apply the transfer function of the display
  norm = native_powr(clamp(filmic_spline(norm, M1, M2, M3, M4, M5, latitude_min, latitude_max, type),
                           display_black,
                           display_white), output_power);

  // Restore RGB
  float4 o = norm * ratios;

  // Save Ych in Kirk/Filmlight Yrg
  float4 Ych_original = pipe_RGB_to_Ych(i, matrix_in);

  // Get final Ych in Kirk/Filmlight Yrg
  float4 Ych_final = pipe_RGB_to_Ych(o, matrix_in);

  // Test-export to output RGB : check if in gamut
  // retain original hue of the pixel and clip chroma at the gamut boundary 
  o = gamut_mapping(Ych_final, Ych_original, o, matrix_in, matrix_out,
                    export_matrix_in, export_matrix_out,
                    display_black, display_white, saturation, use_output_profile);
  return o;

That’s the whole Filmic v6 code. Notice I tried your picture without the gamut_mapping() function, and it yields the same result. Do you see something “trying hard” to do anything ? We log, we curve, and we clamp to the valid range. That’s pretty much all we do. More importantly, all norms are handled by this exact code, so if one works and not the others, that’s only on that particular norm and not on the general method.

[anon41087856]

@kofa

What I did not, and do not understand is that the monochrome version derived from the lowest-valued channel (green, for that clipped magenta) is brighter after the filmic mapping than the original RGB:

The green in color calibration is not a sensor green, it’s already a mix of sensor R, G, B so it averages clipping issues too. Remember green weighs for about 75% in the luminance value, depending on the color space, and the color calibration “green” is actually luminance (in XYZ space) or something close (in LMS space).

[rvietor]

As I understand it, any clipped highlights have to be dealt with before filmic (and as early as possible). So that means raw white and black points have to be set correctly (automatic), and “highlight recovery” should have done its job. Only wrinkle is the input data for highlight recovery: ideally it would take into account the BW multipliers set just before it. Not sure if that’s possible (or if highlight recovery could be used before the WB module).

@anon41087856

The green in color calibration is not a sensor green, it’s already a mix of sensor R, G, B so it averages clipping issues too.

And that’s because colour calibration comes after demosaic. So each pixel is made up of (interpolated) R, G and B. The image also has been pulled through one or two color space conversions. In other words, before we get to color calibration, a lot has been done already to the pixel values, even without any modules added by the user.

[anon41087856]

Also, this what happens if I don’t multiply the ratios back at the output of filmic (filmic’s defaults with auto-set black/white exposures, so max RGB norme):

Smooth.

Now, look at the ratios:

Non-smooth.

Now, change the norm to luminance:

You have to recover highlights at least to get a color consistent with the neighbourhood. That magenta is simply too opinionated in terms of its chroma, considering how off the hue is to begin with.

The reason v5 hides it better is that the ratios are massaged to degrade progressively to { 1, 1, 1 } when we reach display white. Problem is that this “desaturation” is not hue-linear… (pick your poison).

This how the ratios = RGB / max(RGB) look like after only the filmic HL’s reconstruction:

Note that the reconstruction is not smooth at the edges of the clipped region because it’s a stupid 1st order in-painting, so the guided laplacian essentially inherits the same logic but at the 2nd order, which better respects gradients and output smoother results.