Exposure Fusion and Intel Neo drivers

[Entropy512 (Andy Dodd)]

Carmelo - VERY well written, thanks!

@Carmelo_DrRaw

@anon41087856 @Entropy512
If you allow me, I would like to contribute my 2cts to this discussion, even though I am not at all a color scientist…

Let me begin by speculating on this statement:

@anon41087856

That’s how you show you understand nothing to what you are doing. There are no colours at this place in the pipe. Colours exist only in human brain. Mapping the RGB tristimulus (which encode light emissions, aka spectra reduction to a flat vector) as recorded by the sensor to human-related colours (XYZ space) is the job of the input profile (which still does it awefully bad, but that’s hoz it was done 10 years ago…), and it happens in the pipe after base curve.

RGB values are one possible representation of color, even when they directly represent the light emission values as directly recorded by the camera sensor.

RGB values represent some color. Which color exactly is undefined, unless you associate to them a set of RGB primaries and a white point. If you have that, then an RGB color in camera colorspace is as valid as a colors in Rec.2020. Colors are present at each stage of the pipeline, they are just encoded differently.

Looks good to me.

@anon41087856

Because, again, RGB encode light emissions, so everything you do in a pipeline should use light-transport models, aka numerical simulations of what would have physically happened if you had played with actual light on the real scene, thus scene-linear encoding, proportional to light energy.

Then the only things you are allowed to do are basically exposure compensations and white balance corrections. Any other non-linear adjustment, including filmic curves, do not have a physical counterpart, because for a given scene illumination you cannot physically change the relative strength of your light diffusion in shadows and highlights, right?

Yup. It sounds like Pierre works in the cinema industry, and one thing to keep in mind is that in the majority of cinematic productions, a major part of the production is controlling the actual light present in the scene with modifiers (scrims, reflectors, etc) and additional lighting. This means some of the extreme dynamic range managment tricks such as exposure fusion and one of your approaches at PhotoFlow - new dynamic range compressor tool (experimental) - #36 by afre aren’t nearly as necessary, if at all. (Nice approach by the way, I think my next task is to take a look at DT’s implementation of that approach and figure out why it underperformed - FYI fixing the poor performance of enfuse in highlights that you show as an example in that thread is exactly what I’ve been trying to do. The current implementation performs horribly in highlights)

Ideally you do this even in photography - but sometimes you’re hiking on a trail up multiple flights of stairs and merely putting your camera and tripod in the backpack has already increased your burden significantly. There is no way I’m bringing a monobloc and scrims/reflectors on the trail!

So we’re left with the problem of a scene with a very high dynamic range, a camera that can capture it if you expose to preserve highlights, but stuck with the lowest common denominator of a typical SDR display. So the challenge is to not make things look like crap on such displays, even if what is displayed is now at best a rough approximation of the real scene.

Of note, a lot of these problems go away with a wide-gamut HDR display. As an experiment, I’ve exported a few of the images I find need exposure fusion to linear Rec. 2020 RGB, and then feed this to ffmpeg to generate a video that has the Hybrid Log Gamma (HLG) transfer curve and appropriate metadata. The result looks AMAZING without any need for exposure fusion at all in most cases. Sadly, for still images, we have no good way to deliver content to such displays, even of those displays are getting more common. However, if you ever expect your content to be viewed on a phone or tablet, 99% of them are SDR displays and it’s going to be that way for many years to come. Which happens to be why you’ll have to trust my word that exporting to HLG looks gorgeous on a decent HLG display - there’s simply no way to convey that visually though this forum software to the displays that 99% of readers here have.

Notice also that exposure compensation and WB work equally well on RGB values that are encoded with a pure power function (as already mentioned by @Entropy512), therefore linear encoding is not strictly needed from a mathematical point of view, at least in this case.

Yup, and the math behind this is the identity (ax)^y = a^yx^y as mentioned before.

@Entropy512

In the case of exposure fusion where all exposures are generated from the same input, we’ll always be blending the “same” color.

I totally agree with you, but a safer rephrasing of the above sentence might be we’ll always be blending RGB triplets that have the same channel ratios, because they are all obtained from the same initial RGB triplet by linear scaling (aka exposure compensation). I do not know the internals of DT’s base curve module, but this might only be valid if the base curve is a straight line…

Yeah, that’s a better way of wording it.

Now let me try to give you my point of view on some of the facts an myths about linear vs. “gamma” encodings…

Color shifts
When you apply a tone curve to the RGB values, color shifts are generally not a consequence of whether the RGB values are linear or not. If a non-linear tone curve is applied to the individual RGB channels, color shifts are always there.
The usual “workaround” to avoid this is to apply the tone curve to some RGB norm (like luminance), compute the out/in luminance ratio, and then scale the RGB values by this ratio.
This is where the need of linear RGB values comes into play, because:

• luminance must be computed from linear RGB values
• the RGB scaling by the luminance ratio is equivalent to an exposure scaling only if the RGB values are linear (or encoded with a pure power function, but there is no advantage in doing that in this specific case)

This is, as far as I can tell, the fundamental reason Pierre hates basecurve so much. However, this issue with basecurve was fixed:

base curve: add norms

It happens that it was not fixed for the fusion flow (a result of two code paths getting branched, apparently to eliminate a single multiply operation on the “fast” branch ages ago)

I’ll be submitting a pull request later today that reorganizes some of these code paths such that Edgardo’s changes can be used in combination with fusion.

Side note: Getting to the science vs. art discussion I mentioned previously, in some cases such chromaticity shifts actually look really nice. For one particular sunset example, applying the “sony alpha like” transfer curve in the old per-channel way gives a much more “fiery” look to the clouds. Is it in any way a correct representation of the physical realities of the scene? Not at all. Does it look impressive? Yup. Obviously, this is the sort of thing that should be used with caution and should not be the default behavior (which is indeed the case going forward in DT)

When linear is visually better
One domain in which AFAIK linear encoding is truly needed is pixel blending.
Here is a classical example of blending a pure red (on the left) and a pure cyan (on the right) color at 50% opacity (in the middle). First image is obtained in sRGB encoding, second one in linear:

I guess we all agree that the second one looks “more correct”.

The same happens when downscaling images, because downscaling involves blending neighbouring pixels. Downscaling in linear encoding is better.

Yup, no issues with this. All of the gradient examples I posted were of blends between two pixels that were linear scalings of each other (e.g. channel ratios are constant). If channel ratios aren’t constant, things get funky.

When linear encoding is NOT good
One clear case in which linear encoding is not “good” from the practical point of view is when building luminosity masks. For example, one would expect that a basic luminosity mask has an opacity of 50% where the image roughly corresponds to mid-grey, right? However, mid-grey is 18% in linear encoding…

Yup, and this is a major part of why DT’s current exposure fusion workflow tends to pull everything into the highlights and then crush the highlights. It also pulls quite a bit up past the point at which it’ll clip later in the pipeline.

The relationship with enfuse
Correct me if I am wrong, but in a simplified form and for two images, the exposure blending boils down to something like

(1-W)*x_{1} + W*x_{2}

where x1 and x2 are the the RGB channels of images 1 and 2 respectively. W is a factor that gives more weight to “well exposed” pixels, right? And W is derived starting from some RGB norm, like luminance, right?

Exactly! The equation you give is in the enfuse manual as equation 6.1. Alternatively, the manual gives a second equation (6.3), which is ((1-w)x_{1}^(1/y) + wx_{2}^(1/y))^y (effectively, what I did in darktable’s fusion implementation is to replace equation 6.1 with 6.3, where y = 2.4)

To me, this looks really like blending through luminance masks, which can be split in two parts:

• the determination of the weights W, for which perceptual encoding seems to be more appropriate, since you want your weight function to be peaked at 50% and perceptually symmetric around mid-gray, right?

Yup!

• the blending of RGB values weighted by W, for which linear encoding is more appropriate.

I’m not so sure of that. Let’s take the extreme example of blending black with white, with w = 0.5

So x_{1} = 0, x_{2} = 1.0, w = 0.5

Plug this into your equation (corresponds to equation 6.1 in the enfuse manual), and you get 0.5

Plug this into equation 6.3 with y = 2.4 and you get something around 0.18

So perceptually, the blending in linear space gives a result that is significantly brighter than the perceptual midpoint between the two inputs. You can see this in one of the orange gradients I posted.

Therefore, I wonder if the following approach would yield correct results:

• take linear RGB values as input
• compute the RGB luminance, and encoded it with a power function with exponent \gamma = 1/2.45, so that mid-grey is roughly mapped to 0.5
• use the power-encoded luminance to compute the weights
• blend the linear RGB values with the obtained weights

Would this make sense?

That was, in one of the examples I posted above, described as “weight in gamma-space, blend in linear”. The end result was an image that was very bright and washed out. Better than the current lin/lin approach, but still not visually pleasing. Someone posted an example of applying a power transform to that image to make it look much better, that was the case where I responded that doing so was one of the cases where a chromaticity shift could occur (see the gradient I posted with two different shades of orange).

P.S: this answer has nothing to do with the initial issue with Intel Neo drivers, so do not hesitate to split is to a separate thread if you think it would be better!

I don’t think anyone has talked about that in a while, and I don’t expect any more conversation on that particular topic outside of common/opencl_drivers_blacklist: Only blacklist NEO on Windows by Entropy512 · Pull Request #2797 · darktable-org/darktable · GitHub - yes, I submitted a pull request to un-blacklist Neo on non-Windows platforms since it appears that the root cause of failures on Linux was identified and corrected. OpenCL + NEO is working great on my system.