Pipeline Architectures & Color Science: White Balance vs. Demosaicing in darktable and ART

I have long been wondering what role do lenses (and lens coatings) play in all of this? Where does the distinction between a magenta-coated and a green-coated lens get processed if all we have is the sensor data. Maybe I’m misunderstanding something

@Thomas_Link AI is great, but when it turns human writing into its own slop, it’s awful. These translations are worse from the prose side of things, but they’re chaotic enough to appear human and therefore read much better, like talking to a real person instead of a text completion engine

2 Likes

Thanks! I’m still going to take the other users’ criticism to heart and stop using AI for translations. In most cases, I can’t quite get by without a translator, but I’ll use DeepL, which translates more literally and thus stays closer to my original—even though I agree that an AI translation better respects the context and might be easier to read.

1 Like

Just three short comments - confessing i don’t understand why this is an interesting issue at all -

  1. There is a bunch of demosaicers provided in dt, art and rt; they make different assumptions about it’s input and many of them just need some simple white-balance (basic coeffs) to work reliably. Some don’t. Some even do some “internal gamma” correction.
  2. If you want to compare the “pixelpipes” - there is just one way - read source :slight_smile:
  3. There are also more algorithms involved in raw-space, they also make some “assumptions”. And these “assumptions” - there was another thread about “calc towards white” - clearly influence quality of results.
3 Likes

I would take a read here just for some context if you have not… this is Ansel but most of the nuts and bolts wrt are similar enough and the rest is information about wb and illuminant…

https://ansel.photos/en/resources/white-balances/

The way I look at it your photos are always going to have an illuminant that it the mixture of all the light being collected and reflected in the scene. DT takes the appoach of handling that with a perceptual CAT to convert the scene illuminant to an appropriate display illuminant so that scene neutral ~ display neutral…

No one icc is going to do much across a batch of images of blended reflected light from changing or random lighting conditions at the scene… So you wb to that neutral reference (the one with the lowest a and b) under daylight or maybe a couple of other general conditions and then you can use those profiles with some latitude.

A DCP profile which ART supports has dual illuminants and uses interpolation between them to get around some of this…maybe that might be a better solution.

This is an oversimplification and maybe someone will be able to do a better job to address your question

6 Likes

That’s a really great summary.

Lenses do have their contribution to shaping the light. In my home-brew approach to camera calibration, I’ve chosen to dismiss that as insignificant, as I do believe all lens manufacturers strive for ‘achromatic’ in their glass formulations, that is, uniform pass of all the wavelengths of the visible spectrum. If they didn’t do this, they wouldn’t be ‘general purpose’. Your Mileage May Vary.

My camera spectral sensitivity measurement technique requires a lens on the camera to resolve the spectrum. The lab-grade method uses a monochromator shining directly on the sensor, no lens in the way. The profiles I get from my data are within 1 dE max error of lab-grade data for the same camera, so I’m not worrying the lens contribution.

If you want to know the ‘color bias’ of a lens, you need to measure its spectral bandpass. That’s not really hard, you just shine a full-spectrum light source through the lens and measure what comes out the other end with a spectrophotometer. You also need to measure the light source direct, that way you can dial-out its biases. Add back those light-biases to the measurements through the lens, and you get a spectral attenuation distribution for that lens.

Geesh, you’ve given me yet another reason to regret not purchasing an XRite iStudio1 (a spectrophotometer that used to cost about $480US) before Calibrite took over and upped the price…

3 Likes

The only way you’ll get a camera profile that even approximately corresponds to the light at the scene is to shoot a target shot at the scene and make a profile, for that scene. I rather like that approach, but it’s really impractical unless you’re doing color reproduction work, photographing art work and such.

@priort 's recommendation to use dual-illuminant DCPs is a pretty good idea if you want to get close. Anders Torger wrote a great missive on camera profiling for the dcamprof software, in the appendix is a section on how DCPs and the surrounding Adobe workflow work:

https://rawtherapee.com/mirror/dcamprof/camera-profiling.html

RawTherappee and ART are the only FOSS alternatives I know that support the Adobe DCP workflow.

But, keep in mind what Todd said: “The way I look at it your photos are always going to have an illuminant that it the mixture of all the light being collected and reflected in the scene.” Unless you’re in a studio with lighting you control you’re going to get a mishmash of color temperatures depending on what part of the scene you’re looking at. So, my take is, if it’s predominantly daylight, just use the D65 camera profile and pull a cup of tea… :laughing:

3 Likes

Fun reasons to dig deeper! Sorry for the AI worries, there seems to be something about LLM translators that sound very chat bot like to me. Wonder why.

I have started doing some experiments on this front as well and shared in the spektrafilm tech thread. What I learned so far is that we have a high dimensional light spectrum sampled as three dimensional RGB intensities that we then want to display or print. There is thus a huge “loss” of data just by taking a picture. Its this loss that results in the situation that a profile is only really valid for the same illumination as the target it shot under. And to make it even worse, I expect that profile to only be valid for colors inside of the convex hull shaped by those samples… I think its safe to say that any error from the demosaicing will be insignificant compared to all those other errors.

I have some hope to make gains for extrapolated colors by a spectral approach but its all very new to me so far. Would be fun if you join down in that rabbit hole!

3 Likes

That’s an interesting comment. What aspects of light are you thinking of which go beyond representation by three dimensions?

1 Like

You asked @jandren, I’ll pipe in because I have examples…

A particular color in a scene illuminated by a full-spectrum light has plenty more components than just three numbers. Here’s a plot of the relative energies across the visible spectrum for the two ‘skin tone’ patches of the ColorChecker, A01 and A02:

Even colors that align with particular wavelengths have more components, owing to the imperfection of reflective surfaces. You need to get some red, green, and blue LEDs (you can get LEDs for every part of the spectrum, in IIRC 5nm increments) to see real spectral colors.

Anyway, the RGB encoding for each of those patches (sRGB, for viewing) are:

A01: 213, 149, 137
A02: 121 78 67

That’s the ‘loss’ @jandren is referring to. Our entire system of digital photography imaging is based on this abbreviated encoding of rich scene colors, which depends on human metamerism to work.

Note: I’m not a licensed color scientist, more a ‘color mechanic’, so YMMV…

5 Likes

Thanks, Glen,

I think a tiny lightbulb just turned on for me.

A small, broad-spectrum incandescent bulb at 5k, that is. :wink:

1 Like

Heh, like this:

I’ve got a million of 'em… :crazy_face:

1 Like

Thanks for the reply Glenn but I’m not really clearer. I see that there is a whole range of frequencies in a colour, and that the energy varies. Presumably the high blue energy is simply due to the higher frequency. But I’m not understanding what in practical terms is missing or wrong by having just three RGB values. So the question is, what data would have to be added to the camera’s capture system to get a better representation of the scene?

1 Like

Two materials or lights can be entirely different in there spectrum but register exactly the same values in the camera. This is what metamers mean.
But they will likely look different for another camera that has a different spectral sensitivity.

The crux begin when you try map and white balance a camera measurement two a new space like XYZ which is based on the human vision spectral sensitivity. There is no simple linear map between those two due to the loss of higher dimensional data.

Lets make a analogi with sound. We have a microphone that can record three loudness levels using three distinct frequency sensitivity functions (measure the envelope of band pass filters). Now try to reconstruct the original sound using only those measurements! Its easy to see how impossible that is, we basically only know how much bas, midtones and treble the original sound had but nothing more, pretty useless for sound!

It turns out we see levels of bas, midtones, and treble and thats why it works to only record them but there is nonetheless a loss of data and that loss makes our white balance and profiling methods imperfect.

Take a look at what hyperspectral cameras are able to do by not discarding data!

5 Likes

More channels.

In the Bayer mosaic evolution, there were attempts to do that:

https://en.wikipedia.org/wiki/RGBE_filter

But to play in the RGB system, those four channels still have to be resolved to three.

Thing is, the so-called red, green and blue channels we currently use in digital imaging have alignment with the trichromatic human system of vision, so making decent renditions with three channels works, mostly.

Where it can fall apart is if you’re looking for reproduction of a very specific hue, and the available metameric equivalents aren’t so close. Most of us aren’t working in that world…,

Oh, and ‘blue energy’ is lower frequency, ~450nm and below.

I agree with @jandren 100%. But perhaps we can clarify this a bit further by looking at the math.

Before light is detected by the camera/eye, the light source strikes an object and is re-emitted by it. The spectral signal S(λ) that ultimately reaches the eye/sensor is the product of the light source (illuminant) and the re-emission coefficient of the object’s surface:
S ( λ) = I ( λ)⋅R ( λ)
• I(λ): The spectrum of the light source (illuminant); contains the complete physical properties of the light.
• R(λ): The spectral reflectance curve of the object (what percentage of which wavelengths are reflected); contains the complete physical properties of the object we are photographing.

The camera measures this incoming signal S(λ) through its three color-channel filters (red, green, blue). Since the sensor captures all wavelengths simultaneously and sums them up, the result is:


• CR(λ), CG(λ), CB(λ): The camera’s spectral sensitivity functions.
• R, G, B: The final three pure numerical values after demosaicing stored in the RAW pixel.
Ultimately, the eye is also just a sensor and behaves in exactly the same way, except that its spectral sensitivity curves (defined by the CIE as early as 1931, long before digital cameras or computers even existed) are completely different from those of the camera. The same applies here:

• x(λ), y(λ), z(λ): The virtual cone sensitivities of the CIE standard human observer.
• N: A normalization factor

The problem (camera metamerism) is as follows:
There can be exactly two surfaces that, under the same light source, produce identical RGB values for the camera but appear completely different to the human eye (XYZ). To your camera, these two objects are indistinguishable at that moment—it sees one and the same color, while you, as a human, perceive a clear difference in color. This happens particularly often with artificial dyes, textiles (e.g., highlighters or dyed fabrics), and certain car paints. (The same applies, of course, in reverse.)

So if you knew the spectral signal S(λ), you could use it to unambiguously calculate both the RGB values and the XYZ values. However, since a consumer camera (I think I once read that there’s a Hasselblad that actually records the spectral signal directly) can only reconstruct the RGB values, it’s not possible to calculate the spectral signal from them.
That’s why, if you’re mathematically precise, there’s never an exact mapping of camera RGB values to XYZ (or other Rec. 2020) color spaces. Not even—and this, it seems to me, is often overlooked—if you had a profile tailored exactly to the current lighting conditions. It’s not the light source alone that matters, but rather the product of the light source and the re-emission coefficients.

Translated with DeepL.com (free version)

4 Likes

Add on:
Here is a link to an article where I have the most informations from:
Dietrich Zawischa:

1 Like

Very interesting stuff indeed. Yet - this has all gone very far from the title of this thread …

1 Like

I think I’m getting the idea, let’s see… The limitations of a camera sensor (chip construction, filters) mean that some wavelengths register more strongly than others. Therefore the resulting RGB triplets are not based exactly on the light received, but a “weighted” version of it.

The eye works differently and has its own “weighting” which is different to the camera. So the eye and camera are not seeing the same colour.

We know the spectral response of the camera and eye, so why can’t we just convert camera to eye? This would be possible if we had the spectral details of the light falling on each pixel, but we don’t. I think I see how reducing the light to three values is a many-to-one mapping, because of the “averaging” across a range of wavelengths that takes place by the camera at each pixel. Differing light can give the same triplet. So it’s not possible to get back to the original signal.

I asked what extra data would need to be recorded. The hyperspectral camera records the full spectral signal for each pixel, 100+ values, but is prohibitively expensive and unsuitable for general photography.

@jandren , you say “valid for colors inside of the convex hull shaped by those samples”. Is it worse than that, in that even the “nice” colours may appear on surfaces that reflect with an awkward spectral profile for a given camera?

1 Like

This is quite an advanced topic for a near-beginner like me, but here is my take:

As noted above:

The camera records signals from photo receptors the are underneath red, green, or blue filters, usually in a Bayer or X-trans layout.

The raw sensor data discards a substantial amount of information from the spectral field. So its representation of the scene is an approximation, which may be an important consideration for certain scientific applications.

This information is taken into the image processing pipeline there are some preliminary steps including a white balance and demosaic step, translation into a wide color space, followed (at least in dt) by chromatic adaptation, a tonemapper (in most cases), and export to either a display rgb or printer CMYK. Maybe I skipped a couple of steps here…

Then the resulting image is viewed by the eye, which has its own tristimulus:


Source: Wikipedia

So clearly the eye also discards spectral information.

I know little about neuroscience, but just looking at the curves, the brain must be involved interpreting color. So I doubt that a mapping of camera rgb to LMS would be very useful. Just look at the overlap between L and M cones. I doubt the perception is described by a Cartesian (L,M.S) space. Perhaps that is why x, y, and z were established. ? But the latter is an abstraction.

So, how much does all of this matter to general photography?

I have the impression is that the answer should be “not so much.” Confusing different colors due to metamerism may come up now and then, but we wouldn’t have color photography if this were a huge issue (again for general photography). White balance can be a challenge; but modern CAT is powerful.

And yes, camera profiles are important, but at some level we have information for most cameras that is sufficient for general purposes. And if you are fussy there are color checkers for correction.

Now if we could all move to better indoor lighting, I think white balance would be greatly simplified.

I hope I have things right. I’ve been wondering about such things for a while.