Thanks for this clear explanation. I’m learning A LOT every time you post something.
So true. We’ve been all in this position.
4 Likes
First off, thanks for that post! Much needed and very very helpful.
Two immediate things:
- Should one explain what an opponent color-space/representaion is and how rgb-primaries form the triangle in the CIE horseshoe, but chroma could technically lie outside of that triangle (whether this makes sense for display purposes or not would be gamut-clipping or gamut-remapping)? I have the feeling you left that out for clarity.
- this
Probably needs more discussion. Maybe not here though?! Because it
a) touches a spectral region which is for many color-models still iffy to do certain transform aspects in. This probably affects gamut-compression/remapping ‘performance’ or ‘quality’. And
b) it critically depends on what you call UV (380nm is well on the CIE-horseshoe, looks purple in real life as the horseshoe suggests and should definitely not be leaking out of a blue LED. Blue LEDs at the moment have at worst a 405nm center wavelength, not lower, usually they are very very bright narrowband 450nm lightsources. 380-450nm is exactly the spectral region which is…uhm…interestingly handled in many color-models). UV sources below 380nm tend to look very faint and pale blue to the eye, almost white and you should never ever look with your eye into it! I would be a bit surprised if 380nm LEDs are being sold at the moment…but their color should be represented as a purple hue and not nuclear-blue.
On ccd or cmos sensors, if the lens-glass is transmissible for such UV-wavelengths below 380nm (not every glass variant and not every anti-reflex coating is) it can have all sorts of contamination colors, not necessarily just blue. Just like IR-contamination will sometimes ‘only’ have weird colorshifts.
Also, I had to laugh at rat-piss-yellow! 
Thanks again Aurelien!
Cheers
For all we care, chroma is the distance between a color and the achromatic color (“white”) having the same luminance. The max chroma that a color space can handle depends on the hue, the luminance and the color space primaries. Bringing the opponent representation is a bit off-topic for something I wrote mostly to reassure users rather than to be a class about color management for devs.
Don’t forget that sensor RGB is decoded to XYZ using an input profile 3×3 matrix that is a least-square fit of low-saturation samples. As you see on the graph above for the Nikon D810, the fitted coeffs send values to the UV zone, whether the sensor recorded them there or not. Also, sensor metamerism doesn’t match human metamerism (since the spectrum → tristimulus split is not a bijection in both cases, so 2 spectra can produce the same tristimulus on some sensor and not on others). So, in any case, shit gonna happen in the blue-indigo region and, given the circumstances, the highest priority is to get smooth gradients that do not cross several hues rather than aiming at accurate color matching.
Given the first quote, what is the reasoning for darktable not applying black point compensation?
Also, while the gamut compression in Color Calibration is brilliant for smooth gradients in the master/working profile, are there any plans to add gamut compression for the output profile?
Amazing work, once again.
Hello Aurélien,
Thanks for the detailed article. Since I was the guy who triggered it (at least, I had part of it and feel addressed by some of the remarks), please let me say that I was never a ‘gamut-alert freak’ – I just watched your video on the colour calibration module, where you mentioned it, and used it, and I noticed that the settings were not the default ones; that was why I asked.
In what space to show histograms ?
Any. It doesn’t matter. What we look for, when looking at histograms, is the spread. […]
But the truth is, if you wonder in what space to show an histogram, you most likely don’t know how to read it, so don’t bother at all, hide it and look at the picture instead.
I dispute the ‘any’ argument, since the screenshots I posted using your suggested spaces (PQ Rec2020, HGL Rec2020 and sRGB) are dramatically different.
Or maybe I don’t understand what you mean by ‘spread’.
And a personal note: I admit that you and I are not playing in the same league when it comes to colour science and image processing, but please understand that phrasing things like ‘if you wonder in what space to show an histogram, you most likely don’t know how to read it, so don’t bother at all’ sounds condescending and offensive to me. 
2 Likes
Hi @anon41087856, thanks for the explanation, I appreciate it!
When I find RGB values below zero, does that mean that the initial conversion from camera space to working space failed?
How would you handle negative rgb values in darktable?
The black point is read from the dark pixels, a special area of the sensor that doesn’t catch light, and extracted by Rawspeed. You will find it in the raw black/white point module. Sometimes, it’s too aggressive.
You can fix that either in raw black/white point module or in exposure module, with the black level. Basically, you add some constant until your whole range gets positive.
However, the medium black threshold I’m talking about is not zero, but much higher. So even with no negative RGB in the pipeline, if you don’t have black point compensation on, you will still have black clipping on the print.
No reasoning, it’s just not implemented. There is a comment in the code saying it should maybe be exposed to user interface. Also, it’s enabled without telling you if you enable softproofing.
There is some gamut “safety-jacket” in filmic, but then exports are handled through Little CMS2, which does the perceptual thing to gamut map. So, no manual compression.
Well, sorry, but at some point you need to start at the beginning… What is an histogram for ? What are you trying to see in there ? Once these questions are answered, the choice of space is pretty much done. The point is not to be condescending but just to remind users to disregard scopes they don’t understand because they don’t need them. Retouch the picture, and maybe if you see something wrong in the picture, then scopes can help you troubleshoot the issue and spot the origin of the misbehave. But as long as picture looks good, don’t make up problems.
The only use for an histogram is to see how the image fits in the available range, between medium black and medium white. So you only care about how much space you get on the left and on the right, if there is any space at all, or look for a peak at min/max intensities that would suggest clipping.
Any space will do that. Now, sRGB, because of the gamma/power 2.4 will put middle-grey in the middle of the histogram, if that’s what you want, but will not have an uniform spread because it has actuall 2 gammas (one for midtones + highlights, one for dark tones). The PQ curve from Rec 2020 PQ dilates shadows a lot and compresses highlights, so it gives you better legibility near black and worse near whites. But it’s still the same span, only zoomed differently. The HLG curve is more balanced.
Whatever space you choose doesn’t change the meaning of the histogram span : you have some range between white and black, you want to see how the pictures fits in that range, period. The shape of the histogram will change with the OETF of the space you choose, but the shape holds no meaning at all, so it does not matter.
3 Likes
It’s a long one, sorry.
It wasn’t what you said, it was how you said it. But I understand that you and the other knowledgeable people on the forum get lots of questions; some repeated, and surely most way below the level that’s interesting to you. Also, I sometimes fail to remain patient when people ask me or make stupid mistakes, so I can understand your side of the situation, too.
Anyway, I am grateful for your work: both the development and the education (I have also taught software engineers, and know how much time and energy go into even the simplest demos, let away video tutorials).
I think I do understand what a histogram is for; but, like some others, for a long time I simply edited JPGs in the Gimp and similar tools, where I never had to think about colour spaces. Even in RawTherapee (which was my first raw developer, back in the days of Gábor Horváth, who started the project), you could not (at that time; not sure about now) switch the colour space for histograms. In fact, it is a relatively new feature in darktable, too.
Exactly because normally I’m not a ‘gamut freak’ (and I only use sRGB and the display for output, never print), I never really checked the options for soft proofing and gamut check (my general attitude being ‘use the defaults set up by the experts unless you know what they do and know why and how to change them’). Then came your video about the new colour calibration module (‘shiny new stuff’ - and yes, I did use it on some Christmas lights, so I appreciate it a lot!). When you fought the red carpet (https://youtu.be/U4CEN0JPcoM?t=2081), I did not see artefacts in the carpet’s surface (maybe it’s me, maybe Youtube, maybe my monitor), but you did turn on the gamut check and switched the soft-proof profile from the default sRGB (you explained that from https://youtu.be/U4CEN0JPcoM?t=2135 up to about 36:50). I have now re-watched that and re-read Color calibration test and some thoughts - #14 by aurelienpierre (‘color profiles and chromatic adaptation may push colors out of visible spectrum’), and Color calibration test and some thoughts - #19 by aurelienpierre (’ we handle gamut mapping in color calibration only to cleanup after the chromatic adaptation because we know it will push colors out of the visible gamut’) where you answer the question perfectly. So that part is now clear.
I’m still puzzled about the validity (or maybe applicability/relevance to my use case) of the PQ Rec2020 histogram, even with the comment ‘The PQ curve from Rec 2020 PQ dilates shadows a lot and compresses highlights, so it gives you better legibility near black’, as it seems to indicate the photo does not make use of the darkest tones, while the HLG histogram actually indicates clipping in the darks (there’s a minor spike on the left end). Do the two spaces have different black points, or do I misunderstand something? (I’ll just continue to use the default sRGB, I think.)
Thanks,
Kofa
1 Like
Don’t use the PQ histogram neither the hlg as they appear to be broken.
Looks at the PQ histogram, it shows that in your image there are a lot of data at 10000 nits and i don’t think this is the case.
Same issue for hlg, however hlg has a pure gamma (2.2 or 2.4) from 0 to middle gray for legacy display, and a logarithmic shape in the upper part, so i don’t really see why it should be better than srgb.
I checked the dictionaries but keep on stumbling on the meaning of “primaries”…
…and then I’m all lost… Maybe I shouldn’t read in a sleepless night?
Thanks for any help
Here you can find a very good description:
https://ninedegreesbelow.com/photography/all-the-colors.html
See Chapter B2 “Chromaticity coordinates”
It’s also that I’m pinged all over the place since early December, I try to answer everyone quickly, but that often involves reading in diagonal between cooking supper, coding math and spending a bit of time with the wife, so I don’t overthink stuff before writing it. Plus, as many here, English is my third language so when I’m tired, it becomes French translated 1:1, which sounds a lot harsher than what English natives are used to since we don’t do euphemisms.
Bottom line, I apologize, I wasn’t targeting you personally. My point was that histogram, as any other scope, is meant to make your life easier, so if it doesn’t, just disregard it. As I said somewhere, scopes are useful to diagnose where issues come from, as long as the picture looks good, don’t bother.
The “darkest tones” are contextual. The PQ curve really dilates lowlights a lot. First, you don’t need to have true blacks (aka RGB = 0), then it’s all about axes scaling.
sRGB is a safe choice indeed. Black points don’t exist in reference spaces, since they are not tied to a medium, so it’s only a matter of encoding range. And, given that many printing drivers don’t do black point compensation, not having RGB = 0 can be a safety jacket.
And you know that because… ?
HDR formats usually encode RGB over 10 bits, that is 1024 code values. So 1023 encodes 1000 nits. But, for obvious compatibility reasons, all HDR formats encode 100 nits (aka standard SDR white) at 255, which is the max in 8 bits. darktable’s histogram shows the 0-255 range in 8 bits, no matter the color space. So, the 100% of the histogram is still SDR white, thank you very much.
7 Likes
Thank you @anon41087856, this is an excellent post.
I just wish that this (and other educational) post(s) could be preserved as blog-posts — maybe on darktable.org, here on pixls.us or maybe your personal blog.
Over the past year or so I’ve read several of your educational posts and found them very insightful but I’m wondering how many I have missed and how I will find them again in the future.
Maintaining a pinned post “Aurelien’s wisdom” with links to posts like this would work as well.
Thank you for your great work!
Oliver
1 Like
This forum, dakrtable.org, and the official documentation are all open. You’re free to curate these links, write a blog post on dt.org or provide content to dtdocs.
1 Like
With both PQ and HLG, 100cd/m² (nits) is encoded at codevalue 512, not 255. This is very unintuitive but even the SDR range of whichever HDR10 implementation gets more bitdepth…I suspect the much larger bt2020 gamut ‘requiring’ this for there to be no visible banding. Even sRGB primaries can show banding in 8-bit sometimes…makes sense, but is very counter intuitive.
I know you’re active on many fronts – researching, coding and educating. I sometimes wonder if you sleep at all. Kudos, Aurélien, and thanks for all that you do for the community!
4 Likes
The easy fix for the PQ histogram (and color space too) is to divide by 100 in linear rgb before using the PQ equation.
As a simple hack with the exposure tool set to -6.643 EV the histogram is correct.
It would be good to have all those technical pieces in one place to go back to. AP thanks for all the time and effort you take to explain the math and concepts @anon41087856