What I did lately.
With sigmoid: reset history, the red is still > 255, even worse, about 300! Set display profile sRGB → the red is still > 255. So my edits and display profile have no influence.
With filmic mapper, default settings: red is limited to 255 (display profile sRGB and system display profile).
So, sigmoid may have an issue.
As I said it might need more testing but I am using the settings from the OP with no other variations and yours look a bit different…for me if I use his settings I get this …first sample the neck
I then tried just changing the color space for the primaries.
Red actually went up…
If I stay there and now just remove the blue rotation…
I get 236 in the red channel…
If I zero all the primary sliders I get…
Returning to rec2020 with zero primaries I did see 256…and then increasing the skew drove it higher… to like 290 ish
Changing to srgb in the primaries leaves it at 255…
Adding back just the blue rotation under this condition sent things to 360
So it looks like not only blue rotation even though it seems to really drive this but the skew/curve settings can contribute as well…you can reduce contrast and lower the skew to bring values to 255…
I did all these with display and histogram set to DT srgb…
Maybe this is a unique image and set of adjustments??
In your image for me If I don’t use sigmoid and leave all at the default then I get around
568 for red on the neck…
Turn on sigmoid again at the default
299
If I change the color space in sigmoid in the primaries section away from working or rec2020 which are the same for me to srgb I get 253…so this can help along with the smooth preset…
But rotating blue strongly will push the red again…
I just tried another recent play raw image and pushing the skew I could get values > 255 in the blue flower for blue, changing the primaries setting to srgb would bring it under again but then strong red rotation would bump it higher again …so the module can be pushed I think… where as in filmic you just see more and more of the image go to white and show a value of 255… when you bump the red channel for example…
Here is the xmp for my version.
_C7A8415.CR3.xmp (7.7 KB)
Looking at the pics here I am guessing WB differences is making mine look so different. I could justify my choice in WB, but that is merely chasing a red herring (imo). I sincerely say this will all respect: The point is not that WB effects colors. The point is not whether you can push and pull things to make the color picker give less strange answers.
If the color picker provides values outside of sRGB, when it is supposed to have taken display values and converted them to sRGB via setting the histogram profile, then something has gone awry. My guess is that the color picker does not actually perform the steps it claims in the manual or its doing a less than optimal job converting.
If the former,that is fine, so long as it is confirmed that the color picker is showing you the value of in the system display color space…assuming that it is using that space. It loses some value as a tool if it is confined to that space, but would be reasonable nonetheless. If the latter…that might render the color picker useless more generally. Hopefully it is just not clamping at the extreme high end (so we just ignore anything over 255 but dont have to worry about if a 200 value is legitimate).
The only way to know what is actually going on is if someone with the requisite skill base/knowledge takes a dive into the code or can speak firsthand about the behind the scenes action.
And the subsequent and linked discussions and changes branch out of here…
This is why there are some references to making the process a pass through essentially by setting the display profile and histogram profile the same so that you can be sure the data come to the histogram profile with out clipping or distortion from certain display profiles… as this is where the picker values come from…
filmic reads the output profile (for the darkroom = display, for export = export), and compresses colours to fit into that. sigmoid has the option to use sRGB, and in the planned agx, I added code to actually get the export profile (subject to verification and approval by the developers).
When I posted above and quoted
“As the global color picker runs at the end of the preview pixelpipe, it receives data in display color space then converts it to histogram color space”
I mis-read the display bit and assumed it was saying display-referred, since that’s what we have after the tone mapping. So why is the display profile involved in the picker?
Consider a DT user producing a nice sRGB jpeg intended for the Recipient. The user guesses the output module will clip highlights, so has a look at some picker values to see how the edit is going. The process is: edits → output to chosen space → jpeg → recipient → recipient’s hardware. What has the user’s display profile got to do with this? I don’t see why it’s involved in the colour picker.
A few other bits and pieces -
@rvietor , you said “Such a value would be a bug only if it’s supposed to represent a value in an 8-bit integer-encoded image.” …which is the OP’s example!
Also you say "if I keep display, softproof, and histogram profiles at sRGB, the picker values behave. " I did that, and used the OP’s XMP, and got 259/260 without any trouble.
@Donatzsky , you said “They absolutely can. In the scene-referred part of the pixelpipe (so up until the tone mapper), the values are effectively unbounded. 1 (or 255, depending on the scale used) also has no intrinsic meaning and is just another value.”
People often say unbounded but let’s not forget the raw values form the camera are very finite, and so values in the pipe are too! Sigmoid strolls off to infinity but quite quickly there are no more values to map! Do the values have an intrinsic meaning? - I’m not sure but don’t the scene referred values reside in the Working Profile, at least between Input Profile and tone mapper?
P.S. @Jens-Hanno_Schwalm , have you any thoughts on all this pls?
If you read the GitHub links I posted you will get some context for your post…maybe not a complete answer but context…
Well, when you use the color picker, you are still in darktable, afaik. And the internal format in darktable is floating point, not integer, and certainly not 8-bit integer.
|quote]
Also you say "if I keep display, softproof, and histogram profiles at sRGB, the picker values behave. " I did that, and used the OP’s XMP, and got 259/260 without any trouble.
[/quote]
Yep, you’re right. But the issue disappears when switching to filmic instead of sigmoid, even when I push the white reference to very low value (which completely blows out the highlights)
“Unbounded” doesn’t mean “infinite”, it means there’s no hard upper limit.
Of course values are finite in practice. But you can very easily go far beyond the limits of the raw files. Just add a few EV of exposure (keep in mind that +1 EV means that you mulitply the pixel values by two, so +3 EV means ×8). And even raw data from an old camera are 12 bits/channel, modern cameras often have 14 bits/channel, so with those +3 EV you’d need at leat 17 bits/channel to handle all possible values (if you wanted to stay with integer math).
To reach the limits of the float values darktable uses, you’d have to add over 100EV …
So apparently, what I said there is not correct, it seems sigmoid can produce values outside 0…1…
Whether that’s a problem in practice, I have no idea.
Yeah that is part of why I did a strike thu on (almost) everything in that post.
If you put anything after either filmic or sigmoid (with sRGB primaries) the color picker provides values outside of sRGB. even when the display profile and histogram profile are both set to sRGB.
The fact you can increase values past 255 after a tone-mapper is not surprising…and totally fine since they may be valid in the working space or unbounded for math purposes. I think “invalid” values when there is supposed to be a conversion may be a red flag that there is something untoward happening
The pipline remains in Rec2020, even if you compress to sRGB, you can ‘break out of it’.
Haven’t tested myself, but that’s probably due to the gamut handcuffs in Filmic.
Because both the picker and histogram code is a convoluted mess, apparently. It’s certainly very hard to reason about what is going on, as can be seen in this thread, and I do think both the picker and histogram would benefit greatly from a rewrite and simplification.
That’s not correct. Whatever the bit depth of the raw file, it gets mapped to the 32-bit float of the pixelpipe when loaded. And while 3.4028235 × 10^38 is technically finite, it’s such a large number that for all practical purposes the pixelpipe is unbounded in the luminance dimension.
The tone mappers take the [0.0, 3.4028235 × 10^38] values and squeeze them into [0.0, 1.0] (by default). In Filmic white relative exposure is then used to pick the scene-referred value that should correspond to display-referred 1.0 in the output, while Sigmoid manages that for you.
As for intrinsic meaning, in display-referred data (0.0, 0.0, 0.0) is pure black and (1.0, 1.0, 1.0) ((255, 255, 255) in 8-bit) is pure white (display white, actually). That is, they intrinsically mean something. In scene-referred data (0.0, 0.0, 0.0) is still pure black, but while (1.0, 1.0, 1.0) probably appears white, it’s really just another shade of grey.
Technically the data is scene-referred from the moment it is loaded until it is tone mapped. The working profile only defines the primaries that bound the color gamut from then on.
Here’s a good explanation from Elle Stone:
I was under the impression that the pipeline profile was set in the input color profile. The default is lin rec2020, but you can change this to something else if you were so inclined.
I am assuming (now) this is incorrect?
If you set the working profile to linear Rec2020, then that’s what the pixelpipe will use. I assume that what @kofa means is that a module could compress the data to sRGB and output that, but since the pixelpipe is still linear Rec2020 you can just expand the data to that gamut again.
I had thought about that when scratching my head regarding the color picker.
I set the working profile in Input Color Profile to sRGB and clipped the gamut to sRGB.
My naive thought was this would cuff the color picker into (what I take as) normal sRGB values. It did not.
As I explained further up (and Elle Stone even better in her article), the working profile defines the chromaticity bounds, not the luminance bounds, which is why you can get, for example, a red that is seemingly too red for sRGB (where max is 1.0 / 255): (0.0, 0.0, 2.0) / (0, 0, 510)
It’s also possible to have negative coordinates, and that’s true oversaturation.
The Elle Stone link was helpful. Thanks!
I think Aurelien has or is trying to do this. He has merged the picker into his scopes displays. There you can now find 3 options for picker readouts…raw, output and display. Coupled with this he has removed the histogram profile and I think scopes show the results captured from the profile selected for output…
I knew he had referred to it… I went to try and see how it worked just for fun but the picker placement was working erratically for me…might be a bug or what ever but similar changes in DT might certainly clear things up if the pipelines are similar enough…but maybe too much downstream issues would result…
