In that case I agree, but I think this asumption is part of a lot of misunderstandings. For example, this thread is about “removing magenta in blown highlights”
Did you look closely at the hlrecover output? It might be more interesting than at first glance.
We have only two clipped green channels, usually ideal for highlight recovery. Here’s one of them in sign of laplacian view:
OK a few spots, might not be too bad, but after recovery…
Edit: I suppose once filmic is applied, that probably will be OK. Shows the limits of recovery as you say though…
Well they are magenta because that is what your sensor recorded. Turn off all of filmic and reconstruction and lower the exposure , and you’ll see it’s there . Other tools will either remove the color , or will not allow you to try to get detail from it, or have good reconstruction.
If you want the final result to 'clip to white ’ , just raising the filmic white level (hmm… slider to the left at least ) will push them to white . Also ‘luminanceY’ mode will push them to white earlier / easier than the default max-rgb.
So if you do not want to get much from the highlights, and want just the magenta to be away… Push them to white. Or enable the clipping ‘reconstruction’ and push the flat white to bright white . Done.
This all has nothing to do with what ggbutcher mentioned : you need to record shot at a lower exposure . This has nothing to do with intended output . Not blowing the highlights will give you all the options in post , and an easier time in edit .
Actually, the magenta cast is introduced by white balance. All the sensels pile up at the same place, and the white balance multipliers skew those piles by-channel into the magenta color.
Ah ah agree, and I personnally don’t have problems turning the highlights white, but I was pointing out that everybody seems to advise on recovery but this is not really what the op asked for…
As for @ggbutcher please don’t feel offended, I admire your knowledge and your work, I just quoted the last person who said that, and I will try to rephrase: blown highlights happen, for example you want to record the "main"landscape and don’t care for a bit of a white sky.
Edit : and it has to do with the intended output because you chose it when taking the shot
Oh, no worries, I still do just that sometimes. In-scene lights don’t deserve the attention IMHO, so I’ll just let them go and let whatever HL recover I have handy drag the magenta back to white.
For most daylight scenes, we’re trying to cram 5 gallons of milk into a one-gallon jug. Something’s got to give…
The thread, not you specifically. Must have clicked the wrong reply button .
And i had my post written but waa distracted , and before I posted someone else already mentioned that the correct level is 3600 from camconst.
Yeah yeah we had this before in other threads .
Teacher me stops at what i think is enough information for people asking questions . But you are entirely correct .
Your sensor clips, so it records maximum (equal) values for all channels … so , white.
But, this is the raw data with no white balancing. So a kind of regular daylight white balance , will push red a lot and blue by a fair bit and leave green alone .
So what was ‘1 1 1’ can now become ‘2.5 1.5 1’.
And something that is mostly red with a bit of blue , is a magenta.
I also read - from the Color perfect devs if I recall correctly - that trying to recover everything can make things look flat. Most of the time, an image needs a bit of black and a bit of real white to give it some contrast , punch or impact. ColorPerfect had a percentage indicator, saying how much percent of pixels is clipped at your current settings . And they recommended not getting it down to 0%, but like 0.5% at least :).
So , back in Darktable ‘world’. People got to remember that highlight reconstruction does not mean that you want to cram all the dynamic range into your output.
It tries to fix blown out sensor data , so that you an use it.
But it’s perfectly fine to not try to use it! Or not use it all. Clip a bit, no problem ! It all depends on the image and what you want from it , right ?
And yes, sometimes that means you don’t have to reconstruct. Just blow it out.
But in that case , just using the white slider in filmic to indicate that you do not want to map all those highlights in your output is the easiest way to go.
Teacher me gets the need for abstraction, but not at the expense of wrong information. The questions you avoid now will just pop up somewhere else, and maybe wrongly crafted due to a misunderstanding of how things work.
Absolutely. At the hl reconstruction module we don’t have perfect coeffs. Unfortunately by current design of how the white balance module works.
I have direct experience with the implications of such; had to sit through a briefing from a group who devised a rather elaborate plan of attack on a problem based on a misconception from such an abstraction they’d heard some months before. Got to the end and had to say, “well, that won’t work because…”
Sorry to be very late to the party here…
I agree with Glenn’s point here. There are a few alternatives that could be used at capture time to avoid the magenta in the sky:
- reduce exposure enough to not have the sky blown;
- compose the shot to exclude the sky, allowing you to ETTR for the rest of the scene and probably get cleaner data for it (many landscape photographers try very hard to not get any sky in some shots);
- take a shot exposed for the land and a shot exposed for the sky, and blend them.
I know that doesn’t help after the fact, and you may or may not be in a position to revisit the location to re-shoot it. I continue to learn these lessons the hard way myself.
The elephant in the room in many cases for these artifacts the data, those D65 coefficients for one… How good are they?? How accurate are they. How do you assess them and determine them to be accurate/valid. With the modern wb a set of numbers are introduced upon which to base the calculations and if something is off here it will propagate… And maybe also the raw white points at least for some cameras that seem to be not in line with the camera data.
With respect to wb, AP suggested one way to deal with them and create a preset but many users will never come across this or perhaps understand that sequence of calculations. They may think this is the way the wb should be because its newer and called modern… Maybe there is a very systematic way to evaluate the two methods
On several images presented as playraw subjects I have found that using the modern wb introduces the issues in the highlights esp when paired with filmic and esp v6. Often the legacy wb with all things being equal in the edit does not. So then we have to ask is all the dicking around in the highlights to correct this worth what may or may not be gained in the rest of the image??
I know we can resort to tweaking HLR, also HLR in filmic, change chrom pres in filmic, change the version of filmic and actively desaturate HL with color balance or whatever. However if using legacy wb is enough to avoid some or all of this at no real detriment to the edit then it might be a better default option at least for some users.
If the image is a daylight image with fairly predictable lighting is there a tangible gain using the modern wb workflow. Maybe if we get into more complex lighting situations there is and of course you can use masking and multiple instances…
If things are all calibrated and a color checker or gray card are used the issue is likely less important but many users are operating without those aids.
I’m curious what peoples thoughts are around the two workflows…
If I understood the explanations correctly, the color calibration is there to allow proper gamut mapping, which existing modules didn’t do. That gamut mapping is important when you have colors out of gamut and you want your mapping to be correct. That is, you want to be sure your mapping is done as advertised.
This has in principle nothing to do with the illuminant you use. So color calibration can be useful even under daylight, depending on your images and the output space you want/need.
Keep also in mind that for years filmic was used with the standard WB module only, and without many complaints about color accuracy.
Now, if you don’t have highlights you want to recover, I haven’t found any difficulties caused by the color calibration module. As it is a new tool, I have to get experience with it by using it. So I leave it as default (also because I think the devs know what they are doing).
But all the recent discussions about highlight recovery made me think that just sticking to only the old WB module could have its uses when there are over-exposed areas in the image. Highlight recovery is enough of a pain already (and while the Laplacian recovery is nice, it’s also horribly slow, worse than diffuse or sharpen is some cases…)
So a large part or me is learning what works best in each individual situation, I don’t believe in standard recipies. I’ll probably end up using both “workflows” (hard to consider not using one part of one module a change of workflow ). With the “modern” color calibration as default (the gamut mapping doesn’t hurt), and the “classic” method when highlights are being difficult.
I tried using “modern” when it first appeared and stuck with it for a while, but I’ve now almost completely given up using it. Almost all my photography is daylight and to me, “modern” offers no advantage over “legacy” but introduces several difficulties, as discussed upthread. In the very rare case when I have something with mixed lighting then I’ll take a deep breath, put on a pair of thick rubber gloves, and use “modern” for the masking etc.
Lurker and user since DT version 2.something.
The thing is, when dealing with clipped highlights, I have seen no difference in using whatever white balance whatsoever. Also in the image in the OP of this thread.
I haven’t seen a case where the magenta - or something like that - is gone when using legacy WB.
Either there are artifacts because of clipping that need to be dealt with somehow, or there aren’t.
Since my camera profile (DT built-in Sony) only works correctly on proper white-balanced images, I disabled the modern workflow (if the scene is too far off from daylight, the camera profile will mess up the colors. I can’t place ‘color calibration’ before the input profile in the pipe, so I am out of luck). But every playraw I play with, I see it as a test to work with images from different cameras I don’t own, and I always try modern mode first. And - for real in this thread since the discussion went that way - I tried both ways, and have seen no difference.
Take the image in this thread. If I try it legacy style (white-balance set to as shot, no color-calibration) I see magenta skies that need to be dealt with somehow (reconstruction / desaturation / filmic v5 / etc…).
If I use the modern approach (white-balance set to reference, color-calibration set to ‘as shot’) I still see the magenta skies that need to be dealt with? Yes, the magenta is slightly different… but it’s still there, and it’s still a problem.
I have found that using the modern wb introduces the issues in the highlights
So basically, it comes down to this. I think this is just not true. Can you point me to images and a reasoning behind this?
color-calibration does all its calculations expecting a certain ‘good’ base image, yes. And the colors and things it may produce can be slightly different depending on the base white balance. But it won’t suddenly mess up the highlights or go in a different direction when things are clipped.
I have a busy couple of days but I will try to respond… I did a post as I said a short while ago (it may have been on Github and in HLR topics) and it was obvious in several images that with color calibration defaulted to as shot when filmic was enabled that you got artifacts either cyan or magenta that did not appear if you were using the legacy wb set to as shot. Before Kofa’s recent comments on CC and how it can tint highlights when used with a couple of the standard HLR modes I had asked if there was some different clipping going on with legacy to avoid this behaviour. And yes you can make changes to handle the cyan or magenta that appears but my question originally with all things being equal was why did I see these appear after adding filmic with CC setting the wb vs the traditional way… I have a couple of new instruments being installed the next couple of days… I will dig up the images but that was really it…
It’s not to criticize! It’s for me to learn. So I’d love to be wrong, because then I have something new to explore .
So if you can find examples, please. Not for me to be right, but for me to learn something.
The WB multipliers, applied to blown-out raw data are the cause of the magenta highlights. No WB setting will fix them.
The difference between ‘legacy’ and modern:
I used highlight reconstruction in reconstruct in LCh mode. As you can see, since this mode can only recover brightness details, but not colour, it produced a grey value. Since the sky is featureless, it’s pretty much homogeneous grey.
Switching the WB to camera reference (6502K) gives a slight change in the highlights, but that’s not really a problem:
Picking individual pixels:
Now, let’s enable the other part of the modern chromatic adaptation, color calibration:
Let’s have a look at the pixels which reconstruct in LCh set grey, are no longer grey, but bluish:
This is because color calibration did what it was supposed to do: it took a pixel, which had a certain RGB triplet, and transformed it.
- The camera said that for a correct white balance, the R component should be multiplied by 2.508, the green left alone, the blue by 1.844.
- The white balance module, set to 6502K camera reference, was actually using the multipliers 2.692 (more red) and 1.658 (less blue)
The color calibration module applied an adaptation to fix the ‘wrongly white balanced’ values. Something more involved than simply adjusting multipliers, but with the same overall effect: boosting blue and reducing red (to counteract the lower blue, higher red multipliers used in white balance).
This works well, in general. But don’t forget that highlight reconstruction (both the clip highlight mode as well as reconstruct in LCh) produces neutral (R=G=B) values, which are not really the original colours in the image, only a grey approximation. However, that grey is then turned blue by color calibration, which has no idea that those are not actual colours, but synthesised grey.
Now, if you use filmic or other tools that desaturate or otherwise correct the colours in the highlights, this may not be visible.
Chromatic adaptation is the human visual system’s ability to adjust to changes in illumination in order to preserve the appearance of object colors. It is responsible for the stable appearance of object colors despite the wide variation of light which might be reflected from an object and observed by our eyes.
If I understand correctly, color calibration, when illuminant is set to as shot in camera, tries to figure out
- what RGB values should be used to represent a pixel in the pipeline
- that uses a colour space having a D50 white point (‘5000K daylight’)
- if that pixel has some RGB values when applying the reference WB multipliers belonging to 6502K
- given that the camera originally used another set of multipliers (representing its estimation of the original illuminant).
If you set illuminant to some other value (e.g. D or custom), the same applies: given that RGB values of the pixel produced by the 6502K multipliers are so and so, what RGB values should be used in the pipeline instead, given that the original illuminant is described by the parameters selected by the user (illuminant type + settings, like D + temerature, custom + hue + chroma).