I can't believe this works - Thresholded White Balance!

ggbutcher · June 13, 2020, 4:54am

I’m a believer in the singular purpose of raw processors: process raw files to representative images. If I need to further manipulate an image, I fire up GIMP. Well mostly…

My first serious camera, the Nikon D7000, has a built-in flash. I used that flash extensively for family imaging, worked well except for the occasional red-eye. When it happened, it was pronounced, with particular attention to my poor niece, who looked like a demon in every picture. Since I wanted to make presentable proofs for others’ regard from rawproc, I capitulated to my principles and included a redeye tool, and put it to good use during that camera’s tenure.

Now, with the Nikon Z6, I’ve got a sort of dichotomous problem. It doesn’t have a built-in flash, and I haven’t yet bought one, so I find myself doing a lot of ambient light interior shots. All well and good if I set the white balance accordingly, but occasionally I run into a different problem. Here’s an illustration:

Yeah, the interior white balance is still a bit warm, but look out the window: yech. The frosted glass in the door shows the daylight color temperature in it’s full glory. Not nice.

We spent last weekend at a cabin in the mountains, and I shot a lot of interior images. And that pesky picture window is in a lot of them. I was about to power up GIMP and some @patdavid tutorials to do some layer masking, but I got to thinking about the situation: the regions of concern had a significant discriminator, that being a heavy bias in the blue channel. Bear-of-little-brain starts thinking, “how could I use that fact to segregate the ‘exterior pixels’ in a second application of white balance?”

Looking at the RGB values in the exterior pixels, I could reliably see a usable magnitude difference in the blue component from the next highest component. So I came up with a scheme that 1) builds a luminance from the mean of the three RGB values, then 2) divides the blue value by the luminance. That gives a magnitude that the blue is above the next highest value. Now I can use that to set a threshold in the white balance operator where, if that magnitude is greater than the threshold, apply the white balance multipliers.

Took about a half-hour to code that up enough to try it with the img command line processor. For expediency, I used that to process the above JPEG and lo, it works, albeit with a few artifacts. I am quite surprised, because I’ve tried to implement similar masked color manipulations for other purposes with horrible outcomes. Encouraged by the JPEG results, I incorporated the tool in rawproc, and found it gave an even better outcome when working from the raw starting point:

To get to this, I did the normal processing through the non-linear tone operator, with the camera-specified tungsten white balance. Then, I inserted another white balance tool and used the patch option on the frosted window to pick up the daylight multipliers. With that, I enabled the blue threshold checkbox and started dialing it up until the interior part of the image went back to the original white balance, in this case, about 1.2

This particular image shows a problem with this tool. Compare the stars on the nearest napkin in both the first and second renders. That leftmost star should be blue, but it gets caught up in the threshold for the exterior. If I dial up the threshold any more the star regains it’s proper color, but the exterior sky goes blue, which it definitely was not in the eastern sky twilight.

Yes, this is definitely a torx screwdriver sort of tool, made for a very specific purpose. I’m posting about it because 1) I can’t believe it works, and 2) wondering how such would be equivalently accomplished in the other FOSS processors we discuss here. In spite of it’s variance to my ‘singular purpose’ principle, I’m going to keep this tool. Well, at least until I buy a flash…

Edit: Images in this post are licensed Creative Commons, By-Attribution, Share-Alike.

iarga · June 13, 2020, 8:13am

This reminds me of “rg chromaticity” and the “RG color space”.

garagecoder · June 13, 2020, 10:00am

It would be interesting to know how that performs generally. I’ve experimented with various masks and thresholds, power laws etc. and they all have one thing in common - even with user tweakable parameters they never cover all cases easily. Thresholds tend to cause glitches but it’s usually simple to turn it into a softer fall-off. Incidentally, prototyping that sort of thing is really easy in G'MIC (once you know how of course)…

Example, to get blue to channels average ratio:
gmic DSZ_5092.jpg +compose_channels add channels.. 2 +eq. 0 add[-2,-1] div[-2,-1]

And then to view a thresholded image based on it:
gmic DSZ_5092.jpg +compose_channels add channels.. 2 +eq. 0 add[-2,-1] div[-2,-1] gt 50% DSZ_5092.jpg rv mul

ggbutcher · June 13, 2020, 1:48pm

Yes, this is definitely a one-trick pony. I think what made this particular use case viable was the significant distinction between object lit with daylight vs the rest of the scene lit with the interior lights. I have some other images where the distinction is less definitive, objects predominantly lit by interior, but have parts lit by the exterior light from a window, so I’ll be playing with those in a bit.

I was thinking just this as I headed to bed last night. Now, I’ve tried masking operations with G’MIC before and have not been successful, possibly due to the cognitive impariment my wife occasionally points out . So, how would you then apply a white balance multiplication to this image through that mask?

What I’d like to do for rawproc is to figure out some sort of plugin architecture that would allow one to just download, or even script, their own operators for just these sorts of things. But I still have to clean things up first, for 1.0…

ggbutcher · June 13, 2020, 2:05pm

Coincidentally, this thread

has a situation where this sort of processing would likely break down. There are two distinct illuminators in the scene: 1) direct daylight and 2) what appears to be blue sky, and they have a definite influence on the overall white balance.

garagecoder · June 13, 2020, 2:36pm

As you might expect there are many ways. If you haven’t already, I’d recommend creating a text file called .gmic in your home directory so you can easily create lengthier commands. It avoids a lot of terminal character escaping problems too. I presume you don’t mind me using your image! This is using some arbitrary multipliers (not intended to be a good correction):

test_masked_wb :
  +local.
    +compose_channels add
    channels.. 2
    +eq. 0 add[-2,-1]
    div[-2,-1]
    gt 50%
  endlocal
  (1.4,0.9,0.5) diagonal. # a dummy "white balance" matrix
  +mix_channels[0] .
  c. 0,255 rm..
  mul. ..
  eq.. 0 mul[0,1]
  add

That’s quite a mathematical way to do it (multiply by a mask and the inversion of the mask), which is good if you’re prototyping for another language. There are various convenience commands which make the job quicker and shorter.

ggbutcher · June 13, 2020, 2:48pm

I added an appropriate license to the original post.

Your multipliers are close to what I came up with for the raw file: 1.3, 1.0, 0,5. I’ll study your code, thanks for taking the time to write it!

anon41087856 · June 13, 2020, 7:37pm

I remember having read a paper by Sylvain Paris (Adobe guy who also did the local laplacian thing) about a dual white balance correction. Basically they express the pixel illuminant as the linear combination of 2 illuminants. They solve the weights of each illuminant for each pixel through a clever math trick, and are indeed capable of dealing with A and D65 illuminants on the same scene.

Caveats are:

the results look almost too clean to be believable (like a 3D rendering)
I can’t find that paper anymore.

garagecoder · June 13, 2020, 8:26pm

Hmm sounds suspiciously similar to laplacian matting foreground/background separation, in which case would probably take a lot of cpu time. Still, I might try and find that paper…

Edit: in fact that’s exactly what it appears to be here

So I don’t think I’ll be trying it out - those algorithms when I tested had really bad asymptotic performance (probably more than quadratic)

anon41087856 · June 13, 2020, 10:33pm

Thanks for finding it ! It’s exactly the one.