Also notice that all demosaicing methods rely heavily on white balance, which has the same de-saturating effect as any convex channel-wise transfer function. But… accurate chromatic adaptations all rely on a full 3D vector sent to some LMS cone space for that purpose, meaning they need demosaicing and color profiling before… That’s a nasty circular constraint : accurate WB needs demosaicing before but accurate demosaicing needs WB before.
Adobe seems to be doing a shitty 3×1D white balance fix before demosaicing (sensor RGB rescaling), then demosaics, then undo the technical WB and applies a proper CAT + profiling. In any case, desaturating sure takes care of most of chromatic aberrations.
And that’s my biggest grip with all papers about demosaicing… They artificially mosaic image samples from the Kodak and IMAX datasets (that is… digitized film), demosaic them, then compute SSIM or PSNR over the difference. But of course, these are properly white-balanced from the start. So we have no metric of the WB-independence of such methods. We just know how they work for D50 illuminant.
Correlation means the coordinates of the hills and the valleys in the laplacian are the same over all channels. But laplacian is a second order operator, aka the signal modulation around the local average, aka the texture of the image (including noise). Unfortunately, cheap demosaicing methods care about first order gradient only, which magnitude depends on the magnitude of the signal itself (same as the variance — which was the starting point of the exposure-invariant guided filter @rawfiner started).
Fixing the scale of the gradient magnitude is partly taken care of by the WB. As long as you stay in linear space, it’s a matter of applying a coefficient. If you leave linear spaces, good luck… You lose any connection with meaningful stuff connected to real-life to enter the black magic of empirism that looks good® until it doesn’t.
Actually, I’m starting to wonder if normalizing sensor RGB by the local average before demosaicing (kind of the “grey-world” assumption piece-wise) and then undoing it after demosaicing would not be a cleaner way of desaturating while retaining a better inter-channel correlation. Because the examples showcased here clearly spot colored highlights that don’t match the global scene illuminant. That would be equivalent to dividing the whole picture by its blurred version, with a blur radius that still needs to be chosen depending on the alignment of Jupiter with Saturn or something.
in fact i can observe a good amount of impact on the kind of colour fringing when doing denoising before demosaicing. the denoising can aggressively lower the colour frequency and the demosaicing will then spread colour along luma edges… not convinced i found the perfect combination there yet.
That is exactly one of the problems. I searched several time for papers calculating some sort of noise but found nothing really good yet. If you know of one, please let me know.
We would need to have such an algo and agreed real-world test files to check.
Unfortunately we don’t have the raw file. Also don’t know the used demosaicer. I’ve seen such spots with ppg / amaze at such harsh transitions.
I thought long about this (emphasized) and I think I can only partly agree with the logic described here.
A CAT should be picture scale/size invariant (at least to first-order approximation if I am not missing something), so what is holding one back to see a bayer rggb unit-cell as one RGB superpixel? For bayer-CFAs take a quarter resolution version of the sensor triplet, do your initial CAT and then do the demosaic. For x-trans take 1/9-th resolution.
The circular constraint is certainly not nice, for sure. Two things can happen: the algo converges onto one solution or it doesn’t converge. To me this sounds that point one could be a way out of this loop if one subscribes to the idea that WB should be scale-invariant.
I’d love to read a good argument why a CAT should be a crucial part of a spatial-sparse-sampling problem. The CFA-SSFs are not the human visual system. The demosaicing problem should remain the same if you shift all CFA-SSFs 500nm into the IR. Proper WB before demosaicing sounds like a backwards subjective quality control, an additional constraint to have the algo ‘behave’.
I see two ways to solve this. A rendered full-spectral test scene which can be used to construct a ground truth AND the mosaiced data (METACOW comes to mind, or someone renders something specifically for the task, like here), or a forummember with a sensor-shift camera could supply test images to specifically have full resolution and mosaiced image data.
Sorry for drifting away from the question if the LMMSE-gamma-step is a poor-persons VST.
If that is meant in response to point 3 above:
Depending on the illuminant and the sensor-SSFs the ‘sensor achromatic’ is/can be very different to ‘human observer achromatic’. My guess is that a von-Kries transform is in it’s simplicity the exact thing one wants to do for the sensor in order to minimize artifacting in that case, but then the sensor-WB is not a catch22 to the post demosaic observer-WB.
In other words: a sensor-grey-world assumption would be straightforward, not circular and should not be tainted by HVS specific CAT ideas (cone specific nonlinearities etc.).
I think I need to study a bit deeper on why demosaic algos need proper whites… I know they compare gradients between the channels to do the upsamling often assuming green as the luminance estimate.
But it still weird to me why we would be forced to apply white balance before. Having a white balance invariant demosaic step would resolve the problem of proper color handling to be done afterwards as well as not introducing weird scaling that has to be taken into account when denoising afterwards. Heck the demosaic should take the camera noise profile into account such that it doesn’t propagate noise across channels!
i think this is most prominently due to the old ahd implementation dt had copied from ufraw. that one did some fake lab conversion on the pixels and computed gradients only on luminance, not on colour. it does show severe mazing artifacts if no white balance is applied before. i think many others do not have this property. in vkdt, i’m jointly estimating the overall direction of edges by looking at features in the half-size image (including all sensor readings).
before demosaicing, while still working on the single sensor reading per pixel you can’t really apply any sophisticated chromatic adaptation transform. all you can do for a single pixel in isolation is a per-channel multiplication in camera raw. that has some restrictions (blows values out of all gamuts, doesn’t really match physical reference so well etc). but you’d probably apply a matrix later, maybe even an interpolated one? for a specific scene illuminant.
i very much dislike this camera-raw multiplier. once you do that, your colour coordinates are off/irrecoverably out of gamut. the dng matrices come with illuminant anyways, and you can replace them by a more accurate lut/spectral input transform.
Thanks, makes sense to bear-of-little-brain here. I may do some experimentation with the librtprocess algorithms, see if this holds out in current implementations.
From the start, it always seem to me to be a rather egregious thing to do to the data. Some years ago, I tried making a colorchecker matrix profile with an unwhitebalanced target shot, and the results were quite satisfying in terms of preserving color “richness”, for lack of a more precise term. Here’s the post I did on it:
i would second your suspicion that maybe the reason was the quality of the first profile. it seems to me that camera raw wb (a multiplication by a diagonal matrix) followed by a matrix multiplication (the one fitted for the profile) is still a linear operation, i.e. can be expressed by a single matrix multiplication just fine. so given the same input and a capable optimisation algorithm i would expect the same result here.
Besides the demosaicing methods that explicitly try to demodulate luminance from chrominance (whatever that means before having applied a color profile…), and therefore need some proper relative weighting or R, G, and B, most methods try to read the green channel to extract its gradients and bend the gradients over R and B in a way that keep them correlated (meaning no chromatic aberrations). As soon as you start messing up with gradients, you need their magnitudes to be similarly scaled if you want to swap them between channels without overshooting.
Also, behold the X-Trans CFA and its bullshit non-uniform pattern that makes pretty much any uniform discretization impossible. This one has to decorrelate luminance (actually “green”) from the chrominance because of that weird sampling. I will take the fact that Fuji medium format cameras use Bayer as a remorse.
i don’t agree. i’m using the same code for both bayer and xtrans now, and i’m going to claim that it doesn’t depend on white balancing beforehand. the whole difference is that bayer uses 2x2 blocks and xtrans uses 3x3 blocks.
yis, that’s what i started out with. by now i have so many extra regularisation measures in place that i’m not sure i’d call it that. seemed necessary to suppress a few artifacts and to improve sharpness.
