Diagonal interpolation correction artifacts with AMaZE Demosaicing

[LuisSanz (Luis Sanz Rodríguez)]

@agriggio

@LuisSanz

It reminds me to the same kind of effect that occurs when the white balance is applied after demosaicing.

WB is also applied after demosaicing in RT. it wouldn’t be trivial to change this I’m afraid…
do you think there’s a way to compensate for this in RCD?

note that something similar happens also in the original dcraw when you use a nonzero value for -H

I see. I also think it’s more efficient for a software with GUI to apply the white balance after the demosaicing. Otherwise every change in temperature or tint would need a full reinterpolation of the raw data or to set up a rather complex system that is probably not worth the effort.

Any non-balanced scaling of the RGB channels after the interpolation would result in that kind of color misalignement near edges, since the demosaicing cannot take the decompensation into account in order to smooth it. I’ll check if I can mitigate the issue in RCD.

[Elle (Elle Stone)]

@RawConvert

I’m confused when you say

using UniWB as the whtie balance for interpolating the raw file

Why would you do this, wouldn’t you produce a green image? - but I might be confused!

Surely one would use the appropriate factors for the interpolating/demosaicing, not (1,1,1,1) ?

There are two very different issues here:

On the one hand, people will say “You can’t white balance an image after it’s been interpolated.” This is not true. Whether you interpolate using UniWB or else use the actual white balance that you want the final image to have, as long as the image is still in the camera input profile you can change the white balance after it’s been interpolated, the colors will look just fine. Occasionally I have found it convenient to white balance an image after it’s been interpolated instead of before, for example when the lighting was mixed and I wanted to apply two different white balances.

On the other hand, not white balancing before interpolating can produce artifacts, which I didn’t realize was the case. The artifacts aren’t always obvious. My studio lighting, such as it is, is tungsten lighting, and when using tungsten white balance the artifacts from not white balancing before interpolated are not very noticeable. But if the desired white balance is daylight white balance, the artifacts are considerably more noticeable.

So at 400% zoom for tungsten white balance, I ddin’t notice anything amiss until I had the “white balanced before interpolation” image to compare to the “white balanced after interpolation” image. But at 400% zoom for daylight white balance, the artifacts are glaringly obvious.

I’m guessing it’s the high red multiplier required for daylight white balancing that is key? But this is just a guess. Maybe there is a pre-interpolation white balance that is ideal regardless of the desired output white balance? What about images shot under completely overcast skies vs. images shot under daylight/sunlight? As I don’t know what causes the artifacts I’m not able to generalize.

[afre (Alan)]

Good findings @agriggio and @LuisSanz. I had a suspicion that it had to do with wb and highlight mode. That is why I choose to compare output using that particular PlayRaw sample with its narrow spectrum, color imbalance, clipping and noisiness. I just didn’t understand what my own intuition was telling me or how to express it into words .

[Elle (Elle Stone)]

@RawConvert

folks, I think I read some time back, probably on Pixls, that it’s better to do the white balancing along with the demosaicing, rather than afterwards. If you want the best quality then, perhaps one should use DCRAW and input the factors. But how do you know what they are? Can you use RT to set the white balance wysiwyg, and somehow obtain the factors represented by the temp. and tint? I can’t see an obvious place where this is. Looking at the exif in a jpeg produced by RT, there is “ColourBalance” at “10, 798, 1024, 1024…” and MeasuredColour at “12, 442, 1024, 1024…” but these don’t mean much to me and when I expand the pane, the "…"s remain.

You can’t use RT to set the white balance and then somehow squeeze out the RGBG multipliers. Most raw processors provide several ways to set the image white balance, including setting the RGB multipliers. RT does not. I don’t know why, but maybe this is tied up with how RT pre-white-balances or something.

Across the various raw processors, the “meaning” of RGB multipliers is more or less constant, meaning setting the same RGB multipliers in darktable, ufraw, dcraw, photoflow, photivo produces pretty much the same final image colors. But as RT doesn’t provide access, you can’t share the same multipliers when comparing output from RT and other raw processors. The best you can do is hope that the presets and “camera” are the same for the raw processors you want to compare.

Unfortunately, the temperature/tint slider values for the various raw processors don’t correlate at all. The “same” temp/tint taken from one raw processor and fed to another will produce more or less different colors.

This page has summary information for various raw processors’s options for setting white balance. PhotoFlow wasn’t released yet when I put together the information: Raw Processor Review, Part 1

When using dcraw, I find it convenient to get the white balance multipliers from UFRaw and then copy them over to dcraw using the “-r” multipliers. Open UFRaw, set the white balance you want, then click on “Options” and select the Configuration tab - the white balance multipliers are available under “”.

The exif data in the jpeg comes from the raw file, and shows the white balance for various shooting conditions such as daylight, cloudy, etc, as given by the camera manufacturer. Use exiftool at the command line to examine the full contents of the raw file and the corresponding jpeg: “exiftool filename.ext”

[agriggio (Alberto)]

Hi @LuisSanz

@LuisSanz

I’ll check if I can mitigate the issue in RCD.

I hope you don’t mind if I went ahead and tried to do this my own way
I just pushed my “hack” to the rcd-demosaic branch in RT. It basically tries to compensate for the difference in highlight scaling between dcraw and RT. I would greatly appreciate your opinion on this (Also, please keep in mind that I only took some undergraduate courses on image processing a long time ago, and I’ve never worked with image manipulations since… so what I’m doing is dictated purely by “gut feelings”, and as such might be totally off!).

Anyway, for the curious, here are the results I get on the couple of test raws you posted above:

• RCD in dcraw:

• Original RCD in RT:

• Tweaked RCD in RT:

• RCD in dcraw:

• Original RCD in RT:

• Tweaked RCD in RT:

Best,
Alberto

[Carmelo_DrRaw]

I have just finished introducing the RCD demosaicing into photoflow. The implementation is based on the RT version, adapted to process small tiles in parallel.

Moreover, and following what @agriggio has reported, I have modified the RAW data clipping behaviour in photoflow, which results in much lower purple fringing around dark objects on clipped backgrounds. The improvement is actually visible for all demosaicing methods.

A small technical detail: I had to modify the code here and here to avoid inf values when the coefficients at the denominator are either equal but opposite, or very small.

Here is a comparison between Amaze and RCD with the new RAW clipping method:

Amaze:

RCD:

More details can be found in this other post on pixls.us.

For those interested, the photoflow code can be found here.

[agriggio (Alberto)]

So, inspired by what @Carmelo_DrRaw did for PhotoFlow, I generalized my “hack” above so that it works with any demosaicing method, and not just RCD. I’ve just created a new RT branch called highlight-smoothing (for a lack of a better name…suggestions welcome btw) that tries to get rid of the bad purple artifacts described above.
There’s a new option in the RAW tab, under “Preprocessing”, called – guess what? – “Highlight Smoothing”.

Here are a couple of screenshots to show what it does:

• AMAZE demosaicing, highlight smoothing turned off (the default, corresponding to the current behaviour of RT)

• AMAZE demosaicing, highlight smoothing turned on

Now, if you are interested in this, you are very welcome to try it out, but please be aware that this is very experimental. It might actually be a regression for some images, I don’t know yet. I would especially like to hear about artifacts that are introduced by this trick, and under which conditions. In particular, one area that needs special attention is the interaction with highlight compression and the various highlight reconstruction methods.

[sguyader (Sébastien Guyader)]

Will you apply this for x-trans interpolation as well?

[agriggio (Alberto)]

@sguyader

Will you apply this for x-trans interpolation as well?

Yes, this is completely unaware of the CFA pattern – it does the same for all sensor types. I have tested only with Bayer cameras though, so feedback for other sensor types is welcome!

BTW, I should also add that you need to apply this on shots with significant portions of the pixels clipped, otherwise you won’t see any difference.

[sguyader (Sébastien Guyader)]

Cool Alberto. I have some images from Venezia shot facing the sun, with strong clipped reflections on water. It will probably help in such cases.

[TooWaBoo]

Latest compiled test version ( 4b6b806 ) for Windows (x64. Gtk3.22.26) can be downloaded from here:
https://filebin.net/cru16paem7pqbcs9

No installer included. Extract the folder “RawTherapee_rcd-demosaic3” to e.g. your Desktop and run “rawtherapee.exe” inside this folder.
Cache and settings are saved into “%localappdata%\RawTherapee-rcd-demosaic”. It leaves your existing installation untouched.

[Carmelo_DrRaw]

@LuisSanz @agriggio
I have spotted a couple of numerical instabilities in the RCD implementation, or at least in the current RT version, which might lead to inf and nan values.

1. in the lines like

float V_Stat = epssq - 18.0f  *  cfa[indx] * cfa[indx - w1] - …

the epssq factor is mis-placed, and should be put either at the end of the expression or in a separate statement, something like

float V_Stat = 0.f - 18.0f  *  cfa[indx] * cfa[indx - w1] - …
V_Stat += epssq;

The reason is simple: if you add a very small number to a relatively large one (in this order), most likely the small number will not make any difference, because there are no bits available to represent it. Consider this minimal example:

#include <stdio.h>
int main()
{
  static const float eps = 1e-5, epssq = 1e-10;
  float sum1 = epssq + 0.5f - 0.5f;
  float sum2 = 0.5f - 0.5f + epssq;

  printf("sum1=%e  sum2=%e\n", sum1, sum2);
  return 0;
}

At least on my OSX system, this code generates the following output:

sum1=0.000000e+00  sum2=1.000000e-10

You can see that the first sum is identically zero…

2. when computing the direction like this

VH_Dir[indx] = V_Stat / (V_Stat + H_Stat);

I encountered cases in which V_Stat = -H_Stat, and the above expression leads to a division by zero. In my code I mitigated the problem by writing

VH_Dir[indx] = V_Stat / (fabs(V_Stat) + fabs(H_Stat));

but I am not sure if this is correct, therefore I would really like to have the option of the experts…

[agriggio (Alberto)]

hi @Carmelo_DrRaw,
thanks for the freedback! I had fixed some of these already in the latest version of the RCD branch, but it is totally possible that I missed don’t cases… I will take another look

[Carmelo_DrRaw]

As far as I can see, you have introduced two assertions for the finiteness of the values… what I am suggesting removes the risk of having inf, which should be more robust…

[agriggio (Alberto)]

I also added parentheses to force a specific evaluation order… but I admit I did not check formally if that is enough

[LuisSanz (Luis Sanz Rodríguez)]

@agriggio, @Carmelo_DrRaw, that’s great. Thank you very much for your efforts and for spotting that error. Nice to see those changes in the way highlights are smoothed both in RawTherapee and in PhotoFlow.

I made some changes to RCD some days ago and I have just released a new version in the repository:

• Fix to avoid directional discriminations out of the [0-1] range.
• There was no need to compute PQ_Dir and LPF for every pixel, so both are no longer calculated for the green positions of the CFA.
• There is an overall speed up of about 21%.
• VH_Disc and PQ_Disc are no longer needed as a buffer, hence there is a 40% saving in memory consumption.
• There are no changes in terms of image quality.

@Carmerlo_DrRaw, I didn’t know the position of epssq mattered. Anyway, a more exact option would be:

        V_Stat = MAX( - 18.f * cfa[indx] * cfa[indx - w1] - ...  + 1.f * cfa[indx + w4] * cfa[indx + w4], epssq );
        H_Stat = MAX( - 18.f * cfa[indx] * cfa[indx -  1] - ...  + 1.f * cfa[indx +  4] * cfa[indx +  4], epssq );

        VH_Dir[indx] = V_Stat / ( V_Stat + H_Stat );

The same should be done to P_Stat and Q_Stat. You can check it in the repository:

LuisSR/RCD-Demosaicing

[TooWaBoo]

Latest compiled test version ( 6512b9f ) for Windows (x64. Gtk3.22.26) can be downloaded from here:
https://filebin.net/k7spq7kqi3rsy838

No installer included. Extract the folder “RawTherapee_rcd-demosaic4” to e.g. your Desktop and run “rawtherapee.exe” inside this folder.

Cache and settings are saved into “%localappdata%\RawTherapee-rcd-demosaic”. It leaves your existing installation untouched.

[age]

Is it possibile to update the higlight-smoothing branch?
I like this feauture, thanks

[agriggio (Alberto)]

Thanks, but are you sure you really need it? From my tests, it seems to do nothing for modern cameras with a good amount of dynamic range… do you have examples in which it makes a difference?

[age]

No, I’m not sure I really need it but I don’t have done any real test, maybe I guess it could be useful for some canon chdk or android phone raw.