Diagonal interpolation correction artifacts with AMaZE Demosaicing

[agriggio (Alberto)]

Hi @LuisSanz,
so today I was sick at home so I had some time to play with this… here’s what I found.
I think the differences are not due to WB, but rather to the way dcraw and RT deal with clipped highlights. Here is a patch to your dcraw.c that enables a further highlight mode -H 10, which tries to emulate what RT does (when highlight reconstruction is turned off):

diff --git a/dcraw.c b/dcraw.c
index ca28559..f30793b 100644
--- a/dcraw.c
+++ b/dcraw.c
@@ -10206,6 +10206,29 @@ next:
       else
 	ahd_interpolate();
     }
+
+    if (!is_foveon && highlight == 10) {
+        float dmin, dmax, q;
+        unsigned i, c, size;
+        int val;
+        
+        for (dmin=DBL_MAX, dmax=c=0; c < 4; c++) {
+            if (dmin > pre_mul[c])
+                dmin = pre_mul[c];
+            if (dmax < pre_mul[c])
+                dmax = pre_mul[c];
+        }
+        q = dmax / dmin;
+
+        size = iheight*iwidth;
+        for (i=0; i < size*4; i++) {
+            if (!(val = ((ushort *)image)[i])) continue;
+            val = ((float)val) * q;
+            ((ushort *)image)[i] = CLIP(val);
+        }
+        highlight = 0;
+    }
+    
     if (mix_green)
       for (colors=3, i=0; i < height*width; i++)
 	image[i][1] = (image[i][1] + image[i][3]) >> 1;

Here are three crops of your test image above, developed respectively with:

• dcraw -v -T -4 -H 0 -W -o 1 -a -q 4 -c DSC_0934.NEF

• dcraw -v -T -4 -H 10 -W -o 1 -a -q 4 -c DSC_0934.NEF

• rawtherapee-cli -Y -tz -s -c ~/Downloads/DSC_0934.NEF
  (neutral profile, RCD demosaicing)

It seems to me that the dcraw image with -H 10 is very close to the RT one. What do you think? The difference between -H 0 and -H 10 is in the scaling of the raw values to normalize them to the range [0, 65535]. -H 0 is more aggressive in clipping before demosaicing, whereas -H 10 is more conservative before demosaicing, but then rescales the values later. This is closer to what RT does. As to why this has an impact on your algorithm, well, I have no idea but I hope this could be helpful…

[LuisSanz (Luis Sanz Rodríguez)]

Bingo, @agriggio! I was also sure it was related to the highlight scaling rather than to white balance.

Is RawTherapee applying that normalization before or after demosaicing? In your patch to dcraw, it seems to affect negatively not only RCD but also PPG and AHD.

It reminds me to the same kind of effect that occurs when the white balance is applied after demosaicing.

Thank you very much for your findings and I hope you get well soon.

[agriggio (Alberto)]

@LuisSanz

Is RawTherapee applying that normalization before or after demosaicing?

after

@LuisSanz

It reminds me to the same kind of effect that occurs when the white balance is applied after demosaicing.

WB is also applied after demosaicing in RT. it wouldn’t be trivial to change this I’m afraid…
do you think there’s a way to compensate for this in RCD?

note that something similar happens also in the original dcraw when you use a nonzero value for -H

[Elle (Elle Stone)]

@Elle

as a confirmed pixel peeper I’ve never noticed any problems using UniWB as the white balance for interpolating raw files. . . .
The image I chose as a test image was shot using what is supposed to be a very nice high end lens, attached to my Sony A7 by the store attendant before I purchased the camera - I think using an adapter

Hmm, I’m turning in my pixel peeping credentials. It wasn’t the lens, it was the bright sunlight. The artifacts from using UniWB for interpolating the raw file are much less noticeable for images shot under incandescent light. I wonder how obvious these artifacts might be for images shot under complete overcast skies.

At 400% and 800% zoom the artifacts become easy to see, even for images shot under incandescent light, once there is something to compare to. But - as I’m sure everyone else already knew - there really are artifacts from using UniWB as the whtie balance for interpolating the raw file: noisier shadows (color noise) and the wrong color and brightness for highlight pixels and along edges. In the future I’ll use the correct white balance for the raw files even if it means outputting multiple versions for mixed lighting.

[RawConvert (Andrew)]

@agriggio

WB is also applied after demosaicing in RT

folks, I think I read some time back, probably on Pixls, that it’s better to do the white balancing along with the demosaicing, rather than afterwards. If you want the best quality then, perhaps one should use DCRAW and input the factors. But how do you know what they are? Can you use RT to set the white balance wysiwyg, and somehow obtain the factors represented by the temp. and tint? I can’t see an obvious place where this is. Looking at the exif in a jpeg produced by RT, there is “ColourBalance” at “10, 798, 1024, 1024…” and MeasuredColour at “12, 442, 1024, 1024…” but these don’t mean much to me and when I expand the pane, the "…"s remain.

@Elle, I’m confused when you say

@Elle

using UniWB as the whtie balance for interpolating the raw file

Why would you do this, wouldn’t you produce a green image? - but I might be confused!
Surely one would use the appropriate factors for the interpolating/demosaicing, not (1,1,1,1) ?

[agriggio (Alberto)]

@RawConvert if you are interested, the RT issue mentioned above is worth reading. TL;DR: the current method in RT should work just fine in most cases, with the added benefit of not requiring any change

I should also add that just enabling 1 step of false colour suppression gets rid of most of the artifacts you see above for RCD in RT

[LuisSanz (Luis Sanz Rodríguez)]

@agriggio

@LuisSanz

It reminds me to the same kind of effect that occurs when the white balance is applied after demosaicing.

WB is also applied after demosaicing in RT. it wouldn’t be trivial to change this I’m afraid…
do you think there’s a way to compensate for this in RCD?

note that something similar happens also in the original dcraw when you use a nonzero value for -H

I see. I also think it’s more efficient for a software with GUI to apply the white balance after the demosaicing. Otherwise every change in temperature or tint would need a full reinterpolation of the raw data or to set up a rather complex system that is probably not worth the effort.

Any non-balanced scaling of the RGB channels after the interpolation would result in that kind of color misalignement near edges, since the demosaicing cannot take the decompensation into account in order to smooth it. I’ll check if I can mitigate the issue in RCD.

[Elle (Elle Stone)]

@RawConvert

I’m confused when you say

using UniWB as the whtie balance for interpolating the raw file

Why would you do this, wouldn’t you produce a green image? - but I might be confused!

Surely one would use the appropriate factors for the interpolating/demosaicing, not (1,1,1,1) ?

There are two very different issues here:

On the one hand, people will say “You can’t white balance an image after it’s been interpolated.” This is not true. Whether you interpolate using UniWB or else use the actual white balance that you want the final image to have, as long as the image is still in the camera input profile you can change the white balance after it’s been interpolated, the colors will look just fine. Occasionally I have found it convenient to white balance an image after it’s been interpolated instead of before, for example when the lighting was mixed and I wanted to apply two different white balances.

On the other hand, not white balancing before interpolating can produce artifacts, which I didn’t realize was the case. The artifacts aren’t always obvious. My studio lighting, such as it is, is tungsten lighting, and when using tungsten white balance the artifacts from not white balancing before interpolated are not very noticeable. But if the desired white balance is daylight white balance, the artifacts are considerably more noticeable.

So at 400% zoom for tungsten white balance, I ddin’t notice anything amiss until I had the “white balanced before interpolation” image to compare to the “white balanced after interpolation” image. But at 400% zoom for daylight white balance, the artifacts are glaringly obvious.

I’m guessing it’s the high red multiplier required for daylight white balancing that is key? But this is just a guess. Maybe there is a pre-interpolation white balance that is ideal regardless of the desired output white balance? What about images shot under completely overcast skies vs. images shot under daylight/sunlight? As I don’t know what causes the artifacts I’m not able to generalize.

[afre (Alan)]

Good findings @agriggio and @LuisSanz. I had a suspicion that it had to do with wb and highlight mode. That is why I choose to compare output using that particular PlayRaw sample with its narrow spectrum, color imbalance, clipping and noisiness. I just didn’t understand what my own intuition was telling me or how to express it into words .

[Elle (Elle Stone)]

@RawConvert

folks, I think I read some time back, probably on Pixls, that it’s better to do the white balancing along with the demosaicing, rather than afterwards. If you want the best quality then, perhaps one should use DCRAW and input the factors. But how do you know what they are? Can you use RT to set the white balance wysiwyg, and somehow obtain the factors represented by the temp. and tint? I can’t see an obvious place where this is. Looking at the exif in a jpeg produced by RT, there is “ColourBalance” at “10, 798, 1024, 1024…” and MeasuredColour at “12, 442, 1024, 1024…” but these don’t mean much to me and when I expand the pane, the "…"s remain.

You can’t use RT to set the white balance and then somehow squeeze out the RGBG multipliers. Most raw processors provide several ways to set the image white balance, including setting the RGB multipliers. RT does not. I don’t know why, but maybe this is tied up with how RT pre-white-balances or something.

Across the various raw processors, the “meaning” of RGB multipliers is more or less constant, meaning setting the same RGB multipliers in darktable, ufraw, dcraw, photoflow, photivo produces pretty much the same final image colors. But as RT doesn’t provide access, you can’t share the same multipliers when comparing output from RT and other raw processors. The best you can do is hope that the presets and “camera” are the same for the raw processors you want to compare.

Unfortunately, the temperature/tint slider values for the various raw processors don’t correlate at all. The “same” temp/tint taken from one raw processor and fed to another will produce more or less different colors.

This page has summary information for various raw processors’s options for setting white balance. PhotoFlow wasn’t released yet when I put together the information: Raw Processor Review, Part 1

When using dcraw, I find it convenient to get the white balance multipliers from UFRaw and then copy them over to dcraw using the “-r” multipliers. Open UFRaw, set the white balance you want, then click on “Options” and select the Configuration tab - the white balance multipliers are available under “”.

The exif data in the jpeg comes from the raw file, and shows the white balance for various shooting conditions such as daylight, cloudy, etc, as given by the camera manufacturer. Use exiftool at the command line to examine the full contents of the raw file and the corresponding jpeg: “exiftool filename.ext”

[agriggio (Alberto)]

Hi @LuisSanz

@LuisSanz

I’ll check if I can mitigate the issue in RCD.

I hope you don’t mind if I went ahead and tried to do this my own way
I just pushed my “hack” to the rcd-demosaic branch in RT. It basically tries to compensate for the difference in highlight scaling between dcraw and RT. I would greatly appreciate your opinion on this (Also, please keep in mind that I only took some undergraduate courses on image processing a long time ago, and I’ve never worked with image manipulations since… so what I’m doing is dictated purely by “gut feelings”, and as such might be totally off!).

Anyway, for the curious, here are the results I get on the couple of test raws you posted above:

• RCD in dcraw:

• Original RCD in RT:

• Tweaked RCD in RT:

• RCD in dcraw:

• Original RCD in RT:

• Tweaked RCD in RT:

Best,
Alberto

[Carmelo_DrRaw]

I have just finished introducing the RCD demosaicing into photoflow. The implementation is based on the RT version, adapted to process small tiles in parallel.

Moreover, and following what @agriggio has reported, I have modified the RAW data clipping behaviour in photoflow, which results in much lower purple fringing around dark objects on clipped backgrounds. The improvement is actually visible for all demosaicing methods.

A small technical detail: I had to modify the code here and here to avoid inf values when the coefficients at the denominator are either equal but opposite, or very small.

Here is a comparison between Amaze and RCD with the new RAW clipping method:

Amaze:

RCD:

More details can be found in this other post on pixls.us.

For those interested, the photoflow code can be found here.

[agriggio (Alberto)]

So, inspired by what @Carmelo_DrRaw did for PhotoFlow, I generalized my “hack” above so that it works with any demosaicing method, and not just RCD. I’ve just created a new RT branch called highlight-smoothing (for a lack of a better name…suggestions welcome btw) that tries to get rid of the bad purple artifacts described above.
There’s a new option in the RAW tab, under “Preprocessing”, called – guess what? – “Highlight Smoothing”.

Here are a couple of screenshots to show what it does:

• AMAZE demosaicing, highlight smoothing turned off (the default, corresponding to the current behaviour of RT)

• AMAZE demosaicing, highlight smoothing turned on

Now, if you are interested in this, you are very welcome to try it out, but please be aware that this is very experimental. It might actually be a regression for some images, I don’t know yet. I would especially like to hear about artifacts that are introduced by this trick, and under which conditions. In particular, one area that needs special attention is the interaction with highlight compression and the various highlight reconstruction methods.

[sguyader (Sébastien Guyader)]

Will you apply this for x-trans interpolation as well?

[agriggio (Alberto)]

@sguyader

Will you apply this for x-trans interpolation as well?

Yes, this is completely unaware of the CFA pattern – it does the same for all sensor types. I have tested only with Bayer cameras though, so feedback for other sensor types is welcome!

BTW, I should also add that you need to apply this on shots with significant portions of the pixels clipped, otherwise you won’t see any difference.

[sguyader (Sébastien Guyader)]

Cool Alberto. I have some images from Venezia shot facing the sun, with strong clipped reflections on water. It will probably help in such cases.

[TooWaBoo]

Latest compiled test version ( 4b6b806 ) for Windows (x64. Gtk3.22.26) can be downloaded from here:
https://filebin.net/cru16paem7pqbcs9

No installer included. Extract the folder “RawTherapee_rcd-demosaic3” to e.g. your Desktop and run “rawtherapee.exe” inside this folder.
Cache and settings are saved into “%localappdata%\RawTherapee-rcd-demosaic”. It leaves your existing installation untouched.

[Carmelo_DrRaw]

@LuisSanz @agriggio
I have spotted a couple of numerical instabilities in the RCD implementation, or at least in the current RT version, which might lead to inf and nan values.

1. in the lines like

float V_Stat = epssq - 18.0f  *  cfa[indx] * cfa[indx - w1] - …

the epssq factor is mis-placed, and should be put either at the end of the expression or in a separate statement, something like

float V_Stat = 0.f - 18.0f  *  cfa[indx] * cfa[indx - w1] - …
V_Stat += epssq;

The reason is simple: if you add a very small number to a relatively large one (in this order), most likely the small number will not make any difference, because there are no bits available to represent it. Consider this minimal example:

#include <stdio.h>
int main()
{
  static const float eps = 1e-5, epssq = 1e-10;
  float sum1 = epssq + 0.5f - 0.5f;
  float sum2 = 0.5f - 0.5f + epssq;

  printf("sum1=%e  sum2=%e\n", sum1, sum2);
  return 0;
}

At least on my OSX system, this code generates the following output:

sum1=0.000000e+00  sum2=1.000000e-10

You can see that the first sum is identically zero…

2. when computing the direction like this

VH_Dir[indx] = V_Stat / (V_Stat + H_Stat);

I encountered cases in which V_Stat = -H_Stat, and the above expression leads to a division by zero. In my code I mitigated the problem by writing

VH_Dir[indx] = V_Stat / (fabs(V_Stat) + fabs(H_Stat));

but I am not sure if this is correct, therefore I would really like to have the option of the experts…

[agriggio (Alberto)]

hi @Carmelo_DrRaw,
thanks for the freedback! I had fixed some of these already in the latest version of the RCD branch, but it is totally possible that I missed don’t cases… I will take another look

[Carmelo_DrRaw]

As far as I can see, you have introduced two assertions for the finiteness of the values… what I am suggesting removes the risk of having inf, which should be more robust…