Changed the limit for color temperature, now white balance is wrong when using a camera profile

EDIT: Upon further testing, all my own builds, modified or not, have this error, so it is not specific to changing the limit of the color temperature.

I have some odd RAW files that look like they require a color temperature below 1500. I changed the limit in the code to 100 for testing and it looks like it would help. The problem I run into now is that the white balance is borked whenever a camera profile is used. The settings as shot by the camera are off when using the embedded camera profile; and if I use the pick tool to set the white balance it is also off. The histogram also clearly indicates that R, G and B are not matched after setting it to a certain spot.

So I assume something is confusing the white point calculation. Does anybody have an idea where and how to fix that?

It would be very helpful if you could share an affected RAW file and a corresponding pp3 file.

Edit: and welcome to the community!

https://drive.google.com/drive/folders/1huRiqtY2WTOZPbRN9tIEJmqdWPZy67sR?usp=sharing

I uploaded a DNG, JPG, camera profile (taken in daylight, not in this scene) and the pp3 file from my modified RT build. Blue/red EQ range was also modified to go from 0.25 to 4.0. Not sure this is necessary, but makes it easier to get the white balance about right. Heavy noise reduction is set to get the color fields as uniform as possible for easier testing. The JPG is is pretty accurate as it appears to the eye.

Opening this file with the regular RT builds, you should not be able to properly set the white balance. It changes the tint, but temperature is stuck at 1500, which is too high. The photo was taken under bright incandescent light, nothing heavily tinted like sodium vapor. Definitely something that should be correctable with white balance.

I can set the white point properly in Adobe apps, at least if I change the camera profile to offset the white point. But RT does not seem to care about such an offset (the linked profile has no offset), and Adobe apps handle anything else worse. So I am stuck with nothing fully functional.

Edit: I just checked this DNG in Lightroom and it is read as 4150 temperature/+143 tint and selecting the second darkest square as white point moves it to around 3700/130.

I tried it in darktable and the problem is the same.

What happens (in darktable) is this:

  1. WB coefficients are read in the RAW file (they are written by the camera as metadata). It’s only RGB multipliers, so 3 coeffs.
  2. the software applies the input color profile (matrice) on them (matrice multiplication),
  3. the resulting chromaticity is converted into a correlated color temperature (through nasty approximations).

The problem is with step 2., since DNG input matrices are stored into the “raw” file too, and darktable lookup the matrices in its internal database. So the matrice used is wrong or non-existing, and the resulting chromaticity is wrong too.

TL; DR : it’s related to DNG files handling. Someone needs to code the case where raw = DNG.

But, again, this is the darktable case, and while I’m fairly confident that it should be the same in Rawtherapee (given that they often work the same and share some libs), it’s possible that the issue comes from somewhere else.

2 Likes

Not sure I fully understand then. Should it not work with an external camera profile then if it is an issue with handling the embedded profile? The external profile corrects the colors, except for artificial lighting when the color temperature hits the lower limit.

My guess is that it has something to do with how your phone produces the DNGs. The internal software may not really conform to the standards or do something funny.

That is well possible. The sensor uses an RYYB filter, so I would not be surprised.

Any input on why my own builds mess up the colors when using a camera profile, no matter what DNG? Even if I don’t change anything and just build stable 5.8, the colors on DNGs come out wrong. The installed version downloaded works just fine with the same files.

I am pretty confident extending the color temp range would be a workaround for me, but for that my builds need to work…

That basically explains everything. RawTherapee and darktable both assume a RGGB filter array. The “greens” are actually yellows, so the color is way off.
At the moment there is nothing that can be done about this, except to submit a sample to https://raw.pixls.us and file bug reports for camera support here and here. No guarantee that anything will happen with it any time soon though…

The embedded profile is properly used to decode the colors, but not to compute the color temperature. The color temperature is only an interpretation of the WB coeffs × color profiles. It’s really just a code path to add, the case has simply not been planned.

RYYB or RGGB is the same. First of all, none of them is what they say (there is no green, blue, red, yellow or whatever, it’s only arbitrary spectral slices), but then the mechanics are the same : multiply channels by wathever coefficients you have. If you have the right coeff for the right channel, you absolutely don’t care about what this channel is.

1 Like

I did not test, but one reason for this issue might be, that RawTherapee always uses the D65 matrix inside a DNG file (if available). Maybe for this kind of files it would be better to use the Standard light A matrix (which is available in this file btw.) ?

Is there a way I can test it myself?

Search dcraw.cc for

         if( get2() != 17 ) // 17 is Standard light A

and replace it with

         if( get2() == 17 ) // 17 is Standard light A
2 Likes

I’ll give it a shot. Problem is that my own builds are all borked right now, so I need to figure that one out first.

So this changes something. Colors are different and Temerature/Tint are read out as 1706/0.173 rather than 1500/0.87. Very promising!

Now the only problem i am left with is that my own builds don’t work properly.

Stable build, downloaded online: https://i.gyazo.com/63b5477eec8c933b7cbcec9ee5b31cf1.jpg
My own build of the latest data, no modifications: https://i.gyazo.com/c6552f7906d609b2f5c6ba88b4733bb8.jpg
Build with the modification as suggested above: https://i.gyazo.com/191274d92f3fd3aa732852749d6ffefc.jpg

You can see that there is a weird purple tint on my own builds. Any ideas how to fix that? This is my first time building anything, so it might just be a silly rookie mistake. I build for Windows 64 bit, simply following the guide on RawPedia.

Did you build using gcc 10.1? That’s a known issue. Just downgrade to 9.3

Edit: for reference:

It works also with gcc-10 of you disable the optimization option from the quoted bug.
Compiling with fno-tree-loop-vectorize is fine

I know that, as it was me, who bisected the compiler flags to that point. Though I don’t think we want to disable this vectorization…

2 Likes

Sure, it’s just a quick fix for the OP in case he cannot roll back the gcc. I have the problem because i compile on Windows with msys2. Once gcc was updated, I could no longer rollback.

I’m on msys2 as well. Rollback is no problem on msys2