Research and development - White-Balance: auto -temperature correlation - tests - finding the optimum settings

[jdc (Desmis)]

Hello

Several improvements have been brought to the Itcwb result by acting on several internal parameters (change default settings) the basic algorithm has not changed (or very little). The most striking concerns the number of spectral data, which has increased from 201 to 308.

It is from your remarks and the work done for the PR that these changes come.

Set “neutral”

it is very important for this to work correctly (taking into account the new default settings) to set to “neutral”. Of course you can load pp3 according to your habits, but the main thing is that the new parameters of Itcwb are taken into account.

Spectral color datas

I have added since a few days about 100 spectral colors. There are now 308. These data are essential for the correlation calculation to be done well. The fact of being able to choose a “sampling” of the larger image data led me to review that of the references (perhaps it is still insufficient). The system displays in the console the colors of the image that have a deltaE greater than 0.05. You can change this value in ‘options’ Itcwb_deltaspec=0.05… with 0. you see all patch.

This deltaE does not take into account the luminance and calculates the difference between the xyY (only xy) values of the image and the reference. This value seems sufficient to me, because the correlation is done on a large number of data (at least 30). But of course to check if it is not necessary to add some data.

This element is the essential point of the algorithm.

Important point, the algorithm is unchanged.

Size of color patch

I removed a slider which allowed an optimization (by iterations) of the deltaE between the xyY values to find the spectral data. By default I put the value of 2, which should be suitable in most cases.

This free “slider” is now used to choose the size of the patch. That is to say the total number of colors that can be compared and used for correlation. This number is different from “maximum color used in image”. Indeed the size of the patch is linked on the one hand to the “sampling” of the image data, and to the choice of “sort in chroma order”.

By default, before the introduction of this user setting, the patch was at 55 for the “low sampling” and “medium sampling” values and 70 for the “high sampling” and “Force used of entire CIE diagram” values.

To verify…

I added in the console, the number “data samples” at the 2 limits of the patch.

• Number of data samples in beginning patch

• Number max of data samples in last patch

The number of scanned data is that of the size of the image in pixels divided by 9. For example for an image of 45M pixels, 5M will be analyzed.

You can thus see that the value retained for each of the “x” colors retained corresponds to the average of the data. This number of data can vary, depending on the images and the size of the patch from 1000 or less to approximately 2000000.

Demoisaicing

The other important factor is by the demosaicing method (this is not a bug). Indeed “camera” takes into account the Exif data (Color matrix and white-balance camera) either those of the manufacturer, or in RT those of Adobe. It is obvious that each manufacturer has its own method of demoisaicing… but we cannot intervene on it… and this sets the default “camera” values.

The Itcwb algorithm, calculates for each color of the patch (which is an average, see above) the color which will be used for the correlation. The different methods (Amaze, DCB, etc.) have a significant impact and can lead to discrepancies, which are not bugs, but the consideration of the colors generated by these methods.

Using AWB Temperature bias

AWB temperature bias is a simple way to get a quick result.

Indeed, acting on this slider completely reexecutes the algorithm by shifting the initialization temperature. All parameters are re-calculated temperature, tint, correlation.

• it can be to readjust the colorimetry - a bit like CIECAM chromatic adaptation, but it’s not exactly the same thing.

• or to visually obtain an image more in line with your expectations, without color shifts.

Chromatic Adaptation

The results of the Itcwb algorithm, reflect the relevance of calculations on objective mathematical foundations. But this result - as indeed all settings of the white balance - do not take into account in full human perception: surround, simultaneous contrast, … and especially the adaptation of our eye / brain to temperature differences from D50 (which is the reference in colorimetry). The further away from D50 (e.g. 4300K, 6000K), the greater the deviations will be.

To overcome this gap, it is possible to set up a chromatic adaptation with the module “Color Appearance & Lighting”.

To achieve this and limit the role of Ciecam, put Ciecam in “Automatic symmetric” mode, the system will apply 2 chromatic adaptations, the first from “Scene conditions” to the reference illuminant (usually D50, but you can change it), the second from the reference illuminant to “Viewing conditions”. By default the 2 adaptation percentages are set to 90%, you can increase or decrease these values.

You can also modify the temperature in “Viewing conditions” to obtain a warmer or colder rendering.

You can, if you wish, change the Ciecam settings like “Absolute luminance”, “Surround”, etc. See the tutorial on Ciecam.
The choice is often a personal matter of perception that takes into account the environment.

Jacques

[jdc (Desmis)]

I copy the last text of the PR

I think I have finished adding spectral data, which is from my point of view, the essential point for the proper functioning of the algorithm. There are now 348 reference values. Maybe we need (a little) more, only time and difficult images will allow us to answer. If you have images with an average response, give the Raw, or if you can read the data in the console, give the values of xyY image (xx, yy, YY)
I also think that the default settings should be suitable in the vast majority of cases. In particular the 2 settings among the most constraining for the correlation:

• maximum of colors used in picture, 34 seems a good universal value
• Size of color patch, 55 seems a reasonable value.

Indeed, increasing these values comes up against 2 phenomena. The higher the values, the more the “last” data will be with a low or even very low number of pixels.
Secondly, it assumes that there are all the more coherent spectral data…
Certainly I can (further) increase the number of data (with feedback from users, for which image, etc.), but the work is (very) tedious.

So, after many tests, in particular “difficult” images (at night, with mist, badly exposed, etc.), plus all the tests made by the users (thanks to them), I think that the following adjustments must agree in most cases:

• maximum of colors used in picture = 34
• Size of color patch = 55
• Find green student = 3, maybe 2 ?
• No purple color used = unchecked
• Itcwb Observer 10° instead of Observer 2° = unchecked
• Sorted in chroma order instead of histogram = unchecked when used with “Forces use of entire CIE diagram” - to verify
• Sampling = Force use of CIE diagram
• and of course keep “Low sampling 5.9”, to ensure compatibility, even if the result is generally less good.

I think, if nobody objects, that I could ‘merge’ these changes (of course without the GUI part). We could of course continue the optimization in the PR (Sampling, sort, …)

Jacques

[nosle]

I have observed that stepping through the “maximum colours” from 10 to 50 the results roughly “loop” between a couple of tint settings whilst temperature is quite stable. (I look at tint and temp due to familiarity)

Is this expected? A max colours of 17 may look similar to 48 and a max 16 that of 43 and 48 and 43 will be very different. I don’t have the knowledge to understand the algorithms but it seems odd that the results don’t at all converge when increasing maximum colours.

You can also pick basically any combination of the other settings (size, student, observer, sort or sampling) and achieve the same wb by picking one of the max colour settings (low or high number) that give the best results.

In my experience the WB look wont always be the same if you apply the arrived at settings on a new image? Picking a new max colours number may get you there though.

[jdc (Desmis)]

The functioning of the algorithm is not simple and its operation is not always predictable, even if its designer tells you so.

In fact everything depends on the composition of the image - the distribution of colors - (and of course the color matrix). I neglect the other parameters which are “at the service” of this composition.

Of course you can’t put just anything in, hence the tweaking since 2017, and the various parameters that impact.
If you read the information in the console, you will see that the number of significant colors found (237 in total) can vary from 40 to 220. And it is not the most “colorful” images that have the most colors. It is necessary mathematically a minimum of well correlated data for the system to work (in some cases with 10 it is good…).

In the case of a “low” number, the system will not be very sensitive to variations in the cursors, and the reverse is not always true.

Like any algorithm (and this one is complex), its intelligence is limited (I don’t know if mine is limited? , or if age - I’m 75 years old - and illness are not reducing it ). The number of mathematical combinations is impressive.

It’s all about compromise. Compared to the other algorithm “auto wb” which is based on grays, this one is confoundingly simple (a few lines of code). Here in the case of Itcwb, it is at least 3000 lines of code and largely as much for the spectral data.

I would like to remind you that I was the one who wanted these adjustments (which are not very common in development). Once these adjustments are completed, no more (?) parameters will be accessible. The system will work like a black box.

Jacques

[jdc (Desmis)]

Following the exchange, in the PR and Pixls.us, I will make some changes to the algorithm

• eliminate suspicious colors that can disturb the algorithm (very high Y luminance)
• recenter the color selection inside the patch
• increase the possible deviations of Temperature, in the iteration of the joint search for temperature and tint

jacques

[jdc (Desmis)]

I just push, this change.

Unless there are big problems (crash, very bad behavior), I don’t touch anything for several days.
It’s a small adaptation of the algorithm, the values of xy images are roughly centered on the patch, instead of being at the beginning.
I eliminate (arbitrarily) certain data (this process already existed but it is enriched) which have a very high Y luminance.

Temp iteration has its amplitude doubled (but this supposes reset to neutral).

Jacques

[jdc (Desmis)]

Copy of the PR

---

Some additional information

Size of patch
There are no more settings for the size of the patch, for 3 reasons, this one is fixed, 60: a) to simplify the calculations of the useful data (distribution in the patch); b) simplify the GUI; c) and above all to be fairly sure that a reliable correlation will be found in the 348 spectral colors. Maybe we can increase (a little) this value, but is it useful?

The new “Color used in picture (preset)” setting
As I mention, I changed the algorithm a bit so that the colors selected in the patch are around the middle, rather than the beginning. This will make the algorithm (a bit) more stable when changing settings
With the default setting of 34, the colors selected range from 13 to 52 (out of 60). Why I avoid the values that have the most data, because often these colors are those of the sky, which sometimes contains quite bad data that requires a reconstruction, so you might as well avoid them
If you select 40 , the reals color selected will be 10 to 54.
Of course I can change…

Avoid data with too high luminance
It’s not as simple as it seems. The CIExy diagram, is a vertical projection of the maximum gamut, but the complete control of the gamut by xyY is not (always) simple. In order to avoid possible malfunctions, I deleted the values of Y greater than 0.96 (in theory data greater than 1 are not possible,…but sometimes there are)

Increased joint “tint” and “temp” range
I increased (a little) the value of the scan around the initial value found, the amplitude is doubled (temp). Not sure that this changes the result much (to be checked). You can change this value which is ±4 in the associated pp3 ( Itcwb_delta=4) - limited to 6
If you compile, in the console, you will see the 2 amplitudes “tint (green)” and “temp”

Rangegreen begin=24 Rangegreen end=86
scantemp begin=49 scantemp end=58
These values refer to 2 data tables matching these levels ‘24, 86 - 49, 58’ and the “tint” and “temp”

jacques

[jdc (Desmis)]

Hello

I suggest you take a break from evaluating the “wb auto”. Of course you can send me your comments, especially those that do not affect the stability of the results when moving the sliders.

I will open a new branch so as not to pollute the debate. I will give you the necessary information when it is created.

I looked into the code (from 2018) in detail and found a few hints of inconsistent arrays. Moreover I added fields to better visualize the results of the selected data and the patch.
But this does not change the overall result which always leads when changing the size of the data to unexpected results. I tried putting the data at the “top” of the patch… the malfunction is the same, if not worse.
I think it is no longer a problem of correlation, but of consistency of the data read from the image. We can also use the term “neutrality” or “balance” of the patch and the selected data. I hope to find a solution in a few time (days, weeks,?)

Thank you for your participation which is of great help to me.

Jacques

[jdc (Desmis)]

You can follow the progress in the branch “wb_research” (of course with compilation)

Jacques

[jdc (Desmis)]

Some news from Itcwb. Compilable version in « wb_research » branch

As I mentioned, I have just made various adaptations, which do not directly affect the algorithm (well, a little anyway), but the way in which the patch of data extracted from the image is designed. It is this data (this patch) that will be used to find the associated spectral colors. Then during the algorithm we vary several parameters (temperature, green, student, and other internal parameters).

When the algorithm was designed (2017 / 2018) the original idea was:

• to eliminate (or hardly take into account) large areas of flat areas (sky, etc.)
• to play on the number of selections inside the patch
• to sort the patch by (possible choice) the chroma.

It was rather data-centric with a fairly low volume of data – rather structures – hence its high sensitivity to the Raw parameters acting on it, in particular the demoisaicing method (there is nothing more normal as a behavior)

This principle has shown its limits and in the current version of the PR available in executables several changes have been made :

• very significant increase in the possible spectral data, covering I think much more than the visible spectrum (348 colors instead of 201)
• increase image analysis sampling (236 colors instead of 191), also covering the visible spectrum.
• displacement of the patch, towards average data, always avoiding large flat areas…

Again, the operation is quite random, especially when changing (with sliders) the size of the patch or the number of colors taken into account.

In the version on “wb_research” I reversed some principles:

• the patch is more focused on colors with a large number of iterations (sky, skin, buildings, etc.)
• due to the large number of spectral data, the patch size (before recalculation) is 70, modifiable in ‘options:Itcwb_maxsize=70’ which you can change if necessary between 60 and 80.
• the minimum size of the patch is accessible to the user (20 by default), too low a value can harm the correlation.
• there is no major reason - except case in point - to change these 2 values
• as these data are the latest (increasing order) there is no longer any interest in providing sorting by chroma (these are the same data).
• the choice of the size of the patch is made automatically, by seeking as much neutrality as possible (balance) by seeking by iterations a minimum value of the chroma of the patch.
• the “Ponderate patch chroma” checkbox replaces a linear reading, by taking into account (partially) the number of colors found.

To make it easier to understand I have added in the user interface, just below “correlation factor”

• “Patch chroma” average patch chroma value (xy). Attention the values in linear and weighted mode cannot be compared.
• “Size” - displays the size of the patch - often less data is retained, either the number of data found is too low or some data is “out” (Y too high)
• Patch datas x 9: Mini and Maxi, shows the number of colors found (average) for the part of the image with few colors, and that of the maximum (often the sky). To get the true number of pixels (datas) concerned you have to multiply by 9 to take into account the algorithm that “skips” the image to save processing time.

Jacques

[jdc (Desmis)]

I just merge with the wb_rearch branch.

I think this version can be evaluated

Windows and Appimage executables can be used.
https://github.com/Beep6581/RawTherapee/releases/tag/pre-dev-github-actions

jacques

[jdc (Desmis)]

I continue the development on the “wb_research” branch.

A first point concerns “AutoWB update” which worked badly (for the research version). I have just “forced” the update, so that the algorithm reexecutes each time the setting is changed.

I changed the consideration of magenta/purple. Now checkbox active or not “Filter on purple colors” (instead of - No purple color)
Allows you to filter magenta/purple data from the image. If the box is checked a filter limiting the value of Y (xyY) is applied. By default this value is 0.4. You can change it in ‘options’ Itcwb_Ypurple (Maximum 1 = no effect)

I also made configurable “Ponderate patch chroma”. You can change a coefficient in ‘options’ Itcwb_powponder=0.05. You can adjust it between 0.01 and 0.2;

Jacques

[jdc (Desmis)]

Still in the “wb_research” branch, I added 2 things:

• the number of colors read in the image. Max is 237 ;
• a median 5x5 denoise on each channel R, G, B, on the copy of the raw data before reading to build the histogram.

This increases (a little) the processing time. On my machine with a Nikon Z9 Raw, it’s about 150ms. With a RAW PEF (10Mb), about 50ms.

Jacques

[jdc (Desmis)]

After several tries, I changed the median 5X5 to a median 3x3 - 2 passes. Noise handling is apparently substantially the same, but time consumption is lower
For example on a Nikon Z9 (45.6Mp):

• without median = 50ms
• with median 5x5 = 150ms
• with median 3x3 - 2 passes = 76ms

I also improved the GUI code of whitebalance.cc a bit and cleaned up unused variables

[jdc (Desmis)]

I looked for a long time why, wbauto was not recalculating… I finally found…

Moreover there is incompatibility (I think temporary) of the current algorithm and the old one on 5.9. so I disabled it.

In front of these bugs that I hope solved… now (apart from 5.9 compatibility) it seems to work. So for this to be tested I push the change in the PR

Jacques

[nosle]

When testing with

Version: 5.9-291-g9c89c7698
Branch: wb_research
Commit: 9c89c7698
Commit date: 2023-04-22

Compiled myself and from the auto build PR I have this curious issue that “Temperature correlation” always end up very close to “Camera”.

A series of shots of the same scene end up with very different Automatic > temperature correlation wb depending on what wb I set in the camera at the scene. The image shot with shade in camera wb settings become orange also after temperature correlation. The near identical image shot with daylight becomes blue even after temperature correlation. In fact both images only deviate slightly from the multipliers of the camera settings.

My previous test were from another camera but please find attached Ricoh GR II images set to different in camera whitebalance.

The images below are Public Domain and you are free to copy, modify and distribute the images, even for commercial purposes, all without asking permission.

R0009793.DNG (15.1 MB)
R0009794.DNG (15.2 MB)

[jdc (Desmis)]

Thanks for testing

The first thing I notice in your comments is that there is no (or less) a priori color shifts, which for me is an essential point of development and stability.

The 2 images you give are instructive.

The first is substantially around T=5000K green=1. It has sunny parts and others in the shade.
The sunny part, depending on the illuminant, is often around 5000K and the shaded parts between 6500 and 8000K. The white balance only acts on a global coefficient, so either the shadows are poorly processed or the sunny part.
To remedy this 3 possible solutions

• the one that Dx0 implemented 15 years ago and which they abandoned, consists in choosing 2 zones with 2 different WBs. This is what I had done in prototype in 2018 and which was abandoned
• use “Local adjustments” - warm-cool (or the similar tool in "Local adjustments Color Appearance), and in this case, depending on the situation, warm or cool the part of the image concerned
• deeply modify Itcwb to use only part of the image and ignore the rest. Difficult to do and not sure if it is useful.

The system first looks at the “camera” values. if they are obviously “out” (green=4, temp=1500, etc.), I ignore them and start from 5000K, 1.
In other cases I start from the “camera” values. I will not describe the algo in detail, but summarily these 2 values are slightly adjusted to determine the patch and the associated spectral colors. Then we vary temp by a moderate value, roughly ±200K (otherwise the spectral data is bad) and the “green” is scrutinized roughly by ± 0.5 compared to the initial. We calculate the correlations and we keep the best ones (in fact it’s more complex, but let’s simplify).
It is therefore “normal” that in a majority of cases (see the many exceptions) Itcwb gives something quite close to “Camera”.
Now is it good? Mathematically, compared to the starting spectral data the answer is yes. But is this data relevant?
It is here that I make an intuitive hypothesis – which it is desirable to verify on real images, before I continue on this path. This assumption is that the patch choma (gap between the xy data and the white point) is minimal - I think there is probably some background of relevance.

For this you have 2 possibilities

• the linear version of chroma = it corresponds to the average of the chroma values of each data packet in the patch. And this regardless of the number of data there is
• the weighted version = it corresponds to the average of the choma values * a function of the number of datas. I tried the number itself, no convergence, the square or cubic root, no convergence. On the other hand with pow(nb, x), if x is > 0.01 and x< 0.2 it converges. What is the correct values (adjustable in ‘options’ Itcwb_powponder). For example if the minimum data for a color is 10000 and the maximum is 1000000. The weighting coefficient will vary from 1.4 (with x=0.05) and 2.5 (with x=0.2)

We will now manually, to check the relevance of this intuition, vary the patch and the spectral data by restarting the algorithm with “AWB temperature bias”. In the case of your 2nd image (Camera t=7056 g=0.955), with x=0.2 , I optimize chroma with (Itcwb T=4852 g=0.958).
You can also use “Ciecam in automatic symmetric mode” and adjust in Ciecam the 2 chromatic adaptations and the temperature.
You can also combine the 2 methods “chroma optimization” and “ciecam”

Before I continue, it is desirable to have a feedback on the relevance of “chroma”. If so, I might try making automatic (complex).

Another comment, the small denoising with median3x3 essentially aims to reduce chromatic noise to avoid spurious data and not to smooth the data.

Colorimetry is something (very) complex, there is not “one” truth, but several and I do not pretend to be the reference.

Thank you again

Jacques

[nosle]

Perhaps we’re talking past each other but my surprise is that:

1. The results are affected by the wb set in camera
2. That a purposefully wrong in camera wb gets propagated through itcwb.

Please see images below (I make them public domain) The scene is identical, shot on a tripod with only indirect natural light. The first uses camera auto whitebalance and is then processed with Neutral + temperature correlation. The following two images use extreme in camera custom whitebalance.

My expectation was that all three, being exactly the same scene, would result in the same temperature correlated whitebalance. As you can see they are very much affected by the wb metadata set by the camera even when that wb is extremely wrong for the scene.

The following are exported jpegs with wb set by temperature correlation.

R0009800.DNG.jpg.out.pp3 (13.6 KB)

R0009801.jpg.out.pp3 (13.6 KB)

R0009802.jpg.out.pp3 (13.6 KB)

Corresponding raw files:

R0009800.DNG (21.3 MB)
R0009801.DNG (21.2 MB)
R0009802.DNG (21.0 MB)

I expected auto correlation to find the correct wb regardless of camera wb. I know the tools to adjust whitebalance and appearance and do find that temperature correlation does tweak good camera wb to very good wb. However I expected the a very good wb to be found even when the shot was taken with inappropriate in camera settings.

[jdc (Desmis)]

I will have a look tomorrow
Jacques

[jdc (Desmis)]

The fact of having forced wb values to the extremes probably changed the Raw data written on the sensor significantly. Try with the 3 images in “Camera” mode to put the same “temperature” and “tint” settings, you will see 3 different images (with different histograms), but maybe i’m wrong or i’m doing the wrong thing. Still, Itcwb can’t find the same (at least I believe so), moreover Itcw was unable to take these differences into account.

Beyond fine modifications of adjustments of the algorithm, I made 2 important modifications.

• When the Exif data is obviously “out” (I have set arbitrary limits which I can change :camera - green > 1.5 or temp < 2800 or temp > 9000 ), I replaced (as in pre-process) “Camera” by “Wbauto grey”, which obviously brings another starting point, and therefore a less inconsistent result of the algorithm.

• When the values found by the algorithm (often quite close to those of “Camera”) are very far from D50 (I arbitrarily chose t < 4000 or t > 6300), I make the algorithm do 2 passes so that it recalculates both image data and spectral data. I choose between these 2 choices the one which has the best combination (“patch chroma” and “correlation factor”). This has the corollary of preventing the use of “AWB temperature bias” - (in a new commit AWB temperature bias works but a bit erratically)

I recommend (to take new features into account) to reset to “neutral” (of course you can change after “reset”).

Jacques