Best practice for "unexpected white balance coefficients" in darktable

[123sg (Steven)]

One point about presets (actually, here we’re talking about styles - presets are within modules, styles are like what some other software calls presets) and workflow.

@Frisco

So then the flow would be open a new photo, apply the D65 white balance (or maybe eventually auto-apply as suggested in the AP video), fix color calibration if I don’t like the default (which is usually daylight even for nighttime photos), and finally apply my preset, which would possibly include a second color calibration to do creative things?

As well as this, you often need to adjust exposure, the tone mapper and sometimes, if needed, the tone equalizer. These could be included in your styles, or you could adjust them on a per-photo basis.

Exposure is obvious.

Tone mapper, if you use filmic, as a minimum the white relative exposure usually needs adjusting (the auto eyedropper thing works well for this). Sigmoid doesn’t necessarily need any adjustments, although I like using the contrast slider in it, so I might include it as part of the style.

Tone EQ is for adjusting shadows and highlights basically - especially with a modern full frame camera like yours it’s amazing how much shadow detail can be recovered. This is probably best done where needed, so not in the styles. Although you could…

[priort (Todd Prior)]

You did you provided the coefficients and that is what I was talking about …also it makes sense that it would be less yellow you are dropping red and bumping blue with your adjustments… wrt the warnings just disable it in preferences so as not be annoyed by it…

[priort (Todd Prior)]

@Frisco

’ve ordered myself a Spydercheckr, so once I receive it I’ll also be able to see if it’s easier to calibrate the color with the D65 white balance. It still seems weird that setting the white balance to something unrelated to the lighting of the actual photo I’m editing would be helpful. Is that just a total hack, or is there some intuition behind it? I guess what is the “camera reference point” that is used by default anyway?

Its explained well in the manual…

https://docs.darktable.org/usermanual/development/en/module-reference/processing-modules/color-calibration/#white-balance-in-the-chromatic-adaptation-transformation-cat-tab

Also once you have the color card you can use the colorchecker mode of CC module and this will help with exposure wb and color all at the same time…

[Frisco]

@priort

You did you provided the coefficients and that is what I was talking about …also it makes sense that it would be less yellow you are dropping red and bumping blue with your adjustments… wrt the warnings just disable it in preferences so as not be annoyed by it…

I see, so you are saying that the numbers I reported give similar results in the modern flow to what you would get by default in the legacy chromatic adaptation flow, and that this is probably an indication that they are approximately right?

And yes, AP said use a calibrated monitor. I don’t have a calibrated monitor, but I have video lights that are probably somewhat calibrated. I tried both an elgato Keylight at 6500K, and a Luxli Celo and got almost identical results, although using the lens that I actually took the photos with (sigma 85mm dg dn), so getting numbers like 2.574 / 1.000 / 1.578 for R/G/B. This is shooting at f/7.3, focused at infinity, with my lens right up against the lights with the hood removed. I also tried the experiment with an expodisc, and got slightly redder cast (e.g., 2.657 / 1.000 / 1.465) which surprised me. However, I’m going to trust the numbers without the expodisc. I drew various rectangles to sample from around the center of the image, and got pretty consistent numbers even if they weren’t always identical.

As for suppressing warnings, I see a way to suppress all warnings in settings / processing, but might this not suppress other warnings that would be useful to a beginner like me? Is there no way to acknowledge one particular warning but still see others?

[priort (Todd Prior)]

@Frisco

Is that just a total hack, or is there some intuition behind it? I guess what is the “camera reference point” that is used by default anyway?

Not a hack at all just an advanced approach to matchup scene and viewing conditions to give the same perceived colors …

Explained well here …you can take the details that you like but starting at section 4.2 “The challenge”… the process and application are fairly well articulated

Yuhao Zhu

@Frisco

I see, so you are saying that the numbers I reported give similar results in the modern flow to what you would get by default in the legacy chromatic adaptation flow, and that this is probably an indication that they are approximately right?

Right that is what I was saying…sorry if it was unclear…you can see it by using snap shots… and comparing the legacy to the modern… no other modules active but the very basic ones that you would get from setting workflow to none… your coefficients are a much better match to that…

@Frisco

As for suppressing warnings, I see a way to suppress all warnings in settings / processing, but might this not suppress other warnings that would be useful to a beginner like me? Is there no way to acknowledge one particular warning but still see others?

You can leave them on… I don’t think there are that many warnings actually…

[Terry (Terry Pinfold)]

So I decided to have a go at processing this image and aim for the vibrant colors produced by the JPG version. I opened the image with legacy chromatic adaptations (that means just the white balance module and not color calibration module). The white balance looked immediately good as it was using “as shot” by the camera. I personally can not understand why I “must” use the new and improved color calibration module if it is creating problems, while the camera has already created an excellent white balance in the camera. Sometimes I will use the CC module, but it is not my go to workflow.

The next problem was how vibrant the colors were in the JPG. I played with color balance rgb module, which I always use to add colorfulness to the images I process. But with this image the blues looked too dull. So I took a unusual step for me and used old contrast brightness saturation module and went very heavy handed with the saturation slider. This resulted in a blue closer to the JPG. I then used shadow highlights module and local contrast module to add some more punch to the image.

I feel I have got reasonably close to the JPG if that is what is wanted. No color issues if I just stick to the traditional white balance module.

Thanks for the image and challenge.

20230106_0126.ARW.xmp (11.0 KB)

[priort (Todd Prior)]

Your seeing 2 things… one the old module comes after filmic not before so doesn’t get the gamut handcuffs… also this is what I mean about the range of possibility… v6 may somehow be in gamut or technically correct etc etc… take your edit disable the contrast module …reset filmic change to filmic v5 pres no and set latitude to around 25% and saturation to around 10… look at the difference… bright saturated blue not dark and dull…

Or even crazier…

Easily add more exposure as well… I find the picker set to 50% really often gives a good starting value …you used about .6 ev but if you go higher and tone map say 1.6

You get even more vibrant brighter blue… and this is with no intervention or CC or CB modules that could further push and pull as needed…

Edit these screen shots were your edit just disabled contrast and changed the filmic settings as mentioned and visible in the screen shot with the last one just bumping exposure…

Leave the changes alone and look at just toggling from v6 to v5 for this image… this is why i try to say when people are talking about filmic vs sigmoid…there are 2 filmic’s

[paperdigits (Mica)]

@Terry

I personally can not understand why I “must” use the new and improved color calibration module if it is creating problems, while the camera has already created an excellent white balance in the camera.

You don’t have to… and you don’t. I find this comment confusing.

[Terry (Terry Pinfold)]

@priort

reset filmic change to filmic v5 pres no

You have convinced me. There are two filmics. This is what I love about this forum. There is so much to learn from the experiences of others.

[Terry (Terry Pinfold)]

@paperdigits

I find this comment confusing

No offense meant, but there are some users (not necessarily in this thread) who are very adamant about about using the modern chromatic defaults. But what I love about DT is the choice is mine and if the camera has already done a nice white balance then that makes my life easy. However, I really like the color calibration module to color match a series of images or to apply multiple instances when there is mixed lighting.

[priort (Todd Prior)]

So I think you can still also take this potential path , legacy wb , filmic v5 no preservation, and then use CC in bypass mode but use the gamut compression and if things need tweaking and the internal filmic settings don’t hit it you can simply dial in some gamut compression and mask on the blue or hit the whole image if necessary…This way you control how much…

@Terry I am not saying that there aren’t hue shifts or gamut issues or whatever. For those that need to control these things all the choices are there but for me with v5 look at the range of possibility here and i have not done anything but set the latitude to 25 % and left it centered…This is a key though …some people might just enable v5 and say what’s the big deal well with the new defaults there is no latitude applied to you do need to reintroduce that and I think maybe many new people are not aware of how it interacts with the saturation slider…

[paperdigits (Mica)]

@Terry

No offense meant

None taken. I use the new white balancing workflow and it works great for me. But like, the defaults are just someone elses choices and you can change them, in fact darktable wants to you change them that’s why it gives you all the choice

[Frisco]

@priort

Using that general approach and filmic v5 which is my preferred version…

Would you mind posting your xmp file for that image? A couple of the XMP files others posted got the image pretty good, but I seem to be unable to replicated what people did when I edit other images. Is the v5 trick more generic, so that I could apply it to other pictures?

[priort (Todd Prior)]

I may not have kept it but you can generally get it from the jpg…

Did you mean the one I posted here… ie quite a few comments above…

Best practice for "unexpected white balance coefficients" in darktable

DT has a lot of color science…just looking at your jpg for sure the blue is likely way out of gamut… DT will fight you esp if you use v6 ie the most recent version of filmic… If you have a color chart you can do jpg / raw pairs and create a clut tone curve that will help to match them but chasing jpg colors from raw files as you might see by searching the forum is generally not worth the time and effort. You will rarely find a consistent formula… if you like the jpg… just use it …when its crap e…

If you just download the jpg that I uploaded and load it as a sidecar on a duplicate… When you do this the dialogue will only show xmp initially but you can change it to all files and load the jpg as if it was an xmp…

Edit if you look at the video ie the first one I posted you can see the impact of the gamut mapping and the muted blues when all I do is toggle from v5 to v6 and back… it should be obvious… I will try to see if I have the original xmp…I am using a dev version that should load in 4.2 but it might also have an issue…

This should be it or one version… you can make wide swings very easily as I show in my other video link…

20230106_0126_04.ARW.xmp (8.1 KB)

[Frisco]

Thanks for the xmp, @priort. With that I was able to understand a lot. It really seems like using v5 color science in filmic rgb makes a big difference, and disabling color calibration and just using the in-camera white balance also helps. I guess it makes sense that if I’m trying to start by replicating my camera’s JPEG colors, using the camera’s white balance might help.

Unfortunately, I seem to be completely unable to do this to other photos. At the risk of abusing people’s time, here’s another example that has me completely flummoxed (released under CC BY-SA):

20221106_0058.ARW (35.9 MB)

This is a photo I’d really like to edit in darktable, because I blew out the highlights in my camera’s JPEG:

Although I don’t like the blown-out windows in the JPEG, I really like the colors in the red tail light reflection, yellow window reflections, and green traffic light reflection, as well as the rich blue sky that still shows a few clouds through the tree. When I edit the file, I can get the windows to look great, and bring back a lot of detail in the drapes and such. However, the tail light reflection becomes orange instead of red, the yellow reflection is a bit washed out, the green reflection looks almost teal, and the sky is closer but still doesn’t look as punchy.

That said, I can get much closer with filmic rgb color science v5, but when I showed it to a third party for feedback, they instantly said the Sony JPEG looked way better than mine because of the colors.

For what it’s worth, here’s my best attempt so far:
20221106_0058.ARW.xmp (13.5 KB)
I found most of the things in your (@priort’s) xmp file for the blue puddle reflection helpful, except I didn’t like the sharpen, because I didn’t want too much detail in the sidewalk that is reflecting the lights.

Thanks for any suggestions. I’ve already learned so much from this thread, I really appreciate everyone’s help.

[123sg (Steven)]

Here’s a slightly rushed attempt, with xmp.
I used legacy wb (i.e no color cal for that) as that’s my usual workflow at the moment. I don’t know that it matters…
Two main things got me closer. I used the preserve hue slider dialed down to 0 in sigmoid. This allows the hues to ‘twist’ in the highlights, which many OOC jpgs do.
That was close. Then I used the channel mixer in color calibration to shift the reds around a bit. Also used color bal rgb to adjust saturation/chroma. I tried to match the jpg, so didn’t do much in tone eq about the highlights. Got to go, but see what you think. Not the same but close-er?

(upload://aHCmRaC7F4J6krc7hlKeUu0jySb.xmp) (14.1 KB)

[Frisco]

@123sg

(upload://aHCmRaC7F4J6krc7hlKeUu0jySb.xmp) (14.1 KB)

I think your xmp didn’t upload? Picture looks much closer to what I want, though, thanks. Would love to try the xmp.

[123sg (Steven)]

Sorry, I was in too much of a rush More haste, less speed…
20221106_0058.ARW.xmp (14.1 KB)

Happy to answer any questions… not sure if there’s any thing I should say about the approach I took. I suspect this should work OK on the other, reflections shot you posted too. WIll try it later.

[dqpcoxeas]

My version…

20221106_0058.ARW.xmp (17.6 KB)

I love this camera…

[Donatzsky (Nis Donatzsky)]

In the modern workflow, the legacy WB module is mostly there because something at least approximating correct WB is needed for the demosaic (and CC, I believe) to work.

@Frisco

and finally apply my preset, which would possibly include a second color calibration to do creative things?

It’s good practice to only use the first instance for WB and use new instances for other purposes. Sure, you can use the first for creative adjustments, but you could easily find yourself with a mess if you then want to tweak something.

@Frisco

It still seems weird that setting the white balance to something unrelated to the lighting of the actual photo I’m editing would be helpful. Is that just a total hack, or is there some intuition behind it?

The idea is to get as neutral a starting point as possible, which you then colour-grade using the channel mixer (R, G, B tabs in CC) and color balance rgb. This is similar to, and in fact heavily based on, the way grading film works.

This video shows it well:

Darktable Episode 62: Hue shift in highlights

I think it’s this video where AP talks about and recommends the books he used as “reference”:

[EN] Highlights reconstruction : the guided Laplacian

And for your blue lights, this should be helpful:

[EN] darktable 3.4 new module : color calibration, get your Christmas lights back in gamut !