darktable 3.0 for dummies: hardcore edition

On

I was told that properly-lit scenes were easy to process, but issues arised on under-exposed pictures. So, here is a more challenging example.

Step 0 : raw

Christmas time in Montréal, ISO cranked up all the way to the top (12800), bright blue lights that will for sure end up out of any RGB gamut, sodium-vapour surround street lighting, we are on for a lot of trouble in this picture.

Step 1 : filmic

Just enable it with default params:


Then, set up the scene middle grey luminance to 18.45% and don’t touch it anymore.

Step 2 : exposure

Raise mid-tones up to an acceptable level. We don’t care for highlights yet:

Step 3 : tone equalizer

Since we added 2.4 EV in exposure module, and tone equalizer comes after, the first thing to do is to compensate the mask by -2.4 EV so the histogram gets centered in the winlow:

Then, setup the exposure. I just hovered the white part of the balcony, and scrolled down. Then, hovered the bush in the foreground, and scrolled up. Finally, increased the curve smoothing factor just a tiny bit to round the curve a bit more (totally optional):

Pro tip: keeping the first node (-8 EV) and the last node (0 EV) to 0 EV give you more natural results, regarding the local contrast. So you end-up with a kind of sinus curve.

Step 4 : filmic again

Account for the changes in filmic, by setting the scene white exposure accordingly to the image (just use the colour picker), and then raise the scene black exposure to reduce the noise:

The resulting image is obviously too bright, we loose the nightlight feeling, so we need to raise the scene grey level and re-adjust the black accordingly:

Now the image looks quite flat, so we will raise the look contrast and reduce/slide the latitude (still trying to keep it maximal while not clipping, watch out for the orange parts of the curve):

Step 5 : color balance

We obviously have 3 sources of light: street lighting (sodium vapour, orange), house lighting (probably fluorescent, white) and the garland (LED blue). We will try to blend them a bit better, while mostly keeping the character of the scene.

  1. using the hue colour picker of the slope, compensate for highlights (pick on the balcony):

  2. using the hue colour picker of the offset, compensate for the surround (pick on the stairs):

  3. using the hue colour picker of the power, compensate for the average (pick on the the white garland):

Now, the hues are right, but we need to balance the saturation manually to blend the effect better. No trick here, trust your eyes:


Also, to help colour blending, we set the master input saturation to 90%, then the master output saturation to 110%, so the 3-ways settings are applied on a mildly desaturated image.

Step 6 : channel mixer

We still need to deal with those out of gamut blues. There is a trick I took from theACES, which involves applying a custom RGB matrix to force those blues back. Unfortunately, the ACES coefficients expect an ACES P1 colour space, and darktable pipeline works in Rec 2020 by default, so the coeff will need to be adjusted.

The settings are as follow:

  • red destination :
    • R = 1.00
    • G = -0.18
    • B = 0.18
  • green destination :
    • R = -0.2
    • G = 1.0
    • B = 0.2
  • blue destination :
    • R = 0.05
    • G = -0.05
    • B = 1.000

Here is the result:

Comparing a before/after gamut check in Adobe RGB makes the result pretty self-explanatory:


Notice we made those blues lend on the slightly purple side, we could have remapped them to cyan instead. It’s difficult to guess what the “original” blue was since no screen is capable of displaying it, and I took this picture 1 year ago.

(full disclosure: I discovered that channel mixer actually clips the RGB output to [0 ; 1], so these results are obtained with a hacked version of the code, that will hopefully be delivered in darktable 3.0.x as an “unbounded mode”).

And since we were mostly happy with the visual look before the blue lights handling, we will mask the channel mixer to apply it only to the blues shades.

Step 7 : colour balance 2

So far, we have performed mostly a corrective editing, with the technical recovery, trying to compensate for the flaws of the digital imaging system. It’s now time to bring back the Christmas groove of the picture, and warm things up. We just use a second instance of color blance:

Step 8 : color zones

Everything feels warm and cozy, but this sodium light really makes yellow pop up, and we couldn’t get rid of their greenish cast. We will just slightly desaturate and darken them, and drop down their hue a little bit:

Step 9 : denoise

darktable 3.0 makes that part way easier (thanks @rawfiner) : it’s only a matter of enabling the module in auto mode. We kept that part for the last step since it will slow-down the whole software, and also because we didn’t pixel-peep and everything looked great from the full-screen zoom:


The default params were a bit harsh, so I softened them.

Result 1

(With above settings)

Result 2

(We just changed filmic scene parameters, allowing some clipping on the white and added more contrast):

Conclusion

That was pretty much the worst case scenario you could encounter : max ISO, sort of back-lighting, 3 light sources with different white temperature, among which the infamous LED blue that is such a PITA no matter the software you are using.

Still, we managed to get everything under control in 9 steps, using “only” 7 modules. Unrolling the corrective editing first, then the creative editing, in a step-by-step fashion, we can get a better control over the picture, and come back later to any step without messing the whole stack.

Notice I didn’t use any sharpening/local contrast enhancement since the base raw has 36 Mpx shot with a prime lens and I wanted a soft Christmas postcard.

30 Likes

Just tried this on several hard-to-process images that have very blue light… and it does wonders! :smiley:
Thank you for this tip!

2 Likes

nice. could you maybe share the raw file? thanks!

Ok, the blue light channel mixer thing is amazing. I have a concert photo (that I will probably make a PlayRaw of at some point) with very bright blue light, that Lightroom just, I don’t know, dealt with… It made it work and I havent been able to do the same with darktable.
I have toyed around with putting the channel mixer in the pipeline before the input profile. It felt like a very ugly hack, but it did work a bit. And now with these values it’s perfect!

Of course, Lightroom has sliders in the ‘camera calibration’ tool that allow you to adjust the hue/saturation of the primaries. I think this is more or less the dt equivalent of that.

(on a related note: they should make LED theatre lighting illegal.)

1 Like

I have also experienced problems with blue spotlights in darktable. I have found that in Rawtherapee, if you use the dcp profile for the camera, the picture ‘just works’ as in Lightroom (although I don’t have a copy of this to try) but if you use the standard camera profile, you experience the same kind of strange problems with colours as in darktable. I am, guessing that if the colour profile is slightly out then colour extremes go out of gamut and so the channel mixer needs tinkering with. Just my guess, somebody may have a better explanation.

what camera is that? maybe RT just lacks a proper DCP profile for it.

Panasonic Lx100. There isn’t a dcp profile listed in RT so I stole a dcp profile from Adobe dng converter to get colours that look like the OOC jpeg. The RT standard ones don’t look too bad though slightly duller, sometimes I prefer them.

We still need to deal with those out of gamut blues. There is a trick I took from theACES , which involves applying a custom RGB matrix to force those blues back. Unfortunately, the ACES coefficients expect an ACES P1 colour space, and darktable pipeline works in Rec 2020 by default, so the coeff will need to be adjusted.

The ACES Blue Light LMT is meant to be applied to AP0. Not AP1. It comes from this thread: Colour artefacts or breakup using ACES - Post (DI, Edit, Mastering) - Community - ACESCentral

1 Like

Should we add this as a preset ?

8 Likes

I think so (but anyway, I made my own preset already).

Right. I know AP1 is very close to Rec2020, but I never found the actual primaries (and the white point is D60, I think). If anyone can find me the RGB ↔ XYZ primaries, then it’s just a matter of adapting the coeff with a bit of linear algebra.

We need to unbound the channel mixer first. It’s very bad to clip in [0 ; 1]

rtengine/iccmatrices.h in RT’s source code

1 Like

I just pulled DT today and your “for Dummies” posts are great - you get great results and you’re able to explain your process. QUESTION: Is filmic another downlod that I’m going to want?

All modules are included within darktable itself. Check your version - if you have 3.0, you should have module named “filmic rgb”.

Thank you @anon41087856 for posting the article. I got pretty good results following your recommendations on challenging images without humans in them. However I find it hard to do not let skin tones ‘blend in’ if there are people in such pictures. I will keep practicing though. Thanks again.:+1:

1 Like

Don’t hesitate to use masks to isolate people in difficult situations.

3 Likes

Thanks you @anon41087856. Is it possible to have a link to the raw file to practice ? Very nice job. Thanks a lot.

1 Like

Thanks a lot. Tried it and got great results instantly. Learned more about dt in a few minutes than reading through the tutorials for weeks.

@anon41087856 I didn’t quite follow how you derived those adjusted color mixer coefficients. You say the correction matrix assumes a AP-1 colour space. So, to derive a set of coefficients suitable for REC2020 colour-space, we need to transform from REC2020-to-AP1, apply the correction matrix, then transform from AP1-to-REC2020 again, right?

In that ACES code, they supply a matrix to go from AP1-to-XYZ:

0.6624541811 0.1340042065 0.156187687
0.2722287168 0.6740817658 0.0536895174
-0.0055746495 0.0040607335 1.0103391003

and a matrix to go from XYZ to REC2020:

1.716651188 -0.3556707838 -0.2533662814
-0.6666843518 1.6164812366 0.0157685458
0.0176398574 -0.0427706133 0.9421031212

So, to go from AP1-to-REC2020 we multiply [XYZ-to-REC2020] x [AP1-to-XYZ]:

AP1 to REC2020
1.04179138411768 -0.010741562648803 -0.006961875092012
-0.001683127668918 1.00036605066315 -0.001408211010916
-0.005209686580316 -0.022641445739242 0.952302414802367

and we invert that matrix to go from REC2020-to-AP1:

REC2020 to AP1
0.959937154908178 0.01046663483592 0.00703316687532
0.001622552328432 0.999685232225389 0.001490139829589
0.005290030316074 0.023825253906876 1.05016049940683

So, if the blue-correction matrix is:

correct blue AP1 space
0.9404372683 0.0083786969 0.0005471261
-0.0183068787 0.8286599939 -0.0008833746
0.0778696104 0.1629613092 1.0003362486

then the matrix to correct blue in REC2020 space should be [AP1-to-REC2020] x [correct-blue-AP1-space] x [REC2020-to-AP1]:

correct blue REC2020
0.94 0.009 0
-0.018 0.828 -0.001
0.072 0.16 1.001

But this is significantly different to the coefficients you gave:

Aurélien’s magic matrix
1.00 -0.18 0.18
-0.20 1.00 0.20
0.05 -0.05 1.00

So where did I mess up?

1 Like

The method is quite simple:

  1. input the ACES coeffs,
  2. notice they don’t look good,
  3. tweak them a little bit,
  4. iterate until it looks better,
  5. write them down with some technical words so everybody thinks you are a great color scientist, when in fact you are just a decent retoucher,
  6. wait for someone to fact-check and confess.

:slight_smile:

11 Likes