How to handle Filmic middle-grey deprecation?

Software darktable
1

I’m trying to figure out how to use Filmic properly and I think I get the basics but a problem has presented itself to me: Darktable is discouraging (deprecating?) the “middle grey luminance” slider, encouraging use of the Exposure module instead.

I see some comments around the forum by @anon41087856 like this one that suggest that the Exposure module is functionally equivalent to the mid-grey slider but I haven’t had much luck adapting.

The problem is that nearly always, when I expose the mid-tones to a level that I like in the Exposure module, highlights are clipped (e.g. part of somebody’s face in the sun or white fur on an animal). Then when I try to use Filmic, the image ends up garbled because Filmic can’t recover the highlights.

IIUC, since the Exposure module is earlier in the pipeline than the Filmic module, if the highlights are clipped in the Exposure module, there is nothing Filmic can do to recover them, since it doesn’t have access to the pre-clipped values.

However if I use the middle-grey slider, I can adjust the “exposure” in the Filmic module, without irrevocably clipping the highlights, ending up with an image I like much more.

Am I doing it wrong or is the Exposure module not really a replacement for the middle grey slider after all?

EDIT: After screwing around some more, this problem seems to be resolved if I use a standard color input profile instead of my usual one. If I use the standard color matrix, Filmic is able to recover the highlights even if they’re overexposed with Filmic switched off. If I use my own color input profiles, it isn’t able to recover the highlights. I may need to look at my dcamprof parameters.

2

Exposure won’t “irrevocably clip” any highlights, as it works in floating point values, where the allowed maximum value is many orders of magnitude larger than the usual range (0…1). What can happen is that values get pushed out of the range 0…1, so they can appear “clipped” in the histogram. But adjusting the white point in filmic can bring them back in the proper range without losing details or colour information.

I use exposure to get the middle gray where I want it, then adjust the white and black points in filmic to bring everything important back in “display” range. I never had problems with clipping due to the exposure module (not even with crazy corrections like +4 or +5 EV). Any zones where filmic isn’t able to get details back were at least partially clipped in the raw data,…

5 Likes
3

Yep, looks like this was a false alarm after all. When I use the standard colour matrix, everything works just fine as @rvietor described. My understanding of how things worked was wrong, as were my conclusions.

I need to refine my ICC profiles to be less broken.

4

Also if you have the Highlight recovery module on, make sure you choose reconstruct in color or lch and not “clip”

2 Likes
5

Curious to hear more about your particular use case for custom ICC profiles (e.g. which camera or device do the raws come from, how are you profiling etc.)

As far as I understand and recall dt’s code and discussions about the color management, I would say the best bet for a custom input profile is to make just a matrix profile, i.e. no LUT (or shaper curve, whatever it might be called). At least in the scene referred workflow it is best to keep the operations linear at that stage of the pipeline.

Also, nowadays it’s possible to use the Color calibration module to profile a camera using common color checkers.

6

I’m using a Sony A7RIV at the moment.

I’ve been using custom ICC/DCP profiles for a few years, after I noticed that the stock profiles in various software (I think I was using Capture One Pro at the time) handled certain colours badly (for example my wife has a red backpack that never looked right on the screen). When I tried profiling with a ColorChecker Passport, I found that the problems disappeared, so I’ve been using custom ICC profiles ever since.

I wasn’t aware of the Color Calibration module so I haven’t tried it. I believe I am using a matrix profile.

I have two variants of my ICC profiles: one linear if I’m going to use Filmic, one with an ACR curve baked in if I’m not (per dcamprof/Lumariver recommendations).

And FWIW, I’ve been using Filmic for quite a while with no problems with my other profiles. I’ve been screwing around with profiling recently and didn’t realise that that would have led to the issues I was seeing. Switching back to my old profile removes the issues, just like the standard matrix does.

7

I think I’ve figured out the specific problem with the ICC profiles: the issue comes up when you use a profile with a LUT.

In Darktable, by default, the exposure module comes before the input color profile module. When you adjust the exposure to the point where the highlights clip, the input color profile can do two things:

  • In the case of a matrix-only profile, the value just keeps going higher and higher. As @rvietor said, Darktable uses floating point values, so this is fine and Darktable can recover the highlights later.
  • In the case of a profile that includes a LUT, the value is looked up in the LUT, clamping the value to whatever the maximum value in the LUT is (probably 1.0).

I’m not 100% sure about this and welcome input from people who know what they’re talking about but this explanation makes sense to me.

The easy-looking solution to this is to just move the exposure module to after the input colour profile but I’m not sure what the impact of this is. I’m guessing that if I use a linear input profile, it shouldn’t make a difference whether exposure comes before or after. If I use a profile with a curve baked in, it’s not going to work properly if exposure comes second.

Is that right?

If so, the solution to my original problem isn’t to use the middle grey in Filmic as a weird exposure slider but to move the exposure module after the input colour profile module when using Filmic.

I tried this and it looks fine to me so far but my eyes aren’t perfect.

2 Likes
8

Some will say just use a matrix profile. In the past there were some issues with lut profiles. I think this has been resolved but I have in the past seen comments suggesting it’s best to use a matrix profile with DT. I am sure you will get feedback…in the end nothing wrong with experimenting and if you get the look that makes you happy all the better

9

it doesn’t make sense to move exposure in general just because a user can use a profile thats not suitable with the default place in the pipe.
If you’re using such profile, then you can move the exposure at a place you think it‘s better.

10

This is probably already said somewhere, but this part of your thinking is (very) wrong.

There can be loads of data past the white you are seeing now on screen (if we had a proper hdr display workflow you could still see them).

So, on your screw black is 0.0, and white is 1.0. That doesn’t mean that a pixel can have the value 1.5. I hope this is clear, but maybe that’s just for me as more of a programmer than a photographer :).

Filmic is made to pick a range of data that is larger than your display can handle, and map it to what your (intended) display can handle.

This is why older modules are being deprecated, if they didn’t play nice with ‘infinite data’ and destroyed data that was outside of display range.

All new scene referred modules are able to work on the ‘infinite data’ range. But they need to somehow know what the data means.
That’s why your are ‘supposed’ to set the middle grey point first. And then you have infinite data under it, and infinite data above it. Realistically you don’t have infinite data under middle grey, because at some point there will be absolute darkness, but that’s the idea. Set a middle grey point, so all the modules behind it have some sort of reference of what is shadows, what is mids and what is highlights and what is super bright.

I think (but I am not sure) that raising exposure or raising middle grey in filmic results in the exact same output.
BUT… That means that modules that come before filmic have no idea where your middle grey is, and they will differ in their output.

That’s why setting exposure (setting middle grey) is one of the first things to do. Then come a lot of edits to the data, and finally the data is mapped to your intended output (display sdr, print, display hdr of 500 nits, display hdr of 1000+ nits, etc…).

Of course you don’t have to follow this super strictly, and whatever works for you will work for you :). I have luts or local contrast stuff after filmic most of the time for some ‘tweaking’.

But in the early filmic v4 days (I missed v3) I also (mis) used the middle grey slider. When it was hidden by default I always clicked it back on.
Now I don’t. It suddenly clicked in my head. And now with more modules like color balance and tone equalizer I see the need to set middle grey first.

Still leaves the question I’ve seen before ‘so you set exposure with filmic on or off’ and I can’t really answer that.

Moet of the time I set exposure with filmic off. Yes, a lot of bright data will look clipped, but I’m focusing on my mids and those are not clipped.
Then I turn filmic on, the mids should be basically untouched. Now you select how much bright data you want to use, by the white slider. Cram too much data in, and it will look flat. No problem, means you just need to add some clarity or local contrast there. If you don’t see the details in the clouds for example but the clipping indicator says nothing is clipped, it means the data is there. Just do smacked close together you can hardly see the details. But they are there.

For shadows, I hardly use filmics black slider. If the shadows are too dark I use tone equalizer to bring them back. Enabling tone equalizer in the advanced tab, gives a handy mouse control where you can hover your mouse in the image on a region, and use the mouse scrollwheel up and down to brighten or darken that range.

So disable filmic, set exposure ignoring clipping, enable filmic. Set white range. Control shadows with tone equalizer. Use one of the local contrast tools to get details on the brights visible. And from then on I edit ‘into filmic’ (make other changes as required, but leave filmic on). That’s what I do, and it seems to work fine time and time again, with images with low dynamic range and images with lots of range captured.

2 Likes
11

I’m not saying you’re wrong but why doesn’t it make sense to move exposure? Why is it where it is?

12

have a look at Fix pipe order by aurelienpierre · Pull Request #2905 · darktable-org/darktable · GitHub

13

I find this unnecessary.
As filmic doesn’t touch midtones, you can adjust exposure for midtones with filmic enabled or disabled, it doesn’t matter. So I always leave it on.

14

Like I said, I can’t answer the question if you ‘should’ or not, it is what you prefer. I completely understand some people just raising exposure with filmic enabled (since filmic enabled is auto-enabled for most I guess) since that way you kinda see what it’s doing for the picture.

It’s just ‘what I do’… I prefer to focus on the midpoint and I’ll see how the picture comes together. Some people prefer another way, and I understand why :).

1 Like