How do I stop RT from clipping highlights?

Software RawTherapee
1

I can’t get RawTherapee to preserve all the highlights in an image.

My image is near-white on one side of the frame, and near-black on the other side. The object and exposure was such that it shouldn’t be clipped on either end – the histogram displayed by Canon’s remote capture program was bi-modal, with a peak at either end, about 10% from the ends.

Here’s the raw file: https://filebin.net/jvrv6zikwj10apby/rtClippingDemo.CR2

I set profile to Neutral.

That displays a histogram in RT like this:

(I don’t understand why the histogram is crushed to the right.)

Then:

  • click ‘auto levels’
  • set Clip % to 0

When I click ‘auto levels’, the histogram shifts to this:

which is similar to what I saw when I took the photo – great. However, when I run the mouse over the white part of the photo, which has a slight intensity gradient due to being illuminated from the side, half the near-white part of the image is stuck at pixel values of [235, 235, 235]. I think that’s because they were clipped in the processing pipe. (Due to noise, there’s no way the image has exactly those values over such a large area.)

(The pixel values at the mouse are displayed by RT as small lines under the histogram. They flicker nicely as three closely-spaced coloured lines over part of the white area in the image but then become a single white line under the histogram – stuck at 235 – over the other half of the image’s near-white area.)

I gather RT uses dcraw. After experimenting with dcraw for a while, I think what is happening is that dcraw might be clipping the highlights (even with -h 1) before scaling. Perhaps it should instead scale first and then, if necessary, clip.

In any case, is there any way to have RT not clip the values in the image referenced above? (Or, perhaps equivalent, is there a way to get dcraw not to clip any values?)

I’m not sure what ‘Highlight reconstruction’ does. The phrase is worrisome. :slight_smile: Does it mean ‘highlight preservation’?

If RT is using dcraw, what does ‘auto levels’, 0 clipping, and ‘highlight reconstruction’ translate to in terms of dcraw options? (Is there a way at the user level to see what dcraw options are being used by any particular RT settings, as a sort of status report and helpful info?)

For a given camera model, isn’t there a fixed per-channel level corresponding to 100% saturated? If so, could RT (and dcraw) scale to that, such that pixels at, say, 95% get a final value of ‘95%’ (whatever that is after gamma), rather than scaling things to 100% or, gasp, beyond (clipping the top 0.02%)? I don’t want this image to be maxed out at white, I would like it to be as exposed ‘as shot’, at about 90% of max.

I’d appreciate any help regarding what I’m missing/misunderstanding here.

Branch: gtk3
Version: 4.2.1375
Changeset: a07f92c3793273aba07bbf50adc55b1689bccbfa
Compiler: gcc 5.3.0
Processor: generic x86
System: Windows
Bit depth: 64 bits
Gtkmm: V3.18.0
Build type: release
Build flags: -m64 -mwin32 -mthreads -std=gnu++11 -mtune=generic -Werror=unused-label -fopenmp -Werror=unknown-pragmas -msse -msse2 -mwindows -fopenmp -Wno-aggressive-loop-optimizations -DNDEBUG -O3
Link flags: -m64 -mwin32 -mthreads -static-libgcc -mtune=generic -mwindows -s -O3
OpenMP support: ON
MMAP support: ON

1 Like
2

You can read about highlight reconstruction here.

3

Thanks, that helps.

Given that info, I used ‘rawDigger’ to look at the file. Apparently it is over-exposed (!):

dcraw uses a saturation value for this camera of 13584, so those two green channels have values above that.

I’ll head back to the camera to get a properly exposed image… apologies and thanks for pointing to the manual.

4

Enabling highlight reconstruction just triggers the rebuilding of clipped values. You still need to bring those values into view. I prefer to do that by reducing exposure. For your raw, load the neutral profile, enable highlight reconstruction, set method to blend and then set exposure to -0.5 and “black” slider to somewhere about 2000. The histogram should now just about be touching each end. Now try disabling and re-enabling highlight reconstruction checkbox to see what is/isn’t affected.

Another way, keep exposure at 0 and increase the highlight compression slider to about 55.

I prefer method 1 because the exposure compensation is linear, and then I can control the tonal compression using tone curves.

I have just seen your last post. Because only one channel is clipped, the highlight recovery should be good.

1 Like
5

Setting the “Neutral” profile gives you no exposure correction at all, thus judging by your original screenshot, yes it is over exposed. Highlight reconstruction and tuning the parameters in the Exposure tool can bring some of that back; it doesn’t look like it was very over exposed.

6

hi @james, that’s the 2nd time in 2 days I’ve seen “linear” used to refer to photo adjustments, the other being @Elle’s tutorial in the “Leaves in May” thread. I’ve done some maths in the past, so I know linear means a “y=ax+b” relationship, whilst much of photography is logarithmic, i.e. each doubling of light is 1 more stop. I’m worried I’m missing something important here. I use tone curves a lot, either near the top of the exposure tools, or LAB, or sometimes CIECAM. Your comment suggests to me that it might be in some way invalid to use tone curves if you’re also making some “non-linear” adjustments, and that the highlight compression slider is one of these. Am I understanding you correctly?! And if I am, what other non-linear adjustments are there that should perhaps be avoided (in combination with tone curves)? (@Elle gave some references about colour science but I haven’t had chance to look yet).

1 Like
7

Hi @RawConvert I do not mean to imply any method or combination of tools is invalid. Use as many tools as necessary and in any combination, after all, it is the final result that is most important. I also use multiple tone curves in multiple tools and then quite often use contrast and brightness sliders too (which are also non-linear). When I was first learning to use RawTherapee, someone I have a lot of respect for described that they use exposure compensation this way for highlight recovery. So that is how I started doing it and hence is my preference. I tend to do highlight recovery early on in my processing routine.

In this case, by linear I mean if you reduce by 1 EV, then the values of all sensor pixels is halved. This does mean that whites can become grey, for some other photo editing software, exposure compensation pins the white point (top right of tone curve) to avoid whites becoming grey, which makes it a non-linear operation. For me RawTherapee’s behaviour makes most sense.

1 Like
8

I just tried this on a light-ish area of a photo with all the RT tools turned off. With no exp. compensation, the values were
R 57 G 62 B 74 L* 65. With -1EV they were :-
R 41 G 45 B 55 L* 48.
That’s not half! :slight_smile:

9

I am referring to the value of the RAW data (the sensor pixels), this doesn’t have a linear relationship with L or RGB of the output image (the image pixels), which is what you are looking at.

10

Dcraw has the “-S” switch which allows to scale the channel values. But you have to specify the amount yourself. At the bottom of Section E of my (somewhat outdated by now) review of Linux raw processors (http://ninedegreesbelow.com/photography/linux-raw-processor-review.html) is an information box labeled “True raw exposure compensation comparison” that shows how to use the dcraw “-S” switch to make 1 stop adjustments in exposure compensation. The box also compares the approaches the different raw processors take for providing true raw exposure compensation.

As an experiment, awhile back I wrote a floating point version of dcraw (http://ninedegreesbelow.com/photography/dcraw-float-c-code.html - the internal processing is floating point - the output is still 16-bit integer). This code requires that you supply a custom camera input profile and is far behind current dcraw. So most cameras in use today aren’t supported and I’m not suggesting that anyone actually compile and use this code. However, my floating point dcraw had two features that I liked a lot:

  • It automatically scaled the interpolated channel values so that the maximum channel value in the raw file is mapped to 52428 (arbitrary value below 65535) after the white balancing multipliers are applied. Does RT (or any free/libre raw processor) provide this kind of automatic scaling, that is to say, a scene-referred rendering where the image is given maximum/near-maximum exposure compensation after applying the white balance multipliers?

  • It provided terminal printout giving the number of pixels that exceed the sensor saturation value. One thing I miss in free/libre raw processors is the ability to display which channels have clipped pixels in the raw file. The (closed source) rawnalyze utility did this (this page has a screenshot in Section A: UFRaw blown highlights). Has anyone has coded up a free/libre version of rawnalyze?

11

3 Likes
12
2 Likes
13

Because it’s not linear :wink: … it’s gamma encoded …
Set RT’s output profile to one of the linear profiles of RT … RT_sRGBg10 or RT_large_g10 and then you’ll see values halving after -1.0 EV

14

@heckflosse

I see the histogram (that’s a nice histogram), but I’m not sure how to adjust the exposure compensation to see all and only the pixels that are blown in the raw file prior to the white balance multipliers being applied.

@houz

Thanks! for the link. I recompiled darktable and see the new display - that’s pretty nice! I’m have trouble correlating my sample file’s “channel data as interpolated” with the darktable indications of which channels are actually blown, but this might be user error - still working on puzzling this out.

15

OK, definitely user error: I had previously output this file (actually from darktable but quite awhile ago) as a 32-bit floating point scene-referred interpolation - white balanced but with no highlight recovery/repair. For this image I repaired the highlights manually in GIMP-2.9, and so I know from working with the file that only the green channel has clipped pixels in the raw file.

In darktable, when the daylight white balance is applied (and all the “make pretty” algorithms are disabled so the interpolation is strictly scene-referred), the “raw overexposed indication” shows clipped areas in all three channels in the clouds of my test image.

However, unchecking the darktable white balance module immediately shows that only the green channel is clipped. And correlating the CFA marks with the interpolated image in GIMP, the darktable CFA marks exactly match the clipped areas in the green channel. So that’s really nice! Is this a relatively new module?

In RawTherapee, with the daylight white balance applied, with the raw white point correction at 1.0 (which I’m guessing is equivalent to darktable’s default “3726” raw white point, which is the sensor saturation point for the Canon 400D camera), and using settings to produce scene-referred output, in my test image all three channels have areas that are marked as clipped.

If I move the RT raw white point correction to 0.67, which just allows the red channel to not be marked as clipped, then the green channel is also shown as not clipped. Is there an option in the RT interface that allows to show the actual pixels that are clipped in the raw file, independent from white balancing and raw exposure compensation? Also, is there a way to disable the white balance to show the image with the RGBG multipliers set to 1.0?

16

Yes, it wasn’t in any stable releases so far, only in the current RCs.

17

Not a bug, but very odd behavior:

I was interested in the problem posed in the initial post by @Baffin.
I tried to download and open the raw file from filebin:

First, the file downloaded was named rtClippingDemo**.tiff,** not a raw (.CR2) as advertised.

Second, the tiff file would not load in RT; it simply did not appear in the file browser. (using RT 4.2.1375, same as Baffin.) I have no problem loading several other .tiff files in my collection.

Anyone have a clue?

18

No problems here, try again. It is an EOS 600D CR2 file.

19

@Jacal
No joy. When I click on that Filebin link I get a notice:

Do you want to open or save rtClippingDemo.tiff (21.1 MB) from filebin.net?

20

For some reason, Internet Explorer does that, no problems with Firefox or Vivaldi.

If you don’t have another browser at hands, try to open https://filebin.net/jvrv6zikwj10apby and then download the file. (As a ZIP archive.)

Manually changing the extension also works, the “tiff” is an unchanged cr2 with wrong extension.