Puzzling RawTherapee Raw Histogram

I find the Raw Histogram in RawTherapee difficult to read, or perhaps I simply do not understand it despite having researched it a bit. What are its vertical and horizontal units and range when in log-log mode? Do they make sense?

One of the main uses of a Raw Histogram (at least for me) is to check how many pixels, if any, are blocked/clipped. In the image below, it is obvious from the second and third histograms from left (RawDigger and FastrRawViewer resp.) that there is an inconsequential 1/3 stop headroom and a few blocked blue pixels. Good, that’s all I needed to know.

Why can’t I see that in RT’s raw histogram (first from left)? Am I not setting it up properly?

Jack
PS Reserve question: would it be possible to have clipped/blocked stats displayed, perhaps only after pressing a button?

2 Likes

Can you post the raw file? I’m curious how Filmulator’s raw histogram compares to those.

Voilà.

And here Filmulator’s log-linear (vertical is log) histogram agrees with RawDigger and FastRawViewer.

Screenshot_20200830_105139

A reminder that when the brightest highlights are white, such as in the clouds here, you can actually allow the green channel to clip and use highlight reconstruction with little issue.

1 Like

It is possible that the RawTherapee histogram is just of the part of the picture that is visible?

No, it’s built from the whole area

So, I was the one who introduced the log-log and log-linear modes into the histogram in RT. The log scale is defined by this transformation

vsize * std::log(factor / (factor + val)) / std::log(factor / (factor + vsize));

Where vsize is the width of the canvas, factor is determined by how much you stretch or compress (click-and-drag on the histogram) and it can range from 1 to 100, and val is the input value.
The vertical gridlines in the histogram are defined by powers of two. The horizontal gridlines are defined by equal partitions of the height (e.g. 25%, 50%, 75%). Both are appropriately log-transformed if needed, of course.

I am not sure where the difference in shape comes from. I haven’t looked at where the raw values come from and what they represent.

1 Like

This looks “fishy” to me:

just setting the scale to 1 shows that there are “blocked” blue pixels (at least in ART). I’ve checked also with other images and the histograms are in line with what rawdigger shows (again, for ART, which has some slight changes in the way histograms are shown – I didn’t check in RT but I suppose the issue is the same).

1 Like

That’s suspicious at least…

This is probably an artifact from the days of old where some scaling was needed to get (most) values within the display range of the histogram. Maybe that’s not needed anymore. Will test! Thanks @agriggio

Things are even a little more confusing, because the drawCurve function that is called to actually draw the lines always scales values to fit the available vertical height. So, having the scale set to 1 or 8 shouldn’t even matter for the actual output (just a quick guess).

I created a GitHub issue to investigate further: https://github.com/Beep6581/RawTherapee/issues/5899

That makes sense, thanks for looking into this Roel. If I understand your and previous comments correctly the main difference is in the lower granularity of the current result.

As mentioned, my interest in a raw histogram is to check the deepest shadows and the brightest highlights, in part in order to understand how much room I have to raise shadows or lower highlights with real vs reconstructed data. I am usually not concerned if a few hundreds to tens of a percent are clipped/blocked, as implied by the Auto Levels button.

Using the capture above as an example, I would consider it ‘properly’ Exposed, with clouds and non-specular data obviously fully captured, ending around DN 11500 (I like detail in clouds). @CarVac makes a good point that in this case about a stop in the highlights could probably be recovered without visible damage, so it possibly could have used one more stop of Exposure to similar effect.

Jack

@heckflosse @agriggio @Thanatomanic
Here is a PhaseOne IQ260 raw file of a Sekonic calibration chart.
Note the ADC is 16bit, hence Raw Digger picks 8192 as EV0
Here’s Linear

Here’s log

and here is square root

In the RT histo, if you take EV0 as being the vertical line to the left of the center green peak, then there is some odd scaling going on - if that is, you take Raw Digger as gospel truth.
-1.5Ev in RT seems to scale UP as far as -4Ev, then scale down and compress the hell out of -5Ev and lower.

If any dev wants this IQ260 raw I can send it to them, together with the L*ab specs for the chart patches.

I did think it unlikely, but sometimes it is worth making sure the obvious easy ideas are done away with first.

Thanks Andy for another example. There has been quite some discussion here and there about improving the histogram and scales and gridlines. I’m not sure we can ever get everyone on the same page on all intricacies, but at the very least the histogram should show what you expect. I believe RawDigger can be used as a sort-of ground truth for that. So clearly, RT does something else…

I would be interested to obtain your sample and specs for further investigation.

@Thanatomanic @heckflosse
Hi Roel

I’ve uploaded the raw to Dropbox folder ‘For Roel’ https://www.dropbox.com/sh/i4h4jziy7nlq20k/AAB6mZxtb4buh-SzbbhwexEEa?dl=0
Inside you will find the 58.6Mb IIQ raw file and a jpeg shot of the Sekonic data sheet showing patch numbers and grey values in reflectance, Ev and L* formats.

Let me know your thoughts!

I had planned on doing a much requested YT video on the RT histo but because the datum values are not marked I was going to use Raw Digger as well. It was only when running through what I was going to put in the video that I suddenly realised there was something odd going on in the left side of the histogram - it shows how little I use it in day to day work :crazy_face:

But now the point has been raised it would be nice to see at least some scale information.

I like how Raw Digger picks its Ev0 value based on ADC bit depth.

Thanks for sharing! I’ve dived into the code and I’m pretty sure what is going on. It’s actually pretty silly. RawTherapee does the histogram binning before transforming to log. It should definitely do the transformation first, otherwise you bunch up a whole lot of raw values in a single bin that would otherwise be spread out nicely in a log-transformed plot.

3 Likes

That explains the left end then!

Is it fixable?

Toyota.

2 Likes

I never manage well with cryptic clues Ingo - I’m a big fan of simple things like yes and no :thinking: :exploding_head: