Puzzling RawTherapee Raw Histogram

agriggio · August 30, 2020, 7:46pm

This looks “fishy” to me:

Beep6581/RawTherapee/blob/390d62f6384d50300ac62ffa6871cd38f20c7435/rtgui/histogrampanel.cc#L875

    
      
          
          
if (valid) {
              // For RAW mode use the other hists
              LUTu& rh = rawMode ? rhistRaw : rhist;
              LUTu& gh = rawMode ? ghistRaw : ghist;
              LUTu& bh = rawMode ? bhistRaw : bhist;
          
          
    // make double copies of LUT, one for faster access, another one to scale down the raw histos
              LUTu rhchanged(256), ghchanged(256), bhchanged(256);
              unsigned int lhisttemp[256] ALIGNED16 {0}, chisttemp[256] ALIGNED16 {0}, rhtemp[256] ALIGNED16 {0}, ghtemp[256] ALIGNED16 {0}, bhtemp[256] ALIGNED16 {0};
              const int scale = (rawMode ? 8 : 1);
          
          
    for(int i = 0; i < 256; i++) {
                  if(needLuma) {
                      lhisttemp[i] = lhist[i];
                  }
          
          
        if(needChroma) {
                      chisttemp[i] = chist[i];
                  }

just setting the scale to 1 shows that there are “blocked” blue pixels (at least in ART). I’ve checked also with other images and the histograms are in line with what rawdigger shows (again, for ART, which has some slight changes in the way histograms are shown – I didn’t check in RT but I suppose the issue is the same).

heckflosse · August 30, 2020, 8:05pm

That’s suspicious at least…

Thanatomanic · August 31, 2020, 6:08am

This is probably an artifact from the days of old where some scaling was needed to get (most) values within the display range of the histogram. Maybe that’s not needed anymore. Will test! Thanks @agriggio

Thanatomanic · August 31, 2020, 6:32am

Things are even a little more confusing, because the drawCurve function that is called to actually draw the lines always scales values to fit the available vertical height. So, having the scale set to 1 or 8 shouldn’t even matter for the actual output (just a quick guess).

I created a GitHub issue to investigate further: Raw histogram very different and less insightful than e.g. RawDigger · Issue #5899 · Beep6581/RawTherapee · GitHub

JackH · August 31, 2020, 7:16am

That makes sense, thanks for looking into this Roel. If I understand your and previous comments correctly the main difference is in the lower granularity of the current result.

As mentioned, my interest in a raw histogram is to check the deepest shadows and the brightest highlights, in part in order to understand how much room I have to raise shadows or lower highlights with real vs reconstructed data. I am usually not concerned if a few hundreds to tens of a percent are clipped/blocked, as implied by the Auto Levels button.

Using the capture above as an example, I would consider it ‘properly’ Exposed, with clouds and non-specular data obviously fully captured, ending around DN 11500 (I like detail in clouds). @CarVac makes a good point that in this case about a stop in the highlights could probably be recovered without visible damage, so it possibly could have used one more stop of Exposure to similar effect.

Jack

Andy_Astbury1 · August 31, 2020, 11:18am

@heckflosse @agriggio @Thanatomanic
Here is a PhaseOne IQ260 raw file of a Sekonic calibration chart.
Note the ADC is 16bit, hence Raw Digger picks 8192 as EV0
Here’s Linear

Here’s log

and here is square root

In the RT histo, if you take EV0 as being the vertical line to the left of the center green peak, then there is some odd scaling going on - if that is, you take Raw Digger as gospel truth.
-1.5Ev in RT seems to scale UP as far as -4Ev, then scale down and compress the hell out of -5Ev and lower.

If any dev wants this IQ260 raw I can send it to them, together with the L*ab specs for the chart patches.

RichardRegal · August 31, 2020, 12:45pm

I did think it unlikely, but sometimes it is worth making sure the obvious easy ideas are done away with first.

Thanatomanic · August 31, 2020, 6:35pm

Thanks Andy for another example. There has been quite some discussion here and there about improving the histogram and scales and gridlines. I’m not sure we can ever get everyone on the same page on all intricacies, but at the very least the histogram should show what you expect. I believe RawDigger can be used as a sort-of ground truth for that. So clearly, RT does something else…

I would be interested to obtain your sample and specs for further investigation.

Andy_Astbury1 · September 1, 2020, 9:08am

@Thanatomanic @heckflosse
Hi Roel

I’ve uploaded the raw to Dropbox folder ‘For Roel’ Dropbox - For Roel - Simplify your life
Inside you will find the 58.6Mb IIQ raw file and a jpeg shot of the Sekonic data sheet showing patch numbers and grey values in reflectance, Ev and L* formats.

Let me know your thoughts!

I had planned on doing a much requested YT video on the RT histo but because the datum values are not marked I was going to use Raw Digger as well. It was only when running through what I was going to put in the video that I suddenly realised there was something odd going on in the left side of the histogram - it shows how little I use it in day to day work

But now the point has been raised it would be nice to see at least some scale information.

I like how Raw Digger picks its Ev0 value based on ADC bit depth.

Thanatomanic · September 1, 2020, 2:16pm

Thanks for sharing! I’ve dived into the code and I’m pretty sure what is going on. It’s actually pretty silly. RawTherapee does the histogram binning before transforming to log. It should definitely do the transformation first, otherwise you bunch up a whole lot of raw values in a single bin that would otherwise be spread out nicely in a log-transformed plot.

Andy_Astbury1 · September 1, 2020, 7:23pm

That explains the left end then!

Is it fixable?

heckflosse · September 1, 2020, 7:47pm

Toyota.

Andy_Astbury1 · September 2, 2020, 2:14pm

I never manage well with cryptic clues Ingo - I’m a big fan of simple things like yes and no

ChasingShadows · September 2, 2020, 3:26pm

In Germany the Toyota slogan is (translated to English) “Everything is possible” or maybe “All is possible” or maybe “Nothing is impossible”. Well it is OT here, but just a small advertisement with the slogan https://www.youtube.com/watch?v=LUewLaYwVe4

( Sorry, could not resist. I heard it since I was writing my original comment )

Andy_Astbury1 · September 3, 2020, 2:37pm

Ha, having owned a Supra and an Amazon in the past you would think I would have understood, but as I keep saying - I’m thick

Thanatomanic · September 7, 2020, 5:21pm

@JackH I have done some preliminary changes to the histogram. Please see here, Improvements to change main histogram to 65535 bins by Thanatomanic · Pull Request #5904 · Beep6581/RawTherapee · GitHub
Let me know what you think. Suggestions are welcome.

afre · September 7, 2020, 5:53pm

RawToyota: genuine parts, genuine service.

afre · September 7, 2020, 6:09pm

Made a comment.

JackH · September 8, 2020, 8:09am

Me too.

Andy_Astbury1 · September 8, 2020, 11:21pm

Hi Roel, the log-log looks very Raw Digger-ish Is this in dev yet by any chance?