Measure black and white levels?

I have been searching a little bit more and compared Rawspeed/Rawtherapee about the withe levels from 7D II. They differ, but that is an easy task to correct for someone who ones the camera.

In RawTherapee the scale factor changes the white level depending on aperture. If I want to do the same with darktable I could just save a preset in the module black/white point. Easy but time consuming.

But if I do, then the preset for aperture/ISO will also affect the black level. Is there a way to force only the white level for each ISO/aperture and keep the black level reading from the optical black area, because the black level may changes due to temperature or Long exposure noise reduction. I don’t want to have a static number there.

Black level for Canon EOS M5.

Cropped out in Rawspeed
<Crop x="266" y="38" width="0" height="0"/>

Of this the following is used to get the black level

<Vertical x="4" width="260"/>
<Horizontal y="4" height="30"/>

But, at high ISO something is happening at the top of the raw file, 12 px down from top and 264 px from the left.

For Canon EOS M5, I therefore skip 12 px from top and start at line 13.

<Vertical x="4" width="260"/>
<Horizontal y="13" height="21"/>

CC0 20220909IMG_1882.CR2 (39.2 MB)

Command to generate the pgm file above $ dcraw -E -4 -j -t 0 -s all 20220909IMG_1882.CR2

cameras.xml is in Linux found at /usr/share/darktable/rawspeed and in Windows at C:\Program Files\darktable\share\darktable\rawspeed

So, black levels seem easy to get thanks to the optical black fields.

Now the white level. Sample raw file from 6D at ISO 6400 and at f/7.1

Canon Exif reports:

Normal White Level              : 14558
Specular White Level            : 15070
Linearity Upper Margin          : 10000

Adobe DNG Converter reports the same as Canon.

Magic Lantern reports

15189

Rawspeed that darktable uses reports

15000

Skärmbild från 2022-09-19 12-56-20

RawTherapee reports

15100

In darktable and using the raw highlight clipping indicator and stepping down from 16384, 15283 is what makes it to react. Note that 15283 was mentioned in RawTherapee.

So, which one to use in this case? Anything below 15283 should be alright and above might give artifacts when reconstructing highlights?

CC0 20220917_MG_2399.CR2 (25.0 MB)

I would use the 15070. Recorded info above this point (if any) would be treated as clipped in the HL module.

I have a similar question, especially because the reported white levels are so different. This is for a Canon EOS 7D Mk II.

dt:

image

XnView MP:

image

Should I change the dt setting to 12277?

Just going by the EXIF names, I’d interpret those numbers as:

  • Linearity Upper Margin: “from here on, signal is no longer proportional with light energy” (probably less signal than expected, and/or differences between the different colour channels)
  • Normal White Level: “white surfaces (like paper) should be at or below this level”
  • Specular White Level :“beyond this value you shouldn’t expect any detail” (i.e. the sensor is saturated). Somewhat higher values can still appear due to noise, and perhaps a small safety margin in the value.

That would also fit with the higher white points for the extreme ISO values, where you reach the 14 bit limit.

From this, and given the small difference shown between normal and specular white level, I’d use the specular white level as reported by the camera for the raw white point, assuming Canon (like other makers) know their own cameras… Assuming those values are fixed for each camera and ISO value, it’s not that hard to set up one or auto-applied presets for this.

And if you are doing extremely critical work, where linearity is of the utmost importance, you’d have to make sure all pixels are below the linearity limit (10 000).

All that of course once you have reason to think the defaults proposed by your program are wrong…

3 Likes

The value will depend on the ISO and I think on the raw format (sRaw).

edit:
It is still very possible that rawspeed or RawTherapee have incorrect values. I agree with @rvietor and use the Specular value as reported by the camera. If the values are incorrect for a given ISO, I think you should do an Issue to both rawspeed and RT to get it fixed.

Canon white level depends on ISO and/or aperture larger than f/4. Rawspeed can only change white level due to ISO and not aperture.

I’ve checked two Play Raw photos, one from a Canon EOS M6 Mk II, and dt matched the exif data. More investigation is needed.

It would be wonderful if camera manufacturers explained exactly what they meant by their various black and white levels.

Meanwhile, we can explore levels for ourselves with our own cameras. Point the camera at an evenly lit card, vary the exposure (assuming the aperture and shutter speeds are accurate) and examine the numbers in the raw file. Over what range are they linear with respect to the exposure? Are the ratios between R:G:B constant? If the linearity and ratios are not perfect, are they “good enough”? Or can they be fixed in post?

The desired accuracy will depend on the nature of the photography, of course. Do we need scientific precision, or aesthetic usefulness?

2 Likes

30D and optical black area. Rawspeed skips 2 lines from top and uses the 8 lines down. From left it uses all 72 columns.
Skärmbild från 2022-09-21 10-54-56

When checking the optical black area at ISO 1600, There is a brighter line 2 px down from the top. At line 10 there is a grey line and can’t be used for black level.
From left, column nr 72 is grey and can’t be used for black level.

So if your camera has optical black area, take a look yourself because it may differ at different ISO.

BTW, is that a dead pixel in the corner 10 lines down and 72 columns right?

Exif or Rawspeed black levels, which one is best for 7D with these files?

CC0 license.

20230120_M7D6709.CR2 (21.6 MB)
ISO 100 Long exposure noise reduction on. Exif and Rawspeed almost same. Exif 2048 2048 2049 2048 Rawspeed 2048 2048 2048 2048

20230120_M7D6717.CR2 (33.3 MB)
ISO 12800 Long exposure noise reduction on. Exif 2053 2052 2044 2044 or Rawspeed 2040 2040 2019 2019

20230120_M7D6692.CR2 (21.6 MB)
ISO 100 Exif and Rawspeed almost same. Exif 2047 2047 2048 2048 Rawspeed 2048 2048 2048 2048

20230120_M7D6699.CR2 (32.2 MB)
ISO 12800 Exif 2034 2033 2060 2057 Rawspeed 2059 2059 2047 2047

Optical black area for the camera
optical-black-area.xcf (31.3 MB)

The largest difference you report on those files is 25 on ~2050. That is 1.2%.
Is that relevant, i.e. do you see a difference when manually forcing the one or the other value?

I see a clear difference, yes. But I don’t know which one I prefer.
There is an issue at Github talking about using black level from Exif instead of optical black. Better or not?

About white level. I checked here: Adding Support for New Raw Formats - RawPedia

I took pictures with clipped highlights and loaded them in RawTherapee. Thanks to the user guide I could use Navigator to get the raw values Editor - RawPedia
The problem is the part *

The Navigator will show the real raw photosite values after black level subtraction within the range of the original raw data.

So to reset black level that in this case was 512 I just wrote -512.

Now I could check the right white level that is 16382 at this ISO and aperture.

In darktable I loaded the same raw file. For R6, darktable (Libraw?) uses white level Canon writes in Exif (Specular White Level: 14008)

If I activate raw clipping indicator and put white point to 16384 there is no clipping.

But if I move it back a little bit to 16382 I get raw clipping.

So in this case both RawTherapee and darktable shows that the highlights are clipped at 16382.

I tried the same but with Long exposure noise reduction. This is at ISO 1600, but it looked the same at all ISO.

More about R6 from @Thanatomanic here Canon R6 CR3 support · Issue #5901 · Beep6581/RawTherapee · GitHub


Latest Hasselblad has optical black area at three sides.
@LebedevRI, three questions:

  1. is it possible to feed Rawspeed also with the third side, the right one?
  2. if yes, is it of any real use?
  3. the top left corner is bright and not optical black. Will it disturb the calculation of optical black with my BlackAreas settings below?

As far as i’m concerned, black area handling is completely broken
in rawspeed, and i’m not even talking about the cameras.xml part.

1 Like

That’s why you have implemented reading from Makernotes I suppose?

Not really. It’s just that it’s quite not good to hardcode stuff in cameras.xml
that is already in Makernotes. Black area handling needs to be fixed regardless.

1 Like

Just now looked at the link - wow, a lot of work went into that, by golly!

I downloaded it for info and it opens legibly in Coffee Cup (as opposed to NotePad).

Thanks!