Measure black and white levels?

I took some raw files of my Spydercheckr and with my Canon 300D, D30, D60, 30D, 1Ds, 7D, 6D, M5 and compared them after I did a color calibration with darktable. The older camera models like D30, D60, 1Ds had some more greenish shadows than the others. I know the three are missing optical black areas and I wonder if that could be the reason in this case.

If I take some pictures with a D30 with lens cap on and with the same shutter speed I use for the real photo, whould I be able to measure the black levels from that?

And what would be the best way to measure white level?

And what would be the best way to measure, more than using RawDigger that is not FOSS?

I will upload raw samples next time I am at my computer.

1 Like

The camconst.json file with RawTherapee has some good information on divining black and white levels:

https://github.com/Beep6581/RawTherapee/blob/dev/rtengine/camconst.json#L113

Python perhaps: rawpy followed by numpy analysis (and matplotlib for histograms etc.)

1 Like

Checked that earlier today. RawDigger was mentioned there. But yes, very interesting.

Could be an idea.

All raw files but 6D here. It seems D60 has less greenish when using the same black level as D30, Around 127 instead of 134. CC0 license.

20220801.zip (70.4 MB)

Have you looked for the black level in the metadata? I’ve seen one case where it wasn’t quite right (Playraw here, somewhere), but my experience with a lot of raws is that it works…

Edit: post a raw, and I can poke at it with rawproc…

Raw files added in the zip file.

I checked Exif with Exiftool. It actual says 126 127 126 127.

Perhaps a tool I already am familiar with could help…

Yep, here’s the *BlackLevel tags from all the files:

glenn@bena:~/Photography/2022-08-09 Peter's Black Levels$ exiftool -*BlackLevel* *.CR2
======== 20220801_M7D6531.CR2
Average Black Level             : 2048 2048 2048 2048
Per Channel Black Level         : 2047 2047 2048 2048
======== 20220801_MG_6917.CR2
Per Channel Black Level         : 128 127 128 127
======== 20220801_MG_9392.CR2
Average Black Level             : 2047 2047 2047 2047
Per Channel Black Level         : 2047 2047 2047 2047
    3 image files read
glenn@bena:~/Photography/2022-08-09 Peter's Black Levels$ exiftool -*BlackLevel* *.CRW
======== 20220801CRW_6901.CRW
WB RGGB Black Levels            : 125 123 125 123
======== 20220801CRW_7097.CRW
======== 20220801CRW_7108.CRW
Black Levels                    : 126 127 126 127
    3 image files read

Interesting, I’ve not yet seen different per-channel black levels from a camera.

I’ve just opened each raw file with rawproc with the default processing, which subtracts the libraw-supplied black level, and they all look fine. i took one and skewed the subtract value back and forth; green-ish is too big a value, magenta-ish is too small.

Me, I’d just go with what the camera gives you…

For example 1Ds it doesn’t give me anything. But if I take a shot with the same shutter speed and temperature and lens cap on I should be able to use Hraw even if 1Ds is missing the optical black area, because the whole sensor area would act like one when no light reaches the sensor.

Need to try tomorrow.

Ah, missed it, the D30 doesn’t have a black level tag anywhere in the metadata. libraw reports 127, and I used the black margin of the raw to confirm that number. Now, I can’t find any white balance tags, and libraw delivers an invalid set (-1.024,1.000,1.066), so I did an auto calculation based on a patch from the neutral card and came up with this: 2.139,1.000,1.066. Here’s the linear image with this white balance, a bit dark but the blacks and color look right:

Edit: note the black margins at the left and top; these are the 30D masked sensor, and where I confirmed the black level.

Panasonics have this.

1 Like

EOS D60 and measured with Hraw. I took a picture with lens cap on and measured 74 pixels from left.

ISO 100. From Exif 127 126 127 126

ISO 1000. From Exif: 132 129 132 129

CC0
D60.CRW (4.2 MB)
D60ISO1000.CRW (5.6 MB)

Adobe DNG Converter reports for D60 and ISO 100
126 127 126 127

Adobe DNG Converter reports for D60 and ISO 1000
132 129 132 129

With D60 and at ISO 1000 darktable default black level

With D60 and at ISO 1000 with measured black level

It seems like the problem is in Rawspeed at this part

I didn’t think D60 had optical black area, but indeed it has. I inspected it and it is a little bit different from the other Canon cameras I have seen. From top it is pitch black 7 px down. Then grey 5 px down. Unusable in this case for black level.
From left there is a different area 12 px deep, and after that there is, what I think, the optical black area to use. Because of the first 12 px I think I have gotten the wrong black levels from Hraw above.

I changed in my local Rawspeed xml file to

PGM file to load into GIMP and to bright up a lot CRW_7112_0_PGM.zip (7.3 MB)

Now with the new corrected values in Rawspeed it gives me

I loaded D60 ISO 1000 in Canon Digital Photo Professional 4 and the colours look a little bit off or over saturated in the shadows even with the neutral profile. Perhaps Canon Digital Photo Professional 4 only uses the black levels from Exif and not measure from the optical black area?

For D30, the optical black area are 6 px from top and 48 px from left. I changed in Rawspeed to

For 1Ds at ISO 100 I got:

2 Likes

I noticed this issue with my old Canon D60 (which I no longer have, but I have many raw files from stil) and reported the issue a year ago.

BTW: The Canon D60 is not to be confused with Canon 60D, which came out a few years later.

darktable has oddly different values for the black levels than what the camera reports in the raw files. When I change it to what’s recorded in most of the files (something like: 127, 128, 127, 128) according to exiftool, then that fixes the results in darktable… so I made a preset for the black levels and have it auto-apply for my D60 files. That basically fixed the problems with the dark parts of the images and the oddly green color here.

It’s important to note that auto-whitebalance images from these cameras are fine. It’s when you have a custom whitebalance set that things go haywire. (And because the D60 had miserable auto-whitebalance, I almost always manually set it with a grey card, so most of my images are affected by this.)

You can try my configuration in Rawspeed, cameras.xml. Then you should get the most out of it no matter which ISO you use or what temperature it is at the moment.

I have been searching a little bit more and compared Rawspeed/Rawtherapee about the withe levels from 7D II. They differ, but that is an easy task to correct for someone who ones the camera.

In RawTherapee the scale factor changes the white level depending on aperture. If I want to do the same with darktable I could just save a preset in the module black/white point. Easy but time consuming.

But if I do, then the preset for aperture/ISO will also affect the black level. Is there a way to force only the white level for each ISO/aperture and keep the black level reading from the optical black area, because the black level may changes due to temperature or Long exposure noise reduction. I don’t want to have a static number there.

Black level for Canon EOS M5.

Cropped out in Rawspeed
<Crop x="266" y="38" width="0" height="0"/>

Of this the following is used to get the black level

<Vertical x="4" width="260"/>
<Horizontal y="4" height="30"/>

But, at high ISO something is happening at the top of the raw file, 12 px down from top and 264 px from the left.

For Canon EOS M5, I therefore skip 12 px from top and start at line 13.

<Vertical x="4" width="260"/>
<Horizontal y="13" height="21"/>

CC0 20220909IMG_1882.CR2 (39.2 MB)

Command to generate the pgm file above $ dcraw -E -4 -j -t 0 -s all 20220909IMG_1882.CR2

cameras.xml is in Linux found at /usr/share/darktable/rawspeed and in Windows at C:\Program Files\darktable\share\darktable\rawspeed

So, black levels seem easy to get thanks to the optical black fields.

Now the white level. Sample raw file from 6D at ISO 6400 and at f/7.1

Canon Exif reports:

Normal White Level              : 14558
Specular White Level            : 15070
Linearity Upper Margin          : 10000

Adobe DNG Converter reports the same as Canon.

Magic Lantern reports

15189

Rawspeed that darktable uses reports

15000

Skärmbild från 2022-09-19 12-56-20

RawTherapee reports

15100

In darktable and using the raw highlight clipping indicator and stepping down from 16384, 15283 is what makes it to react. Note that 15283 was mentioned in RawTherapee.

So, which one to use in this case? Anything below 15283 should be alright and above might give artifacts when reconstructing highlights?

CC0 20220917_MG_2399.CR2 (25.0 MB)

I would use the 15070. Recorded info above this point (if any) would be treated as clipped in the HL module.

I have a similar question, especially because the reported white levels are so different. This is for a Canon EOS 7D Mk II.

dt:

image

XnView MP:

image

Should I change the dt setting to 12277?

Just going by the EXIF names, I’d interpret those numbers as:

  • Linearity Upper Margin: “from here on, signal is no longer proportional with light energy” (probably less signal than expected, and/or differences between the different colour channels)
  • Normal White Level: “white surfaces (like paper) should be at or below this level”
  • Specular White Level :“beyond this value you shouldn’t expect any detail” (i.e. the sensor is saturated). Somewhat higher values can still appear due to noise, and perhaps a small safety margin in the value.

That would also fit with the higher white points for the extreme ISO values, where you reach the 14 bit limit.

From this, and given the small difference shown between normal and specular white level, I’d use the specular white level as reported by the camera for the raw white point, assuming Canon (like other makers) know their own cameras… Assuming those values are fixed for each camera and ISO value, it’s not that hard to set up one or auto-applied presets for this.

And if you are doing extremely critical work, where linearity is of the utmost importance, you’d have to make sure all pixels are below the linearity limit (10 000).

All that of course once you have reason to think the defaults proposed by your program are wrong…

2 Likes

The value will depend on the ISO and I think on the raw format (sRaw).

edit:
It is still very possible that rawspeed or RawTherapee have incorrect values. I agree with @rvietor and use the Specular value as reported by the camera. If the values are incorrect for a given ISO, I think you should do an Issue to both rawspeed and RT to get it fixed.