Variable white clipping points in raw file

Software
1

My sister’s 80D seems to have a variable white clipping point.

Usually, the white clipping point matches what LibRaw gives me, but occasionally when there’s a strongly clipped highlight, the raw histogram shows a spike very far from the nominal white clipping point.

For example, this photo:

IMG_0768.CR2 (26.5 MB)

has a raw histogram that looks like this:

Screenshot_20191225_233239

whereas most mildly-clipped files have a histogram like this: Screenshot_20191225_233342

It seems to happen any time there’s something extremely clipped; the special lowered clipping point seems to be relatively fixed.

Has anyone else experienced this with other camera models?

I know in RawTherapee you can adjust the raw white point to fix issues like this, but I’d like to make this automatic in Filmulator instead.

I’d like to see other examples of this from other cameras to see if I can come up with a detection heuristic.

2 Likes
2

Sometimes it comes up to a lesser extent.

For example, this D800 file I received once has this raw histogram:

Screenshot_20191226_124343

3

Looks like the sensor saturation tops out well below the 14-bit maximum reported by libraw, ~13542 in the red channel. Here are the stats for this image:

channels:
rmin:     0.030563	rmax:     0.206375
g1min:    0.030731	g1max:    0.206161
g2min:    0.030762	g2max:    0.204697
bmin:     0.030563	bmax:     0.203796

channel means:
rmean:     0.045369
g1mean:    0.043255
g2mean:    0.034956
bmean:     0.043259

These are internal rawproc 0.0 -1.0 numbers; multiply by 65536 to get the integer number read from the raw file.

On the black side, Libraw reports a black value of 2047, or 0.031234741 in rawproc terms, and the channel mins are a bit below that.

In rawproc, I recently included a mode to set the white point to the actual data maximum from all channels, and an option to set it at the minimum of the channel maximums to obliterate the whitebalance shifting, or as I’ll call it “highlight de-construction”… :smile:

4

Have a look at the camconst.json entry in RT for this camera (especially the white part):

    { // Quality B, White Levels not properly indicated, aperture scaling..missing scaling factors are guessed
        "make_model": "Canon EOS 80D",
        "dcraw_matrix": [ 7457,-671,-937,-4849,12495,2643,-1213,2354,5492 ], // DNG_V9.5 D65
        "raw_crop": [ 264, 34, 6024, 4022 ], // full size 6288x4056, official crop 276,46,6275,4045
        "masked_areas": [ 40, 96, 4000, 260 ],
        "ranges": {
            "white": [
                { "iso": [ 100 ], "levels": 16100 }, // nominal 16383, LENR blue 16243
                { "iso": [ 125, 200, 250, 400 ], "levels": 16200 }, // nominal 16383, LENR blue 16243
                { "iso": [ 160 ], "levels": 13000 }, // nominal 13097,
                { "iso": [ 320, 640, 1250 ], "levels": 13250 }, // G1,G2 13415
                { "iso": [ 500, 800, 1000 ], "levels": 16150 }, // nominal 16383, LENR ISO3200 16150
                { "iso": [ 1600, 2000, 3200 ], "levels": 16100 }, // nominal 16383, LENR ISO3200 16150
                { "iso": [ 2500, 5000 ], "levels": 13150 }, // G1,G2 13415
                { "iso": [ 4000, 6400 ], "levels": 16050 }, // nominal 16383, LENR ISO6400 16050
                { "iso": [ 8000, 10000 ], "levels": 16000 }, // R,G1,G2 16383, B 16243, LENR B 16000
                { "iso": [ 12800, 16000, 25600 ], "levels": 15900 } // R,G1,G2 16383, B 16243, LENR BLUE 15750
            ],
            "white_max": 16383,
            "aperture_scaling": [
                // need for more data to properly fill all scale factors
                { "aperture": 1.4, "scale_factor": 1.200 }, // guessed
                { "aperture": 1.6, "scale_factor": 1.130 }, // guessed
                { "aperture": 1.8, "scale_factor": 1.090 }, // 14338/13097 = 1.094
                { "aperture": 2.0, "scale_factor": 1.060 }, // 13934/13097 = 1.063
                { "aperture": 2.2, "scale_factor": 1.025 }, // 13438/13097 = 1.026
                { "aperture": 2.5, "scale_factor": 1.005 }, // 13190/13097 = 1.007
                { "aperture": 2.8, "scale_factor": 1.000 }  // 13097/13097
            ]
        }
    },
3 Likes
5

Maybe I need to have my own camconst.json…

6

Why your own?

7

A copy for Filmulator so that it can know these sorts of things.

8

Maybe we can share it via librtprocess in future? Just a thought…

9

Didn’t @darix start putting together a database for things like this?

10

I’d rather have a library separate from librtprocess to hold camera-specific data.

librtdata for data, librtprocess for algorithms?

2 Likes
11

Sounds good to me :+1:

12

For me, the key point is to avoid collecting this kind of data in different places. We should try to find one central place for this data which is accepted by all applications which use the data, even if getting the acceptence can be hard…

4 Likes
13
14

So in there we need the data itself and probably some basic methods for accessing it on a per-camera and per-image basis.

15

In my opinion, to get acceptence for RT and dt we need to make a superset of the features supported by RT and dt. For example RT supports white-level scaling based on aperture, while dt does not support that. On the other side, iirc dt supports some denoise parameters based on camera and ISO which is not supported by RT. A Database has to support all this values even if an application does not use all of them.

5 Likes
16

Yay!! Get Lensfun profiles in there and it’ll be dope!!

(Am I saying that right?..)

17

Actually yes. :wink:

1 Like
18

Maybe also RT camera/lens/aperture dependent capture sharpening settings…

19

We’re going more and more off topic. Should we create a new issue or continue the discussion in this issue? I would prefer the latter.

20

Discussion here is fine. The general camera info is more important than just the white point.

2 Likes