Magenta highlights vs raw clipping indicator vs filmic white level

@anon41087856 , hi, I’ve just built Master to try the latest changes, and I’m getting strange results with guided laplacian and the sun disc picture above. The strange things are happening as I change the iterations from 9 to 10, 11, 12 16. I’ve switched off as many modules as possible in case they were involved.
Note that 9 and 11 iterations are better than 10 and 12, that seems odd to me.
Todays’ dt is executing MUCH faster than my 1st May dt doing guided lap.

P1070278-sun-disc-16-iterations.xmp (65.1 KB)

I tried your XMP with 9 to 16 iterations, and I never get the magenta blob. Does anybody else reproduce ?

What is your hardware & driver ?

Also, why do you have rawprepare v2 ?

@RawConvert Also, when and for which commit did you start noticing this? As well, did you change the compiling method or anything in your system?

I made a recent change to rawprepare params for this PR: DNG GainMap support by paolodepetrillo · Pull Request #10289 · darktable-org/darktable · GitHub

I cant replicate on 3.9.0+1468~g0e64b4646

Hardware is an i7 processor (several years old model), no video card (I use the built-in graphics), 32Gb memory.
I noticed this just today using today’s Master build 3.9.0+1484~gdc929e1f9

I did two different things when building today ! ?

  1. I did “sudo git config --global --add safe.directory /home/user/DT-top/DT-Mast-12May22/darktable-code” just before the cmake, this seemed a good idea in the light of
    dt unknown-version? ["solved" -- sort of] - #16 by pehar

  2. I copied my previous cache and config folders. That is, I ran up dt immediately after building with new folders; then quit dt; then copied the cache and config from previous 1st May build to be the new folders. When I ran dt there was a popup about updating the database, I did that, all seemed fine.

I’ll also mention that I quite commonly get a message like this in the Terminal window -
(darktable:6146): Gtk-CRITICAL **: 14:11:54.144: gtk_render_frame_gap: assertion ‘xy1_gap <= width’ failed

where the 6146 can be various numbers. I’ve not been able to spot what causes this.

I already have, 3 times. 2 girls and a boy. :smiley:

3.9.0+1484~gdc929e1f9, Linux, ‘Release’ build.

OpenCL with NVidia 470.103.01: no magenta.

Without OpenCL: yes, reproducible.

1 Like

phew!

I can reproduce the issue only when using the CPU but not when using OpenCL.

The xmp I loaded 60 min or so ago had WB on reference with no CC module…was that my bad loading…does that impact the end result if not??

I can replicate on CPU. OpenCL it looks correct.

Interesting the Raw WP is something like 3971 which seems arbitrary but could be right…changing it to 4096 removes the issue…

EDIT

image

Changing it to 4095 effectively disables highlight reconstruction: no pixel is brighter than that value, therefore none is considered blown, so highlight reconstruction does nothing.

How is the white value determined ?? Most cameras of often it is very close to the bits of the file saved so like 1023 for 10 bit …looks like the LX-7 is 12 bit…exif data says linear limit of 4095?? Just wondering how it comes to be 3971 in DT??

Even at 4095 the new sensor clipping will show some clipped data… moving it to 4096 would do as you say…

image

@priort The tag output looks about right. Although maxes may not always be at 4095, they are usually close for this kind of camera. Are there another entries that say 3971?

I could not and its a different Panasonic camera but the Puffin picture loads with 4095 which matches the exif data…so just wondering about the deviation for the LX-7??

EDIT:
Interesting with this one…if you double click the slider it will go to 3971 if you reset the module it will go to 4095??..If you do this for the LX-7 photo …both will go to 3971…I’m curious why this would differ…must be in the camera config files??

If you check camera parameter databases, you’ll see that sensor saturation levels don’t necessarily match the theoretical maximum (4095 for 12 bits, 16383 for 14 bits, 65535 for 16 bits).

RawTherapee camconst.json:
https://github.com/Beep6581/RawTherapee/blob/dev/rtengine/camconst.json
Look for ‘ranges’:

"ranges": {
            // measured at ISO 100. ISO differences not measured, but known to exist
            "white": [ 16300, 15700, 16300 ], // typical R 16383, G 15778, B 16383
            "white_max": 16383
            // aperture scaling not measured, but known to exist, at f/1.8 the G channels hits white_max
        }

or for ‘levels’:

    { // Quality B
        "make_model": "NIKON COOLPIX A",
        "dcraw_matrix": [ 8198,-2239,-725,-4871,12388,2798,-1043,2050,7181 ], // DNG v13.2
        "ranges": {
            "white": [
                { "iso": [ 100, 125, 160, 200, 250, 320, 400, 500, 640, 800 ], "levels": [ 16300, 15700, 16300 ] }, // typical G1,G2 15760-15800 R,B 16383
                { "iso": [ 1000, 1250, 1600, 2000, 2500, 3200 ], "levels": 16300 }, // typical G1,G2, R,B 16383
                { "iso": [ 4000, 5000, 6400 ], "levels": 16200 }, // typical G1,G2, R,B 16383
                { "iso": [ 8000, 10000, 12800 ], "levels": 16000 }, // typical G1,G2,R,B 16383
                { "iso": [ 16000, 20000, 25600 ], "levels": 15700 } // typical G1,G2, R,B 16383
            ],
            "white_max": 16383
        }
    },

white_max is the theoretical maximum.

I don’t know where the same kind of data for darktable is stored.

Check this out though…so if I double click the raw white slider in any raw file right now it will go to 3971. If I reset the module it goes to what I can find in the exif data…for that LX-7 file both actions yeild 3971. I feel like this is a coincidence and that it should be 4095 as it is with the other Lumix file…

Coming from the rawspeed directory…

@kofa

Perhaps, I was making a generalization that may not necessarily apply to this model.

@priort Interesting. I see the white="3971"! Unfortunately, there is no note to say why. That is what I appreciate about camconst.