DNG HDR created in DT has "unsupported input profile"?

[hmeine (Hans Meine)]

I have taken a bracketed exposure shot in raw mode with fixed camera position and am trying to merge them in DarkTable. I am relatively new to DT and raw photography, OTOH I got a lot of technical background knowledge (physics, vision, digital image processing). My goal was to bring out the perceived “glow” of the mountains in my picture, taking during dawn, but even after quite some processing, my results looks even more dull than the built-in camera JPEG. (I guess this happens to many RAW newbies, and I take it as a sign that my camera is in general doing “the right thing” already.)

_R007447.dng (27.5 MB)
_R007446.dng (28.5 MB)
_R007448.dng (30.5 MB)

One thing I wondered about is that opening the “DNG HDR” created by DarkTable (4.0) itself results in the message “unsupported input profile has been replaced by linear Rec709 RGB”. Is this expected behavior? My input DNGs have an embedded matrix that results in a different appearance than any of the standard input profile setttings.

Furthermore, is my assumption correct that combining multiple (DNG) RAWs in the lighttable view does not depend on their history? Having just read Aurelien Pierre’s Hugin HDR workflow post from 2021, I wonder if DT just takes the raw data (someone even wrote the HDR composition takes the raw sensor sub-pixel data before demosaicing) and if the result really is linear Rec709 RGB. If I would do the HDR composite in Hugin, I could obviously choose the output profile (HDR stitching workflow profile) myself, but I assume I can’t with the built-in HDR feature?

[kmilos]

@hmeine

I am relatively new to DT and raw photography, OTOH I got a lot of technical background knowledge (physics, vision, digital image processing).

Great, so you can start w/ some troubleshooting yourself

E.g. by inspecting the output of exiftool -a -u -s -g1 on both the original DNG files and the HDR one produced by dt. The tags are explained in the DNG spec, and you want to focus on the color profile related ones obviously.

[hmeine (Hans Meine)]

Indeed, the original DNGs contain

ColorMatrix1                    : 0.6829681396 -0.2806396484 -0.03115844727 -0.3809967041 0.9970397949 0.4467773438 -0.0111541748 0.03507995605 0.7161254883
ColorMatrix2                    : 0.6279602051 -0.1450653076 -0.07955932617 -0.4613952637 1.272781372 0.2065429688 -0.06590270996 0.1471405029 0.5855102539
CalibrationIlluminant1          : Standard Light A
CalibrationIlluminant2          : D65
ColorSpace                      : Adobe RGB
ColorMatrixA2                   : 5857 2334 0 0 8192 0 0 335 7856
ColorMatrixB2                   : 13952 -4416 -1344 -1472 13792 -4128 128 -2816 10880

but the resulting -hdr.dng only has

ColorMatrix1                    : 0 0 0 0 0 0 0 0 0
CalibrationIlluminant1          : D65
ColorSpace                      : Uncalibrated

However, I don’t know if this is intended, since I am not 100% sure what the HDR composition actually does. So far, I have found my results to be superior if I just picked the “best” exposure from a bracketed shot and worked from there (often, the one preventing overblown highlights most), but I wonder if I am missing anything.

[kmilos]

@hmeine

However, I don’t know if this is intended, since I am not 100% sure what the HDR composition actually does.

Thanks for checking those out. It does deserve looking into, I’m not sure what the expected functionality is either at the moment.

[kmilos]

I think I figured out what’s going on, and it’s unfortunately not pretty ATM

Your GR III DNGs use a denominator of 65536 (2^16) for the ColorMatrix rational entries, while the rawspeed decoder is hard-coded for 10000 (the default for Adobe DNG Converter output), otherwise it gives up and so we end up with a matrix of zeros…

You might want to:

a) convert your OOC DNGs w/ Adobe DNG Converter and try dt on those again to confirm this is indeed the case, and
b) file a bug report on rawspeed so we don’t loose this @cytrinox (also need to handle the case when D65 calibration is first, and not assume Adobe DNG Converter output order as well)

Edit: a possible workaround could be https://github.com/darktable-org/rawspeed/pull/380

[cytrinox (Daniel Vogelbacher)]

Just for reference to clarify the “Adobe DNG Converter output order”. The DNG spec says:

If two calibrations are included, then it is recommended that one of the calibrations be for a
low color temperature illuminant (e.g., Standard-A) and the second calibration illuminant be
for a higher color temperature illuminant (e.g., D55 or D65).

This is true for many DNGs from Adobe Converter, but not for all. For example, some Phase One IIQ files are converted with another set of calibrations. The IQ3 100MP Trichromatic DNGs uses Flash + Tungsten, IQ150 uses A + Flash and iXU180 only D65. The Canon 1D uses only D65, too.

[kmilos]

That’s right, the spec is ambiguous enough that there’s no guarantees and one has to check all calibrations to find the D65 one, if present at all.

[Terry (Terry Pinfold)]

@hmeine

Having just read Aurelien Pierre’s Hugin HDR workflow post from 2021

where do I find this post as I am interested in reading it. Thanks

[hmeine (Hans Meine)]

Hugin pano/HDR + darktable in scene-referred

This is why I like the scene referred way over the basecurve these days. In the dark days of Lightroom having its panomerge feature you had to bake in your edits before the merge or else you’d lose access to the camera styles (I don’t remember what the exact term was, but it is where Adobe kept their version of basecurve for “Camera Neutral” or “Camera Landscape” settings) and a few other ACR gubbins. Fimic doesn’t care, linear data be linear data. TIFF, RAW, etc. Things may have changed in th…

[kmilos]

Made an alternative fix that doesn’t depend on rawspeed, please try it out: https://github.com/darktable-org/darktable/pull/12436

[hmeine (Hans Meine)]

I must admit that your explanation of the bug above was much more clear to me than how it is addressed by the PR. (Then again, I am already sweating when I read isnan()… I learned to avoid NaN whenever possible.) I will try to build DT, but it will obviously take a while.

Wouldn’t an issue at https://github.com/darktable-org/rawspeed still make sense?

Furthermore, should/could there be a unit test? My camera (Ricoh GR III) has sample DNGs in the database, so from my (outsider’s) perspective, it could make sense to implement a test that prevents similar issues for any other camera as well.

[Pascal_Obry]

@hmeine

when I read isnan()… I learned to avoid NaN

In this specific case it means (!isnan()) that the array element has been initialized, so that the actual matrix was present in the exif data. By default the matrix is initialized with NaN.

[hmeine (Hans Meine)]

Thanks, that’s how I read it. It’s just that those pesky, infectious NaN have bitten me often enough that I would not have initialized anything with it…

I must admit, building DT went much more smoothly (and quickly – this M1 is really a beast!) than expected.

The merged hdr file generated with the PR does indeed contain a non-zero matrix now:

ColorMatrix1                    : 0.628 -0.1451 -0.0796 -0.4614 1.2728 0.2065 -0.0659 0.1471 0.5855
CalibrationIlluminant1          : D65
ColorSpace                      : Uncalibrated

After removing the already imported image from the DB and re-importing, the message is not only gone, but the colors look much better!

[paperdigits (Mica)]

FYI there is room for improvement for dt’s internal “create HDR” function. If you’re planning on doing a lot of exposure bracketing, it would be worth it to set up the lua script for HDR Merge or Luminance HDR.

[LebedevRI (Roman Lebedev)]

FWIW, since the source raws were already uploaded,
it would have been nice to also have the final problematic image in the same place.