Color matrix for a DNG file - darktable Windows 3.0.0RC1

#1

I’m just trying to view some DNG files in lighttable, but the colours are way off and I’m getting the error “Xiaomi Mi A2 color matrix not found!”.
The original photos were jpegs from a smartphone, then a panorama created and saved in DNG format. The smartphone jpegs are displaying just fine, but the panos in DNG format are not.

I can get ok results if I go into darkroom and turn off the default modules applied, e.g. white balance, base curve, highlight reconstruction, etc., then changing the input color profile to linear prophoto RGB. But I don’t actually want to edit the photos, I just want to view them in lighttable.

As the smartphone jpegs are showing up fine, can I tell darktable to use the same color matrix as the jpegs?
And is darktable thinking the DNG is a raw file, which is why it’s applying the default modules? And if so, is there a way to tell it that it’s not actually a raw file?
IMG_20190113_143422-Pano.dng (58.1 MB)

#2

I think you’ll have better luck if you save your panorama as a jpeg. Not a solution, but when you bring a jpeg into DT, it won’t apply the base curve, etc like it does for a raw.

#3

Yep, I will do from now on, but for my existing files I’d rather not compress them further. And I’d also like to do the conversion in darktable if possible. But can’t with the current colors.

#4

You should be able to extract the jpeg from the dng with exiftool.

#5

Downloaded your DNG, RawTherapee opens it okay. My hack software did not, but I use Libraw on DNGs and only to fish out one mosaiced channel. Darktable may be operating under a similar assumption, that it’s a camera raw mosaic.

Also ran exiftool to look at the metadata; looks like you made the pano in “Lightroom Classic 8.1 (Windows)” - that software wouldn’t let you save a TIFF? Darktable can ingest that, I’m pretty sure…

#6

Yes, the pano was done in Lightroom. I’m sure TIFF was an option, but I made it back when I thought I’d be using Lightroom forever, and I just used the default.

So, I guess the question is whether this is a bug or a case of “unsupported”…

#7

You might consider using RT to open the DNG just long enough to save it as a 16-bit TIFF. I’m at the wrong computer, or I’d do it.

#8

Now at the correct computer. Here 'tis, 16-bit compressed TIFF:

https://glenn.pulpitrock.net/Individual_Pictures/IMG_20190113_143422-Pano.tif

It’s on my personal web server; let me know when you have it. I’ll need to remove it after a bit, it’s big…

#9

I can do it no problem, and I do use RT as well. I’m just trying to work out if darktable can handle it. Ultimately, I’d like to use just one solution for 90% of my photography work. At the moment, I keep flitting back and forth between RT and darktable because of functionality that I need that only exists in one of them. RT certainly seems to have the fewer problems with colour management.

#10

Oh, a little light just came on. I had a Topaz-processed DNG, modified while using LR. dt was unable to open it, Irfanview can only see it as a thumbnail… which occupies 148Mb.

So it’s possible that DNG’s created inside LR using plug-ins etc may modify the exif in such a way that they can no longer be read with a program other than LR (or PS, probably). Looking in the exif of that file, the “flags” line is missing an “r”… I suspect this is a modification of the read permission?

Thanks for the tip :slight_smile:

#11

My experience was that darktable spit out that error because it couldn’t find a color matrix match in its internal database, but then went ahead and used the color matrix embedded in the DNG file. (It definitely DOES use the built-in matrix within the DNG, or at least did back in August)

That was on my list of things to investigate before I switched tools. Fairly certain it’s just a nuisance error popup.

The Topaz-processed DNG might be demosaiced “Linear DNG” - I recall seeing some reports that dt had issues with some of the more esoteric methods of saving image data. I’m 90% certain it’s not an EXIF issue.

Edit: Also to @europlatus - as far as the colors being way off, it may not be that darktable is doing anything wrong, it’s more likely (being a Xiaomi camera product) that they completely and totally botched the color matrix metadata in the DNG. The Mi Sphere 360 camera has that EXACT issue. The matrix in the DNG for the Mi Sphere is SO broken that you get superior colors by completely turning the color matrix off. Even if you inject a “good” color matrix with exiftool (for the mi sphere, you can pull an acceptable one from the output of Yoichi Hirota’s Mi Sphere Converter), darktable still spits out the color matrix not found errors when it’s clear the embedded color matrix is being read.

The broken DNG matrix has been documented for well over a year and MAdventure/Xiaomi still haven’t fixed it.

Based on comments I see on 360rumors, cameras from the Asian budget brands have broken DNG color profiles on a regular basis, to the point where the guy at 360rumors assumes severely broken color management is just how raw files work.

1 Like
#12

Stitched panoramas sometimes have small glitches. These can be smeared areas at image transitions or parts that appear multiple times.
In many occasions these corrections are easier done in tools like GIMP than in a RAW processor. DNG is not a good format for that since it’s not made for being edited.

Therefore I always export stitched panoramas as PNG or TIFF.

In the case of the example image above there is a glitch in the top of a tree right to the cables to be corrected. If you want to invest more time and effort you could work on the crook cables.
But imho this is a job for TIFF/PNG and GIMP, not for DNG and a RAW processor.

#13

Thanks for your reply.
The Xiaomi phone only saves jpegs and these are showing up correctly in darktable. The pano was done in Lightroom from several Xiaomi jpegs. Lightroom created the DNG. So does this still point to a Xiaomi problem?

The DNG pano shows up fine in RawTherapee, Picasa viewer, Microsoft Photos and any other random image software I have installed, so although darktable might not be doing anything wrong per se, it’s doing something different and undesirable. I’m just wondering if this is a bug or “as designed”.

Thanks everyone for your input. To be clear, I usually shoot raw on a DSLR and use Hugin or ICE to do my panos. I’m not looking for a new workflow on how to create a pano, but just want to ensure all my images are viewable properly in darktable.

#14

I know in the past there were issues with darktable not handling certain DNG compression formats and bit depths, it was my understanding that most of those issues had been fixed, but perhaps some variations on the “Linear DNG” (aka demosaiced DNG) variation of DNG may still be broken. I’m fairly certain some variations of “linear DNG” (such as that saved by Mi Sphere Converter) were supported by dt at least a year ago, but some variations may not be.

I do know that for any unrecognized camera “Color Matrix not found” populs in dt are a nuisance error - it will print this if there is no entry for the camera in its internal database, even if the color matrix data is found AND used within the DNG metadata. (Or at least it did back in August, there could be a regression since then, as I said, I’ve switched tools for reasons unrelated to that particular issue.)

Now if the DNG metadata is just plain broken, nothing can be done about that. Any tool that is generating a DNG from JPEGs is one that’s doing some strange and unusual stuff, and so I don’t trust it. Look at how much of an epic fail “DNG RAW” (which isn’t raw in any way, shape, or form as the input data is heavily processed and frankly severely broken JPEGs) has been for the Xphase Pro. At least Mi Sphere Converter’s linear DNGs were generated by stitching and demosaicing raw sensor data without doing any color management steps.