That’s unfortunately a red flag that the manufacturer did not fully test their Camera2 implementation and reports default/filler metadata to Open Camera.
For a while even Pixels had this flaw - if you queried Camera2 for metadata like the color matrix, you got garbage. So at least for a while after release, the Pixel 4 XL had its color data overridden by the camera app!
A DCP from another phone isn’t guaranteed to work - one that shares the same sensor MIGHT work, but not all sensors share a CFA design, and sometimes the lens has significant color effects.
The big question is whether or not you can trust the GainMap - I would take a picture of a very flat evenly lit white surface (or even better, shoot a white wall while holding a translucent object right against the lens like a slice from a plastic milk jug) and verify that the raw, when processed with software that supports GainMap (not all do), it looks evenly lit. If you can, then the remaining task is to shoot a ColorChecker in daylight and tungsten to generate an appropriate profile.
@hannoschwalm ColorMatrix3 is relatively new, part of the DNG 1.6 spec which is almost entirely new tags Adobe co-developed with Apple, I’m not sure if anyone other than Apple have implemented it camera-side yet.
I’m not sure about the Open Camera app being responsible, according to @Entropy512 it’s the phone manufacturer who messed up the phone’s Camera2 API implementation, on which Open Camera connects.
We can ask Open Camera app to take the Fairphone’s lacking Camera2 API in consideration and provide sensible Color Matrix values instead, but I’m not someone to ask that of an open source contributor.
I’d have to look at the source for Open Camera, but I BELIEVE they try to use metadata provided by the camera HAL - which if the phone doesn’t offer raw capture itself may be incorrect. DngCreator | Android Developers
There’s not really any way for a camera app to fix this, except to detect phones with known-bad metadata and replace it. But that’s asking a huge amount of the camera app developer, because correcting the data requires spending the time to characterize the camera, which requires owning that particular phone and a ColorChecker.
Reporting the problem to the hardware manufacturer is unlikely to go anywhere. The Android world is a shitshow in that regard. I’ve seen critical flaws remain unfixed for more than a year after not only was the problem reported to a manufacturer’s developer relations, but a fix was provided as a patch. (For example, the Galaxy S2 kernel had a bug in the power management, where it would enter a state that would keep the phone from entering deep sleep if the fuel gauge was critically low. For one, this seems ridiculous, if you’re almost out of battery why use more? They may have been worried about current consumption during resume… But clearing the condition required the state of charge to be EXACTLY the threshold value. They used == instead of >= to clear the condition if the SoC rose above that threshold when on charger. I submitted a patch to all of my devrel contacts, and more than a year later, people still constantly complained about fuel_alerted wakelocks.)
After all, this is fairphone we’re talking about. A manufacturer who got so caught-up in anti-Qualcomm whining that they didn’t realize that for all of their flaws, Qualcomm was the least of all evils, and went with Mediatek who were blatant GPL violators instead for their first generation or two.
Well I would say open camera generates a problematic or at least unconventional DNG on my Pixel and its pretty mainstream so I just think its the way it takes the data and packages its version of DNG and since this differs likely from what is expected by standard protocol when opened in raw processing software you get a distorted image… I might have to go back and see …there was another app called hedgehog2 or something like that. I think its really a different face on opencamera but maybe it offers a different DNG output… I will dig it up and check…
EDIT…it was called Hedgecam…ha ha … you could give it a whirl…
I’ll have to check …very interesting…I seem to get much better DNG with the Pixel using it…also it exposes quite a few options in bug fixes and other areas…one is to use their DNG or android DNG…also some other settings are there that could alter things… I am experimenting now…one thing I noted is the illuminants are way off comparing native camera app to Open camera…will have to check HGH2… this is Google…
And this is open camera with D55 illuminant vs d65 and they are reversed??
Well for a phone image, in the end I don’t think its too bad. I wasn’t there to see those flowers but if you want a pretty picture you can get one…not sure if its better or worse than your experience but I took your image and did a quick edit…
Funny thing, at least when the Pixel 4 XL shipped, there was evidence that the HAL was reporting bad color metadata, because the issues I had with DNGs on day -1 (my phone arrived one day before the “official” release date) were fixed by a day zero camera app update, not an OS/HAL update.
Interestingly - different ColorMatrix1/2 tags than the Fairphone, but the EXACT same ForwardMatrix tags. That’s deeply concerning.
It’s clear that Google is overriding whatever the HAL provides and embedding a full blown DCP including HueSatMap 2.5D LUTs in every image.
Its interesting when I make a DNG with DNG converter it sets Illum-1 to Std-A and illum-2 to D65. My pixel does the same for the native camera app but using opencamera it sets illum-1 to D55 and Std-a for two. The fairlight phone sets it to D65 for 1 and Std-A for 2. So correctly D65 but reversed and then has the same matrix issue… So I guess it buyer beware on how open camera might interpret and save the image data as a DNG… and how that might be interpreted as a raw file
As long as the CM matrices are correctly calibrated for the given illuminant, the order doesn’t matter as far as DNG spec (and darktable implementation) are concerned. If there is no D65 matrix, dt will look for the closest one and do chromatic adaptation on it. But if the matrices are garbage in, you’ll get garbage out…
@priort I managed to import your sidecar file, thanks!
Now what are those color matrices anyway ? I tried to look it up but there’s a lot of math involved apparently, and I’m just a humble developer. Anyone have a good explanation, or a link to follow?
This is a bit of math at least worth appreciating. Whether in-camera to a JPEG or raw file through a raw processor, the camera data has to be transformed in color down to a gamut of colors the rendition media can handle. There are a number of ways this can be done, but the simplest and most predominant method is a bit of linear algebra using two sets of 9 numbers, the first set describing the input data, and the second set describing the output destination.
The ColorMatrix sets we’ve been discussing here are the first set; they describe the color response of the particular camera. These numbers are sometimes called the “color primaries”, and they consist of three numbers for each of the red, green and blue encoding of color. The destination numbers describe the gamut of a particular display medium; when we export a JPEG, we usually use sRGB primaries because we don’t know what all is out there on which folks might display our images, and sRGB is a sort of least-common-denominator.
