Issue with desaturated DNGs produced by a phone camera

PS I have also noted this on my Pixel . The native DNG are much darker and more saturated even for a raw file than the Opencamera files. The open camera files are not nearly the same DNG either. They are much larger than the native camera app ones and have no embedded preview. They appear alsmost as if they are a simple compressed linear tiff file or something in a DNG container… likely more complicated than that but the results are clear… I will post a couple later…

1 Like

Thanks for the insights. What’s the best android app to create raw/dng files, if not the Open Camera App, according to you?

I’d be eager to see the examples you mention.

Try renaming it so it matches your current raw file, mine has a “_02” in the filename. I use:
Local contrast
Levels set to auto
Color balance RGB set to global vibrance and global saturation to +30%
Haze removal
Just to offset the orange tint I got and approximate the more reddish flowers I used the color correction module

The sidecar xmp file was already well named, I double checked. Something escapes my understanding here.

Anyway thanks for the details!

I wouldn’t be so sure of that.

I cannot directly access the DNGs from my current location, but I suspect I know what the problem is because it’s a common mistake made by manufacturers who do not bother to actually read the DNG specification - if someone uses exiftool to pull the ColorMatrix tags (and also tell me whether or not there is a HueSatMap table), I can confirm the suspicion.

The issue seems to be in the way the image reads in … at least for me. The raw points go to this for the images below in open camera…

image

If you reset the module same values …but if you double click the sliders…

You get this and the image goes black… these are however the values used by the Google pixel and would appear more normal…

image

The raw image in DT for me from open camera has no color in the flower… and I can’t get it there…

Raw

image

JPEG camera

image

Same pictures taken from Google camera app… not both sets were done with auto exp and spot focus on the central flower…

Google Raw

image

Google JPEG fairly faithful to the scene…green might be just a bit too yellow…

image

Not sure where the color gets lost in the raw…the open camera JPG has the correct color in the flower… You cant set the exposure the same on the two apps very easily…interesting the exposure done basically the same way is not as close as you might think it would be…

These are just screen caps…if anyone want to see the files i can post them…

EDIT…

Just checked out ART…same raw B and W points. Auto tone curve doesn’t work on these DNG’s but tried ART as I could use the Adobe DCP and its elements… That made a pretty big difference but it was the auto WB that I just thought to try… boom color back… So the WB setting at least for me is not good in the raw in OPEN camera even though the jpg seems to be okay…

Open camera Raw now in ART with Adobe DCP and auto WB…


Further turning on the log tone mapping with auto settings leave a pretty nice point from which to edit…

All of those scream to me “bad metadata”. Likely an incorrect ColorMatrix tag (as I said, I can’t access the DNGs from where I am, but if the ColorMatrix is what I think it is, I’ve seen it so many times that if someone pulls it with exiftool I’ll recognize it.)

Based on your comments other problems could be incorrect black/white points in the DNG metadata, I might be able to sanity check those against actual image contents tonight. If you post the DNGs as attachments here instead of a random cloud provider I can do analysis during a compile break earlier.

This week happens to be bad, I’m in the process of packing for vacation so might not be able to do histogram analysis. But as I said, if someone posts the ColorMatrix, I’ve seen one particular matrix that is improperly used as a default value so many times that while I haven’t memorized its values, when I DO see them I recognize them.

2 Likes

I have tried to create a balanced version.

1 Like

Compare this picture with the picture on the phone at 100% magnification.

2022-08-06_en-chemin-vers-JP_0007-2.jpg.out.pp3 (15,4 KB)

You could try using grapheneOS’s camera, as far as I know it will provide better quality than open camera. I’ve compared it on the pixel 6 with google camera and they look almost the same.

There’s no way that is the case, given that Graphene’s app does not support burst stacking, and in fact does not even support raw images. https://github.com/GrapheneOS/Camera/issues/14

1 Like

By burst stacking you mean various shots at different exposures and then put them together? If that’s case it already supports 3 frames at once.

No. HDR+ Pipeline covers the original HDR+ algorithm and a desktop-based implementation of it in detail. Note that Google Camera has gone even further along those lines with Night Sight (and even normal non-NS shots) using a multiframe superresolution approach. [1905.03277] Handheld Multi-Frame Super-Resolution

Also I see no evidence in the GrapheneOS camera source of even bracket merging. Again, it doesn’t even have RAW support, per their own documentation.

2 Likes

Got it, I had it wrong, thanks for the sources, I’m gonna take a look. And yap there’s no raw or Night Sight support in their app. I’ve tested NS in GCam and it’s quite magical how well it works even with slightly moving scenes.

In my opinion for people who don’t need raw or those features, it’s still a decent camera app. I still use GCam but wouldn’t hesitate to use it day to day if I had no sandboxed google play services.

Here’s a quick comparison, shot on Pixel 6, of a daylight scene just for the fun of it(the only one I have from both cameras):

Google Camera

GrapheneOS Camera

Wow so many replies and debates, I’ll spend some time depiling all that, there’s a ton to learn.

Here’s the DNG for the enthusiasts who want to check the metada (the cloud link is okay to me though, I created it myself on a cellar add-on at Clever Cloud):
2022-08-06_en-chemin-vers-JP_0007.dng (23.3 MB)

I’ll come back to the thread when I’m done parsing all your detailed answers :slight_smile:

1 Like

@Entropy512 I extracted the metada with exiftool:

exiftool 2022-08-06_en-chemin-vers-JP_0007.dng                                                             3.65s 21:10
ExifTool Version Number         : 12.42
File Name                       : 2022-08-06_en-chemin-vers-JP_0007.dng
Directory                       : .
File Size                       : 24 MB
File Modification Date/Time     : 2022:08:13 23:56:05+02:00
File Access Date/Time           : 2022:08:13 23:56:05+02:00
File Inode Change Date/Time     : 2022:08:13 23:56:05+02:00
File Permissions                : -rw-r--r--
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Subfile Type                    : Full-resolution image
Image Width                     : 4032
Image Height                    : 3024
Bits Per Sample                 : 16
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Image Description               :
Make                            : Fairphone
Camera Model Name               : FP3
Strip Offsets                   : (Binary data 25848 bytes, use -b option to extract)
Orientation                     : Horizontal (normal)
Samples Per Pixel               : 1
Rows Per Strip                  : 1
Strip Byte Counts               : (Binary data 15119 bytes, use -b option to extract)
X Resolution                    : 72
Y Resolution                    : 72
Planar Configuration            : Chunky
Resolution Unit                 : inches
Software                        : Fairphone/FP3/FP3:10/8901.3.A.0129.20210805/08051356:user/release-keys
Modify Date                     : 2022:08:06 14:46:34
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Copyright                       :
Exposure Time                   : 1/148
F Number                        : 1.8
ISO                             : 100
Date/Time Original              : 2022:08:06 14:46:34
Focal Length                    : 4.4 mm
TIFF-EP Standard ID             : 1 0 0 0
DNG Version                     : 1.4.0.0
DNG Backward Version            : 1.1.0.0
Unique Camera Model             : FP3-Fairphone-Fairphone
CFA Plane Color                 : Red,Green,Blue
CFA Layout                      : Rectangular
Black Level Repeat Dim          : 2 2
Black Level                     : 64 64 64 64
White Level                     : 1023
Default Scale                   : 1 1
Default Crop Origin             : 8 8
Default Crop Size               : 4016 3008
Color Matrix 1                  : 1.3828125 -0.65625 -0.2109375 -0.96875 1.875 0.0390625 0.0390625 -0.1484375 0.78125
Color Matrix 2                  : 2.2421875 -1.546875 -0.390625 -1.0703125 2.1796875 -0.0078125 0.0625 -0.140625 1.2578125
Camera Calibration 1            : 1.03125 0 0 0 1 0 0 0 1.0078125
Camera Calibration 2            : 1.03125 0 0 0 1 0 0 0 1.0078125
As Shot Neutral                 : 0.4609375 1 0.7734375
Baseline Exposure               : 0
Calibration Illuminant 1        : D65
Calibration Illuminant 2        : Standard Light A
Active Area                     : 0 0 3024 4032
Forward Matrix 1                : 0.4375 0.3828125 0.140625 0.21875 0.71875 0.0625 0.015625 0.09375 0.7109375
Forward Matrix 2                : 0.4375 0.3828125 0.140625 0.21875 0.71875 0.0625 0.015625 0.09375 0.7109375
Opcode List 2                   : GainMap, GainMap, GainMap, GainMap
Noise Profile                   : 0.0001669236782103 2.641018082876e-06 0.0001669236782103 2.641018082876e-06 0.0001669236782103 2.641018082876e-06
Aperture                        : 1.8
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 4032x3024
Megapixels                      : 12.2
Shutter Speed                   : 1/148
Focal Length                    : 4.4 mm
Light Value                     : 8.9

Can’t find any “color matrix” in there, though… Oh wait!

Color Matrix 1                  : 1.3828125 -0.65625 -0.2109375 -0.96875 1.875 0.0390625 0.0390625 -0.1484375 0.78125
Color Matrix 2                  : 2.2421875 -1.546875 -0.390625 -1.0703125 2.1796875 -0.0078125 0.0625 -0.140625 1.2578125
Forward Matrix 1                : 0.4375 0.3828125 0.140625 0.21875 0.71875 0.0625 0.015625 0.09375 0.7109375
Forward Matrix 2                : 0.4375 0.3828125 0.140625 0.21875 0.71875 0.0625 0.015625 0.09375 0.7109375

I’m currently in a location where ALL cloud storage providers are blocked (especially the lesser known ones, but even the “big boys” like Google Drive are), but forum attachments usually work.

Interesting, so it’s not the usual problem of them putting an sRGB XYZ->CAM matrix into the CM1 tag. It is a bit strange to see two different ColorMatrix1 values, but only one ForwardMatrix value for both 1 and 2…

White and black points seem correct - the smallest value in the raw data is 64, largest is 1023.

2 Likes

ColorMatrix1 are D65 values, so I extracted them from the metadata and used them dcraw-style. The black level from libraw didn’t seem to be right, so I dialed it up. That, and a rather sporty loggamma-lift/s-curve-shape, oh, and highlight recovery for the blown pavement, got me this from rawproc:

Edit: I’d daresay that poppin’ color in the OOP JPEG is an application of ludicrous color saturation, added that to my processing chain:

Edit #2: @Entropy512 is right, the black point corresponds with the minimum value of all three channels. I just hate milky shadows… :laughing:

1 Like

Apologies that this is specific to Pixels (possibly just the Pixel 6 Pro with the default camera app’s raw). But the same idea applies to every camera, including every phone.

I shot my Color Checker Passport with my Pixel 6 Pro and made a daylight profile that gets auto-applied. (It even helps when it’s not purely daylight, but a specific profile per non-daylight scene would always be better.)

The colors before versus after were… wow. After using the color calibration’s color checker part, the colors are bold and saturated like the JPEGs without manually tweaking things.

It would be better, of course, to get improved colors by default in darktable. Meanwhile, I wonder how applicable my profile would be to other Pixels.

(I still have to calibrate the noise and lenses sometime, somehow.)

Anyway, here’s my makeshift profile for the color calibration module for the built-in camera app’s raw on the Pixel 6 Pro’s main camera. I should make some for the other 3 cameras too, and probably for open camera as well. (I assume it might all be similar… but it might not.)

Firstly: You need to use “modern” color “chromatic adaption defaults” (in darktable’s processing settings) to use this profile.

Pixel 6 Pro - Standard.dtpreset (1.1 KB)

You can import this preset: go to darktable’s settings, select the “presets” page, then click on “import preset…” and choose this file (probably in your Downloads directory — after you download the file, of course).

Then, to apply it, choose a DNG photo from your Pixel and click on the three-line menu icon on the “color calibration” module and select “Pixel 6 Pro - Standard”.

3 Likes

Thank you all so much for such an enthusiast enquiry in the consistence of these DNGs :smiley:
I understand nothing yet about these color matrix. Is it something I should worry/know about, or is it a minor issue in my case?

I mean. Is the Open Camera App doing utter nonsense?