Issue with desaturated DNGs produced by a phone camera

Hello everyone,

I’ve gone on a quest to get the best possible pictures out of my Fairphone 3 before thinking of moving on to a proper camera.

What I do for now:

  1. Use the Open Camera App to shoot raw DNGs
  2. Import the DNGs on darktable
  3. Learn all I can about white balance, color grading and so on, play around
  4. Print that stuff and give it to my friends.

Where did the colors go, though?

Here is an DNG, as produced by the phone and freshly imported in darktable, with the usual set of automatic settings (white balance, filmic rgb, exposure…):

400x300

Pretty flat, huh? It pales in comparison to the vivid colors of the JPEGs produced by the Open Camera App (or by the standard android app):

400x300

(but there’s waaaaaayyy too much sharpness)

Should I push the saturate button?

What I like with Darktable and all those rants by Aurélien Pierre, is the scene-referred workflow, where we avoid as much as possible to lose data.

I tried playing around with the color balance rgb module, to increase the chroma and avoid clipping, but the result isn’t so convincing to me.

Or I can be a complete noob, crank to the max the saturation button of the contrast brightness saturation module, and it does yield something comparable to the phone app:

400x300

But the clipping indicator isn’t quite happy with this, I’ve got a red alert all over those flowers.

My questions

  • What am I missing here?
  • Should I reasonnably expect such vivid colors with such a workflow (phone-dng-darktable), without losing nuance?

Feel free to check out the photos by downloading the DNGs, JPEG and darktable XMP files

These photos are not worth much, but by all means, use them, have them, along with Creative Commons, By-Attribution, Share-Alike.


2022-08-06_en-chemin-vers-JP_0007_02.dng.xmp (14.8 KB)

2 Likes

(darktable 4.0 user manual - process)

This is normal…

You can change the degree to which you drive them out of gamut …tough sometimes with flowers… the gamut slider in CC module does a nice job at attenuating the issue…you can rightclick and add a larger number if needed than the initial range of the slider…

@Mike_Bing Thanks for prooving it is possible.
For now I have trouble importing the sidecar xmp file into darktable to view what you have done (how do I do this best? if I just paste it next to the othe sidecar file, darktable doesn’t seem to recognize it).

Would you mind explaining how your workflow works?

PS I have also noted this on my Pixel . The native DNG are much darker and more saturated even for a raw file than the Opencamera files. The open camera files are not nearly the same DNG either. They are much larger than the native camera app ones and have no embedded preview. They appear alsmost as if they are a simple compressed linear tiff file or something in a DNG container… likely more complicated than that but the results are clear… I will post a couple later…

1 Like

Thanks for the insights. What’s the best android app to create raw/dng files, if not the Open Camera App, according to you?

I’d be eager to see the examples you mention.

Try renaming it so it matches your current raw file, mine has a “_02” in the filename. I use:
Local contrast
Levels set to auto
Color balance RGB set to global vibrance and global saturation to +30%
Haze removal
Just to offset the orange tint I got and approximate the more reddish flowers I used the color correction module

The sidecar xmp file was already well named, I double checked. Something escapes my understanding here.

Anyway thanks for the details!

I wouldn’t be so sure of that.

I cannot directly access the DNGs from my current location, but I suspect I know what the problem is because it’s a common mistake made by manufacturers who do not bother to actually read the DNG specification - if someone uses exiftool to pull the ColorMatrix tags (and also tell me whether or not there is a HueSatMap table), I can confirm the suspicion.

The issue seems to be in the way the image reads in … at least for me. The raw points go to this for the images below in open camera…

image

If you reset the module same values …but if you double click the sliders…

You get this and the image goes black… these are however the values used by the Google pixel and would appear more normal…

image

The raw image in DT for me from open camera has no color in the flower… and I can’t get it there…

Raw

image

JPEG camera

image

Same pictures taken from Google camera app… not both sets were done with auto exp and spot focus on the central flower…

Google Raw

image

Google JPEG fairly faithful to the scene…green might be just a bit too yellow…

image

Not sure where the color gets lost in the raw…the open camera JPG has the correct color in the flower… You cant set the exposure the same on the two apps very easily…interesting the exposure done basically the same way is not as close as you might think it would be…

These are just screen caps…if anyone want to see the files i can post them…

EDIT…

Just checked out ART…same raw B and W points. Auto tone curve doesn’t work on these DNG’s but tried ART as I could use the Adobe DCP and its elements… That made a pretty big difference but it was the auto WB that I just thought to try… boom color back… So the WB setting at least for me is not good in the raw in OPEN camera even though the jpg seems to be okay…

Open camera Raw now in ART with Adobe DCP and auto WB…


Further turning on the log tone mapping with auto settings leave a pretty nice point from which to edit…

All of those scream to me “bad metadata”. Likely an incorrect ColorMatrix tag (as I said, I can’t access the DNGs from where I am, but if the ColorMatrix is what I think it is, I’ve seen it so many times that if someone pulls it with exiftool I’ll recognize it.)

Based on your comments other problems could be incorrect black/white points in the DNG metadata, I might be able to sanity check those against actual image contents tonight. If you post the DNGs as attachments here instead of a random cloud provider I can do analysis during a compile break earlier.

This week happens to be bad, I’m in the process of packing for vacation so might not be able to do histogram analysis. But as I said, if someone posts the ColorMatrix, I’ve seen one particular matrix that is improperly used as a default value so many times that while I haven’t memorized its values, when I DO see them I recognize them.

2 Likes

I have tried to create a balanced version.

1 Like

Compare this picture with the picture on the phone at 100% magnification.

2022-08-06_en-chemin-vers-JP_0007-2.jpg.out.pp3 (15,4 KB)

You could try using grapheneOS’s camera, as far as I know it will provide better quality than open camera. I’ve compared it on the pixel 6 with google camera and they look almost the same.

There’s no way that is the case, given that Graphene’s app does not support burst stacking, and in fact does not even support raw images. https://github.com/GrapheneOS/Camera/issues/14

1 Like

By burst stacking you mean various shots at different exposures and then put them together? If that’s case it already supports 3 frames at once.

No. HDR+ Pipeline covers the original HDR+ algorithm and a desktop-based implementation of it in detail. Note that Google Camera has gone even further along those lines with Night Sight (and even normal non-NS shots) using a multiframe superresolution approach. [1905.03277] Handheld Multi-Frame Super-Resolution

Also I see no evidence in the GrapheneOS camera source of even bracket merging. Again, it doesn’t even have RAW support, per their own documentation.

2 Likes

Got it, I had it wrong, thanks for the sources, I’m gonna take a look. And yap there’s no raw or Night Sight support in their app. I’ve tested NS in GCam and it’s quite magical how well it works even with slightly moving scenes.

In my opinion for people who don’t need raw or those features, it’s still a decent camera app. I still use GCam but wouldn’t hesitate to use it day to day if I had no sandboxed google play services.

Here’s a quick comparison, shot on Pixel 6, of a daylight scene just for the fun of it(the only one I have from both cameras):

Google Camera

GrapheneOS Camera

Wow so many replies and debates, I’ll spend some time depiling all that, there’s a ton to learn.

Here’s the DNG for the enthusiasts who want to check the metada (the cloud link is okay to me though, I created it myself on a cellar add-on at Clever Cloud):
2022-08-06_en-chemin-vers-JP_0007.dng (23.3 MB)

I’ll come back to the thread when I’m done parsing all your detailed answers :slight_smile:

1 Like

@Entropy512 I extracted the metada with exiftool:

exiftool 2022-08-06_en-chemin-vers-JP_0007.dng                                                             3.65s 21:10
ExifTool Version Number         : 12.42
File Name                       : 2022-08-06_en-chemin-vers-JP_0007.dng
Directory                       : .
File Size                       : 24 MB
File Modification Date/Time     : 2022:08:13 23:56:05+02:00
File Access Date/Time           : 2022:08:13 23:56:05+02:00
File Inode Change Date/Time     : 2022:08:13 23:56:05+02:00
File Permissions                : -rw-r--r--
File Type                       : DNG
File Type Extension             : dng
MIME Type                       : image/x-adobe-dng
Exif Byte Order                 : Little-endian (Intel, II)
Subfile Type                    : Full-resolution image
Image Width                     : 4032
Image Height                    : 3024
Bits Per Sample                 : 16
Compression                     : Uncompressed
Photometric Interpretation      : Color Filter Array
Image Description               :
Make                            : Fairphone
Camera Model Name               : FP3
Strip Offsets                   : (Binary data 25848 bytes, use -b option to extract)
Orientation                     : Horizontal (normal)
Samples Per Pixel               : 1
Rows Per Strip                  : 1
Strip Byte Counts               : (Binary data 15119 bytes, use -b option to extract)
X Resolution                    : 72
Y Resolution                    : 72
Planar Configuration            : Chunky
Resolution Unit                 : inches
Software                        : Fairphone/FP3/FP3:10/8901.3.A.0129.20210805/08051356:user/release-keys
Modify Date                     : 2022:08:06 14:46:34
CFA Repeat Pattern Dim          : 2 2
CFA Pattern 2                   : 0 1 1 2
Copyright                       :
Exposure Time                   : 1/148
F Number                        : 1.8
ISO                             : 100
Date/Time Original              : 2022:08:06 14:46:34
Focal Length                    : 4.4 mm
TIFF-EP Standard ID             : 1 0 0 0
DNG Version                     : 1.4.0.0
DNG Backward Version            : 1.1.0.0
Unique Camera Model             : FP3-Fairphone-Fairphone
CFA Plane Color                 : Red,Green,Blue
CFA Layout                      : Rectangular
Black Level Repeat Dim          : 2 2
Black Level                     : 64 64 64 64
White Level                     : 1023
Default Scale                   : 1 1
Default Crop Origin             : 8 8
Default Crop Size               : 4016 3008
Color Matrix 1                  : 1.3828125 -0.65625 -0.2109375 -0.96875 1.875 0.0390625 0.0390625 -0.1484375 0.78125
Color Matrix 2                  : 2.2421875 -1.546875 -0.390625 -1.0703125 2.1796875 -0.0078125 0.0625 -0.140625 1.2578125
Camera Calibration 1            : 1.03125 0 0 0 1 0 0 0 1.0078125
Camera Calibration 2            : 1.03125 0 0 0 1 0 0 0 1.0078125
As Shot Neutral                 : 0.4609375 1 0.7734375
Baseline Exposure               : 0
Calibration Illuminant 1        : D65
Calibration Illuminant 2        : Standard Light A
Active Area                     : 0 0 3024 4032
Forward Matrix 1                : 0.4375 0.3828125 0.140625 0.21875 0.71875 0.0625 0.015625 0.09375 0.7109375
Forward Matrix 2                : 0.4375 0.3828125 0.140625 0.21875 0.71875 0.0625 0.015625 0.09375 0.7109375
Opcode List 2                   : GainMap, GainMap, GainMap, GainMap
Noise Profile                   : 0.0001669236782103 2.641018082876e-06 0.0001669236782103 2.641018082876e-06 0.0001669236782103 2.641018082876e-06
Aperture                        : 1.8
CFA Pattern                     : [Red,Green][Green,Blue]
Image Size                      : 4032x3024
Megapixels                      : 12.2
Shutter Speed                   : 1/148
Focal Length                    : 4.4 mm
Light Value                     : 8.9

Can’t find any “color matrix” in there, though… Oh wait!

Color Matrix 1                  : 1.3828125 -0.65625 -0.2109375 -0.96875 1.875 0.0390625 0.0390625 -0.1484375 0.78125
Color Matrix 2                  : 2.2421875 -1.546875 -0.390625 -1.0703125 2.1796875 -0.0078125 0.0625 -0.140625 1.2578125
Forward Matrix 1                : 0.4375 0.3828125 0.140625 0.21875 0.71875 0.0625 0.015625 0.09375 0.7109375
Forward Matrix 2                : 0.4375 0.3828125 0.140625 0.21875 0.71875 0.0625 0.015625 0.09375 0.7109375