Quad Bayer Demosaic support

Software darktable
,
1

Hello!
Up until now raw image data was typically associated with the bayer color filter array where neighboring image sensor pixels feature a different color filter to sample color information on an otherwise monochrome detecting sensor. This repeating 2by2 CFA pattern is defined in DNG metadata by these tags

CFAPlaneColor: 0 1 2 #defines RGB color channels
CFARepeatPatternDim: 2 2
CFAPattern2: 0 1 1 2 #RGGB bayer phase

However, for many years now image sensors for use in phone camera modules feature a color filter array, where individual color filters are spread over square pixel groups resulting in 4by4 (or even 6by6 / 8by8) CFA pattern commonly referred to as quad bayer CFA.

CFARepeatPatternDim: 4 4
CFAPattern2: 0 0 1 1 1 1 2 2

This allows the image resolution to be reduced by binning pixel values of neighboring pixels of the same color channel to achieve fast sensor readout for video capture in the typical 2by2 bayer CFA. Reducing the resolution on image data featuring a 2by2 bayer CFA is a far more complex problem with many caveats in practice. This describes the main tradeoff for QBCFA as demosaic of a full resolution unbinned readout becomes more complex on the other hand.

QB demosaic is largely overlooked by the industry as it describes a more complex problem where neighboring are no longer always of a different color channel like on bayer CFA. To alleviate this issue, image sensor manufacturers introduced real time on-sensor processing to remosaic quad bayer to bayer CFA with varying success on most QBCFA sensors. Only this preprocessed unbinned output is intended for end user delivery by phone OEMs. This constitutes a lossy conversion and is unacceptable as a long-term solution.

However, while most phones require root or even modified camera drivers to access quad bayer unbinned raw image, Pixel phones expose this for 3rd party camera apps by default. To humor this, an option was introduced to MotionCam Fuse to transfer QBCFA raw image captures into DNG files with a properly defined QBCFA like described above. If the DNGs metadata defines a usual bayer CFA instead, QB image data will be misinterpreted by image processing environments. This is common for example QB DNGs shared so far. As an example two DNGs are provided of the same image data. One with the properly defined QBCFA and the other with the wrong bayer CFA for testing purposes.

So far image processing environments that feature the necessary QB demosaic are rare. Darktable and RawTherapee fail when confronted with QBCFA. Only Lightroom, Photo Editor by dev.macgyver or PhotonCam feature this processing.

https://drive.google.com/drive/folders/1gmwd_2oqmrAgB8GeiHhv7XKZVeBE8XXF?usp=sharing

1 Like
2

It was my understanding that quad-Bayer is mostly used in small sensors to reduce blooming. The idea being that, when pixels get very small, the walls between them become somewhat permeable, and charges leak to neighboring pixels. To reduce this, we split every pixel into four sub-pixels; this has twice as many “walls” between pixels, and thereby reduces leakage.

But there’s still enough leakage that there’s not much to be gained from reading each subpixel. Thus the quad-Bayer subpixels are meant to be binned into normal-Bayer macro-pixels.

1 Like
3

certainly esoteric enough to be interesting :slight_smile: do you have any visuals? proof that demosaicing quad-bayer actually yields tangible improvement? i haven’t looked into it much, the papers i read were more like negative results on why the frequency content in quad-bayer is so much worse than bayer that demosaicing is really bad guesswork.

4

Not blooming. It is mostly used in analog binning mode so the charge from a cluster of 4 pixels is converted as a single larger pixel by the ADC, hence the e.g. 48MP to 12MP resolution switch (and the result is regular Bayer). This analog binning provides better SNR and about half a stop of additional dynamic range when small sensors struggle in lower light conditions. If there’s plenty of light it switches to quad 48MP mode.

2 Likes
5

Yep, most likely ML is used to improve on the initial subpar result.

6

Hey, Thanks for chiming in!
Since we’re speaking of unstacked frames (it’s doubtful that a stacked Quad Bayer raw would be a thing, since a good reason to stack it would be to get RGB data rather than stay with the problem of QB), yes, you can achieve an improvement in level of details over the normal binned mode, as long as you ETTR enough not to suffer by noise. This can be shown by sensor-remosaiced raws even on sensors as dated as IMX586. Since sensor can’t really invent details out of nowhere, it’s fair to conclude that demosaicing instead of remosaicing plus demosaicing would serve as a further improvement.
We absolutely know that unstacked Quad Bayer sensor can’t come close to actual Bayer of the same resolution (you can see that Nokia 808/1020 keep winning with high res modes of most current phones to this day), but at the same time, it’d be very cool to squeeze what we can from what we have on the market.

7

The samples I shared were directly obtained in real time through Camera2 API, so I can vouch for absence of ML in them. One of the files has incorrect CFA metadata to read it more easily in existing tools, another has correct metadata.