Same phone, different raw... rgbw and rgb versions?

[ggc]

In the first link, i read that the image sensor is a sony-imx278 .

I don’t understand if imx278 is a rgbw or an rgb, and doesn’t matter that much anyway, what would be cool is to be able to develop the files also from that version of the phone.

And anyway, i think there has been a “first release” of the phone with a kind of sensor, than mine with a normal sensor.

[snibgo (Alan Gibson)]

@snibgo

Using OtherImages.zip, messing around with the channels, they seem to be [green blue][red green]. This is neither the common [r g][g b] pattern nor the pattern claimed by exiftool: [b g][g r].

Aarrhhh, sorry, I’m so stupid. I was looking at looking at the image of the green tree and blue sky. But this is potrait format, so the camera was rotated, so the top-left pixel of the final image was the bottom-left pixel as recorded by the camera.

So the bottom-left 4 sensels as captured by the camera were:

b g
g r

After rotation, these become:

g b
r g

So the exif metadata is correct. The CFA really is [b g][g r]. That solves that problem.

@ggc

I don’t understand if imx278 is a rgbw or an rgb, and doesn’t matter that much anyway, what would be cool is to be able to develop the files also from that version of the phone.

The only problem is those four weird sensels in each 16x16 square. Ideally a raw processor would know what to do with them. If they are “white”, it could incorporate them into the demosaicing algorithm (but I don’t know how).

Next best: a raw processor would ignore the values in those sensels, treating them as having no R, G or B value, so these sensels need interpolated values for all three channels instead of just two. Again, I don’t know how.

Worst option: ignore the values in those sensels, but assume they should be Blue values, and make them the simple average of the four nearest Blue sensels. Then the image can be demosaiced in the usual way. And I can do that. The commands are for Windows BAT scripts.

Get the raw sensel values. However, dcraw re-orients to make the image “portrait”, even with the “-j” option:

set CAMERA_SRC=IMG_20191105_134321.dng

%DCRAW% -v -W -o 0 -6 -r 1 1 1 1 -g 1 0 -D -d -T -O cam_gray.tiff %CAMERA_SRC%

Make a list of options for exiftool:

(
  exiftool -args -make -model %CAMERA_SRC% 

  echo -DNGVersion=1.4.0.0
  echo -DNGBackwardVersion=1.3.0.0
  echo -EXIF:SubfileType=Full-resolution Image
  echo -PhotometricInterpretation=Color Filter Array
  echo -IFD0:CFARepeatPatternDim=2 2
  echo -IFD0:CFAPattern2=1 2 0 1
  echo -Orientation=Horizontal
  echo -BitsPerSample=16
  echo -SamplesPerPixel=1
)>exifargs.txt

Note the CFA pattern is correct for this “portrait” image.

Make a 16x16 mask that is white where the sensels are weird, otherwise black:

magick ^
  -size 16x16 xc:Black ^
  -fill White ^
  -draw "point 1,6 point 5,6 point 9,14 point 13,14" ^
  m.png

This is correct for this “portrait” image. For other orientations, it should be rotated.

Do some magick on the image: transform from the range 0-1023 to 0-QuantumRange (eg 0-65535); make every pixel the mean of the four nearest pixels of the same channel; composite that over the original, with a tiled mask, so only the weird pixels are changed.

magick ^
  cam_gray.tiff ^
  -evaluate Multiply %%[fx:QuantumRange/1023] ^
  ( +clone ^
    -define "convolve:scale=^!" ^
    -morphology Convolve 5x5:%KNL% ^
  ) ^
  -size %%[fx:w]x%%[fx:h] tile:m.png ^
  -compose Over -composite ^
  cam_gray2.tiff

Make this TIFF into a DNG:

del cam_gray.dng
exiftool -@ exifargs.txt -o cam_gray.dng cam_gray2.tiff

Finally, test the DNG by using dcraw to make a sRGB TIFF:

%DCRAW% -v -6 -T -O x.tiff cam_gray.dng

The result, x.tiff, is good, There is no visible pattern of repeating noise.

This is not a great solution. The code is klunky, and would be worse if we had to properly process camera rotations. It ignores the data that is in those 4 every 256 sensels, although this might be useful “white” data.

But it works.

[ggc]

Great work!

But i am on linux and i don’t know what is magick (i have imagemagick installed, but i can’t call it with magick).

Thank you!

[Claes]

Hi @ggc,

Presumably your imagemagick is too old? Check your version.
Mine is ImageMagick 7.0.9-2 Q16 x86_64 2019-11-06.

Have fun!
Claes in Lund, Sweden

[ggc]

I have this one, from ubuntu 18.04 packages…

imagemagick 8:6.9.7.4+dfsg-16ubuntu6.8

[snibgo (Alan Gibson)]

Yes, my commands use ImageMagick. I’m using version 7. For version 6, use “convert” instead of “magick”.

In v6, “-evaluate Multiply %%[fx:QuantumRange/1023]” won’t work. Assuming your ImageMagick is Q16 (use “convert -version” to verify), you can substitute “-evaluate Multiply 64.06158357771261”.

In v6, “-size %%[fx:w]x%%[fx:h]” won’t work. Substitute the size of you image, eg “-size 3136x4224”.

I expect this work could be done with G’MIC, but I don’t know how.

An alternative might be to declare the weird pixels to be “dead”, and use dcraw’s “-P” option. I’ve never tried this.

[Entropy512 (Andy Dodd)]

@Claes

Yes, it does, indirectly.
The front lens is handled by an IMX278.

Have fun!
Claes in Lund, Sweden

Front? Could the inconsistency the OP is dealing with be a front/back lens thing?

Or was one of the units purchased in a different country? Sometimes phones that carry the same “basic” designator (such as Galaxy S3) are VASTLY different internally (and do have a different detailed model number.)

For example, from long ago (but a particularly notable example):
GT-I9300 - International Galaxy S3 (Exynos CPU)
GT-I9305 - International Galaxy S3 (Exynos CPU with ??? LTE modem)
SGH-I747 - AT&T GS3 (Qualcomm Snapdragon SoC with Qualcomm LTE)

I believe some of them had different camera sensors too…

[Claes]

@Entropy512 Ho-hum… My Italian was a bit too fast! Here is a more truthful translation:

• a rear 13MP image sensor signed Sony (model IMX278), and
• a front 5MP one

Scusi, Dottore!

Claes in Lund, Sweden

[afre (Alan)]

There is another possibility: the part has the tech but didn’t pass QA and therefore is processed as if those pixels were dead.

[snibgo (Alan Gibson)]

The Sony “Product Brief” for IMX278 (http://cdn.specpick.com/images/photonics/products/IMX278.pdf ) says:

RGBW coding (R,G,B, and WHITE*1) color filter and RGB primary color mosaic reproduction on chip

Note) 1. ”WHITE” means the pixels are using white color filter.

We can pull out all the x0 pixels and make an image. Then the same for x1, x2 and x3.

These are, essentially, lightness. But x0 and x3 are lighter than x1 and x2, by about 10% overall. So these sensels can’t be used in a simple manner for lightness. Further work might establish if the difference is deliberate manufacturing choice or manfactured bugs or whatever.

Demosaicking algorithms for RGBW color filter arrays looks interesting, but is for RGBW CFAs that have a greater proportion of white sensels than the IMX278.

[ggc]

Thank you i’ll try, yes my imagemagick is Q16.

Both phones are been bought in Italy, my friend’s one from Honor store, mine from Amazon.it . Maybe the one from honor store was an international version, i don’t know. Look, software and so on is really the same.

Unfortunatly i have posted a list of phones that should use always this sensor, but on raw.pixls.us there is no other sample from another phone. Iiirc i remember the huawei p8 doens’t create raw files, only jpeg.

Also mine (that demosaics more ok thank this one) has a lot of white dots (see some previous post), that should be used for autofocus, but maybe they also contain some image info.

Maybe ignoring them as a first strategy isn’t a bad idea.

[afre (Alan)]

@ggc

Both phones are been bought in Italy, my friend’s one from Honor store, mine from Amazon.it . Maybe the one from honor store was an international version, i don’t know. Look, software and so on is really the same.

Unfortunatly i have posted a list of phones that should use always this sensor, but on raw.pixls.us there is no other sample from another phone. Iiirc i remember the huawei p8 doens’t create raw files, only jpeg.

You might be able to find out in the info pane. Often this is hidden but you could access it by a long press or something to that effect. It depends on the software. Or you could crack open the phone and examine its innards but you would risk damaging it and voiding the warranty.

@snibgo

Demosaicking algorithms for RGBW color filter arrays looks interesting, but is for RGBW CFAs that have a greater proportion of white sensels than the IMX278.

I briefly examined this and others (e.g., ones with yellow pixels) but they are way beyond my ability to implement.

Since this phone’s is sparse and different (x0 and x3 vs x1 and x2), I still think they might be used for something other than straight up RGBW interpolation. Besides focusing, it could be used for depth, transmission or salience mapping. Everything I say is speculation: it is fun to guess what it is.

[snibgo (Alan Gibson)]

@afre

… but they are way beyond my ability to implement.

And beyond mine, unless I spent loads of time on it, which I won’t as I don’t have a RGBW camera.

I haven’t found any online information about how to interpret these weird sensels. A more enthusiastic person might contact Sony. But Sony might mutter about proprietary information.

[ggc]

the info panel reports the same things unfortunatly…

[ggc]

thank you anyway for your great study.

I need to find out how to translate your “way” to make dng usable to linux, and it will be a great thing!

For the dcraw part, should be ok. I don’t have a -O option, but i think it won’t matter that much.

But i don’t understand how to translate the exiftool part so i am unable to proceed.

Thank you again.

[afre (Alan)]

Use -c to output to stdout, then redirect to a file. In CMD, I could do

dcraw -T -c _DSC0488.NEF > 488.tif

-T Output to *.tiff.
-c Write decoded images or thumbnails to standard output.
command > filename Redirect command output to a file.

With redirecting, notice I could name it any way I want. I like to truncate it to *.tif.

[snibgo (Alan Gibson)]

Here is a bash script version of my above commands. Tested with Cygwin bash.

CAMERA_SRC=phone/IMG_20191105_134321.dng

dcraw -v -o 0 -6 -r 1 1 1 1 -g 1 0 -D -d -c -T $CAMERA_SRC >cam_gray.tiff

(
  exiftool -args -make -model $CAMERA_SRC

  echo -DNGVersion=1.4.0.0
  echo -DNGBackwardVersion=1.3.0.0
  echo -EXIF:SubfileType=Full-resolution Image
  echo -PhotometricInterpretation=Color Filter Array
  echo -IFD0:CFARepeatPatternDim=2 2
  echo -IFD0:CFAPattern2=1 2 0 1
  echo -Orientation=Horizontal
  echo -BitsPerSample=16
  echo -SamplesPerPixel=1
)>exifargs.txt

magick \
  -size 16x16 xc:Black \
  -fill White \
  -draw "point 1,6 point 5,6 point 9,14 point 13,14" \
  m.png

KNL=\
0,0,1,0,0,\
0,0,0,0,0,\
1,0,0,0,1,\
0,0,0,0,0,\
0,0,1,0,0

magick \
  cam_gray.tiff \
  \( +clone \
     -define "convolve:scale=!" \
     -morphology Convolve "5x5:$KNL" \
  \) \
  -size %[fx:w]x%[fx:h] tile:m.png \
  -compose Over -composite \
  cam_gray2.tiff

rm cam_gray.dng
exiftool -@ exifargs.txt -o cam_gray.dng cam_gray2.tiff

dcraw -v -6 -c -T cam_gray.dng >x.tiff

This assumes ImageMagick is version 7. For v6, see my comments above. V7 is better than v6, and I recommend upgrading, unless there are particular reasons for staying with v6.

[ggc]

Great thank you!

I have been able to run the script and to get the picture!

[snibgo (Alan Gibson)]

Excellent!

[kmilos]

Here’s an idea, if you feel like experimenting further just for kicks:

Open all x1/x2/x3/x4 channels in overlaying windows/layers and try flipping between those quickly. If you see the in focus objects not changing but out of focus ones moving around or changing intensity slightly, then those are indeed phase detect pixels.

Most probably the case of pre-production camera firmware not interpolating those pixels out (some method similar to above) before saving to DNG.