Compressed RAF: help needed.

raf
libraw
fujifilm
rawspeed
rawtherapee

(Roman Lebedev) #1

Hi everyone.

Uwe Müssel has finally PR’d his work on porting libraw’s compressed RAF support into RawSpeed, and it is now avaliable in a raf-compressed branch.

To proceed with integration into the develop branch, among the other things, i need to have a full sample coverage of that code. Thankfully, RPU exists, and it contains all 5 out of 5 compressed raf samples (FIXME: or are there more cameras with that feature?).

However, as the code suggests, such compressed raw file may be 14-bit, and 12-bit.
All these RPU samples are 14-bit. I need one such raw, which is 12-bit.
Specifically, i need this branch to be triggered: https://github.com/Beep6581/RawTherapee/blob/6e55f6bab5ac9289c4bacf2e6cded9b92c3a3d24/rtengine/fujicompressed.cc#L76-L80

I don’t have a complete guide on how to acquire such a sample, i do not know which camera with which settings produces it. I can guess that you can test whether some raw file is the sample i need by adding either derror(); or merror (0, "that is the sample"); into that branch, building RT, looking through the raws, and seeing if some RAF no longer loads.

Perhaps someone can help.

Roman.


X-trans compressed -- status?
(Ingo Weyrich) #2

Maybe @sguyader can help…


(Sebastien Guyader) #3

I don’t know if I can help. To my knowledge, there are currenty only 5 Fujifilm cameras with lossless raw compression:

  • 4 cameras with X-Trans CMOS III sensor:
    X-100F, X-T20, X-T2 and X-Pro2
  • 1 camera with a regular Bayer CFA:
    GFX 50S

I looked at the official specifications of all these cameras, and on the Fujifilm website it always reports 14-bit raw. I saw no mention of 12-bit raw whatsoever.

There’s one blogger (https://alikgriffin.com/uncompressed-vs-lossless-fujifilm-x-t2/) who said compressed raws are 12-bit, but he failed to give any reliable source about this. He even “saw” differences in image quality between compressed vs. uncompressed… He’s certainly wrong.


(Andrew Cox) #4

This thread mentions 12-bit raw files for the older X-E1 and X-Pro1 cameras, but I can’t find any official Fuji documentation to back that up. And if those raw files are indeed 12-bit, I don’t think they’re a compressed version; they’re just the uncompressed raw files.


#5

X-E1 and X-Pro1 for sure have no compressed RAW format. Only the newest generation of cameras with the new 24 MP X-Trans sensor (the four cameras @sguyader stated) plus the medium format GFX 50s support compressed RAW format. I also never heared about 12 bit compressed RAW, wouldn’t be lossless anyway, so I think the blogger alikgriffin.com is just wrong since there is clearly now difference in image quality between compressed and uncompressed Fujifilm RAWs.


(Roman Lebedev) #6

I agree, but the code suggests that 12-bit compressed RAF’s exist…
I would guess it is created by some of the older cameras in high-speed shooting mode (14/12/etc fps)
I did mail the libraw with this question, hopefully they may know whether it’s a dead code not.


(Sebastien Guyader) #7

I just did some empirical tests with my X-T2: I shot compressed raws, in S (single frame), CL (low-speed burst), and CH (high-speed burst). Bursts are around 8 fps without the battery grip and with mechanical shutter.

Then I compared the file sizes, as I suppose 12-bit vs 14-bit would make the files differ in size:

  • S : 1 file of 23.3 MB
  • CL : 4 files of 23.6, 23.7, 23.4 and 23.7 MB
  • CH : 6 files of 23.7, 23.6, 23.7, 23.8, 22.5 and 22.9 MB

Clearly, there’s no significant difference in file size, so I guess they are all 14-bit.

[Edit] Now a test with battery grip and electronic shutter (burst speed is 11-14 fps):

  • S : 1 file of 23.1 MB
  • CL : 4 files of 23.7 MB
  • CH : 7 files of 23.7, and 1 of 23.6 MB

Same tests, but without lossless compression: all files between 50.5 and 50.6 MB.


(Roman Lebedev) #8

JFYI:

$ exiftool -bitspersample compressed-raf/*.RAF 
======== compressed-raf/20170114_174341_TFW04727.RAF
Bits Per Sample                 : 14
======== compressed-raf/20170525_0037TEST.RAF
Bits Per Sample                 : 14
======== compressed-raf/DSCF0151.RAF
Bits Per Sample                 : 14
======== compressed-raf/DSCF0527.RAF
Bits Per Sample                 : 14
======== compressed-raf/_DSF5844.RAF
Bits Per Sample                 : 14
    5 image files read

I DO NOT know if that is the field i’m looking for, but if it prints 12, it may be interesting nonetheless.


(Sebastien Guyader) #9

I get the same result on all my RAF files using this exiftool command.


(Matthew Fells) #10

I’ve got a RAF file from an X10. According to EXIF data, it’s 12 bits per sample. Is this what you need? DSCF1024.RAF (18.9 MB)


(Sebastien Guyader) #11

The X10 doesn’t compress its raw files. If I’m not mistaken, here we’re trying to deal with compressed raw files support.


(Matthew Fells) #12

Ah, I see. In the exif, it says JPEG under compression, which made me wonder.

Good luck with your search.


(Roman Lebedev) #13

After a brief mail exchange with the original author[s] of the code, i’m now under the impression that such sample may simply not exist in the wild yet, it is possible that no current camera produces it.


(Roman Lebedev) #14

(Rhonald John Rose) #15

I own Fujifilm X-T3 with bunch of lenses, what can I do? white balance information of X-T3 RAW is not recognized by Darktable at this moment and so I am relying on JPG.


(Mica) #16

the 2.6rc0 should be of use: darktable 2.6.0rc0 released


(Sebastien Guyader) #17

@rhonaldjr I have a simple solution: trade your X-T3 for my X-T2, the latter is perfectlly recognized :laughing:


(darix) #18

My Darktable handles the X-T3 just fine :wink:

Even with profiled denoise and all!


(Rhonald John Rose) #19

@paperdigits: I am glad, will give a try

@sguyader: nice try, will stick to X-T3 + JPG till the support is released.


(Rhonald John Rose) #20

@darix: really!!? mine keep throwing “failed to read camera white balance information from…”. I use uncompressed RAW and that could be the reason. Compressed RAW is recognized by DT I believe.