Sony ARW: 12-bit/14-bit sensor vs 16-bit file format waste of space

Processing
1

According to Sony
ILCE-7 / ILCE-7R / DSC-RX1 / DSC-RX1R
ILCE-7M2 / ILCA-99M2 / DSC-RX1RM2
ILCE-7S / ILCE-6100 / ILCE-6300 / ILCE-6400 / ILCE-6500 / ILCE-6600
ILCE-7RM2 / ILCE-7SM2
ILCE-7RM3 / ILCE-7RM3A / ILCE-7M3 / ILCE-7RM4 / ILCE-7RM4A / ILCE-7C
ILCE-9 / ILCE-9M2
can record 12-bit / 14-bit per pixel color but write 16-bit values into ARW file.

This is ridiculous beside missing lossless compression.

With 61Mpixel@14-bit we waste ~15MiB or ~14% space for nothing:
9568x6376 x 16-bit = 116MiB (real raw file size [meta information not counted])
9568x6376 x 14-bit = 102MiB (expected raw file size [meta information not counted])

With 61Mpixel@12-bit we waste ~29MiB or ~33% space for nothing:
9568x6376 x 16-bit = 116MiB (real raw file size [meta information not counted])
9568x6376 x 12-bit = 87MiB (expected raw file size [meta information not counted])

Question:
Is there any image format which can be used afterwards to save space but not losing any information?
Opensource image format preferred. Looks like PNG only support 8/16/32-bit steps?
Hacking a custom ARW12/ARW14 format needed?

tags: 12bit 14bit uncompressed raw arw sony alpha lossless

2

I don’t know if you lose any MakerNotes or other metadata these days. If you don’t care about that, DNG is the apparent choice.

That’s a very bad option as your raw files would have to go through demosaicing and you will no longer have raw files with all the benefits that this comes with.

The fact that they listened to their (uninformed) customers and went for uncompressed raw rather than lossless is insane. Can you imagine all the wasted storage space, globally? Petabytes after petabytes…

For my a7 III, DNG with lossless compression cuts the file size in half. 50 MB ARW vs average ~25 MB DNG. I’m very tempted to go this route and throw away the ARWs.

3

If the format is based on Tiff-6, it should be able to write out a non-8-bit integer in baseline non-compressed. I suppose there are speed disadvantages to not using 16-bit words…

4

ARW is also nothing but a TIFF, just like DNG. And just like DNG, it now supports lossless (JPEG) compression since ILCE-1 (late to the party, but kudos to Sony for not going for something proprietary here and reinventing the wheel). So no “new” custom format is needed, but a tool to convert uncompressed to lossless compressed ARW if you don’t want to go the DNG route. Would actually be cool to see if those would still be compatible w/ Sony and other software even if the model isn’t ILCE-1. Mind you, this ARW mode is not yet supported by dt/rawspeed.

5

Uncompressed lossless was a stopgap compromise because their informed customers were legitimately unhappy with their lossy raw compression algorithm.

Lossless compression was not possible with acceptable performance without an updated BIONZ, and it took them FOREVER to do that. (A number of major new features are tied to the BIONZ XR - lossless raw compression is one, 10-bit video is another)

@kmilos - is there actual evidence the A1 uses LJ92 as opposed to another algorithm? Your comments in the link hinting at 512x512 tiles kind of hint that a different algorithm may be in play.

6

I have a a7rii and look at the images, and some look good. And if not then I blame it on me, and not the camera. I really don’t care about the datasheet, that for people with a GAZ addiction.

7

Camera design is a trade-off between time and power requirements. Reviewers often comment on battery life, frames per second, or how long it takes to record one image, but almost never on the images per gigabyte. So guess what camera manufacturers concentrate on?

TIFF can store 12 or 14 bits/channel/pixel, uncompressed or with Zip or LZW compression. We can do the work with ImageMagick, and verify with exiftool:

f:\web\im>%IMG7%magick toes.png -depth 14 -compress LZW t.tiff

f:\web\im>exiftool t.tiff
ExifTool Version Number         : 12.13
File Name                       : t.tiff
:
:
Bits Per Sample                 : 14 14 14
Compression                     : LZW

Like any format, grayscale PNG can be used for mosaic data. No problem as a temporary file during a processing workflow, but unwise as long-term storage.

But disk space is cheap. Less than one UK penny per exposure covers the cost of multiple drives for backups and archive for ten years, or whatever we take the life of a disk drive to be.

8

Take a look at the full metadata here: https://raw.pixls.us/getfile.php/4466/exif/A1_full_lossless_compressed.ARW.exif.txt

I’m >50% confident it is the lossless JPEG described in the DNG spec. :wink: Don’t think tile size plays a factor for the algorithm, apart from the I/O speed and memory tradeoff. The predictor could be an “unusual” one though, like Apple is using in their linear raws…

If someone feels inclined, I venture this can be read as an experiment in Python using tifffile+imagecodecs (don’t have time to try it out right now myself). Won’t work probably as there is another CFA specific step needed.

1 Like
9

Now that I took another look at the metadata, it’s very weird: it has both strips (expected for this ljpeg scheme as you point out) and tiles (and StripOffset=first TileOffset)! They should be exclusive in a TIFF… :exploding_head: I guess inspecting the JPEG header at the start of the bytestream specified by that offset would reveal what is the actual case here, and whether Sony made a faux pas here…

And, indeed, tiles it is, so every tile is ljpeg compressed individually…

10

Modern file systems support transparent compression, and if this is just wasted space, they should compress quite well. Try zfs or btrfs with a high level of compression.

1 Like
11

For archival reason I prefer uncompressed original file and use appropriate archive compress algos.
Image embedded compress algos are most weak in compress-ratio or non-appropriate for archival purpose.

I use lzip for archival compression.

Anyway, it is pity to have unneeded 16-bit with only 14-bit payload bloated original file which can not redeemed by any generic compression algo later.

Lets compare some compression algos:
74297243 a7rm4a-cap_on-iso12800-30sec.arw.lz (lzip -m273 -s512MiB)
74354112 a7rm4a-cap_on-iso12800-30sec.arw.xz (xz -9e)
75733776 a7rm4a-cap_on-iso12800-30sec.dng (DNG 1.6 compressed w/o JPEG preview)
98162569 a7rm4a-cap_on-iso12800-30sec.d0e9.jxl (JPEG XL encoder v0.5.0 [-d0 -e9])
110554082 a7rm4a-cap_on-iso12800-30sec.zstd.tif (TIF with embedded Zstd)
123365376 a7rm4a-cap_on-iso12800-30sec.arw (original file [~61Mpixel@16-bit])
124219186 a7rm4a-cap_on-iso12800-30sec.none.dng (DNG 1.6 uncompressed w/o JPEG preview)
126330560 a7rm4a-cap_on-iso12800-30sec.lzma.tif (TIF with embedded lzma)
127946613 a7rm4a-cap_on-iso12800-30sec.png (PNG max compression level 9)
129964199 a7rm4a-cap_on-iso12800-30sec.jp2 (JPEG2000 lossless)
134810062 a7rm4a-cap_on-iso12800-30sec.zip.tif (TIF with embedded zip)
167087712 a7rm4a-cap_on-iso12800-30sec.lzw.tif (TIF with embedded lzw)
183034792 a7rm4a-cap_on-iso12800-30sec.none.tif (TIF w/o compression [~61Mpixel@3*8-bit])

DNG 1.6 also use 16bit per pixel for 14bit senor payload.

12

Yeah, they had more than a few years to fix that.

13

I don’t think much other cameras actually use a 14bit file format. I think they all use 16bit, with different methods of compression.it makes processing of the data much easier to just read 16bits at a time.

You are talking about 2 bit or 4 bit per pixel of wasted space. Wow you’re crazy.

It makes the format easier to read and write, so maybe there are even speed benefits :).

But stop hurting your brain about stuff like this…

Now, the maker of flif format experimented a bit with writing the 4 Bayer channels out to monochrome files, and a extra data layer containing the rest of the file (like all Metadata, etc…). Compress those 4 layers with lossless compression of your choice.

Then, afterwards, those 4 layers could be decompressed and with the extra data layer the original raw file could be reconstructed bit for bit exactly the same.

But it was a hassle to support more formats, and with all the different compressions used out there the gains were not always worth it.

I would love a tool that used jpegxl to compress raw Bayer data with the ability to get the original file back… But in these days of storage availability, it’s quickly not worth the time invested.

14

You are talking about 2 bit or 4 bit per pixel of wasted space. Wow you’re crazy.

Yes, wasting 1/4 or 1/8 space is “crazy” :slight_smile:

I would love a tool that used jpegxl to compress raw Bayer data with the ability to get the original file back… But in these days of storage availability, it’s quickly not worth the time invested.

Sad, JPEG XL at lossless and max effort is still behind generic compression.
98162569 a7rm4a-cap_on-iso12800-30sec.d0e9.jxl JPEG XL encoder v0.5.0 (-d0 -e9)

15

Yes but jpegxl leaves you with a viewable image, where lzip et. al. does not.

16

I don’t know about the newer bodies like the A1, but uncompressed 14bit files from my A7III span from 0-16383 when opened in RawDigger (15871 after black point subtraction). Are you saying that these are still 16bit files with 4x more blank bits than actual data bits?

17

What do you call ‘generic compression’? Are you comparing a debayered 16bit image to the raw file compressed with 7zip or something?

The raw file is a monochrome file. If you write it as a TIFF file or something first, you’re comparing a lossless-encoding of a RGB 16-bit per channel image, (48bit per pixel), vs the RAW file which is single channel 16bit (16bit per pixel).
Debayering (demosaicing) ‘generates’ extra data, which you’re then asking to compress. You lose the possibilities of the RAW file (white balance applied, demosaicing applied, color-matrix applied) and you make the file bigger.

JPEG-XL’s near-lossless-mode (VarDCT) is groundbreaking vs the other formats out there, and is actually good at preserving small details and grain and stuff like that (where other ‘modern image formats’ like to smooth details away).

Use dcraw to convert the raw data to bayer pixel data (so a 61 megapixel monochrome file), use imagemagick to select the right pixels, so you get the 4 channels inside it in separate files (R, G1, G2, B) and compress those with jpeg-xl in losssmode mode. No way a generic data compressor can beat that. But like I said, there is no tool yet to reconstruct the original raw file again :).

(Although if you visit encode.su, there are a lot of generic compressors which are smart enough to ‘recognize image data’ and actually use a lossless image compressor behind the scenes).

I’ve done a little test with a Sony A7Rm4 file I downloaded from Sony A7R IV Sample Images (SOOC JPGs + RAW) - AlphaShooters.com.

Raw lossy (Which isn’t that bad at all, but people freak out about it being lossy): 61.684.736 bytes.
The bayered uncompressed data inside of it: 122.419.200 bytes (raw pixel values, no file format header or something).

That is 30.604.800 bytes per channel (R, G0, G1, B).
Compress those 4 channels with Jpeg-XL gives me:

r: 9.042.750 bytes
g0: 10.840.330 bytes
g1: 10.835.900 bytes
b: 10.575.782 bytes

41.294.762 bytes total, 33.7% of the original raw pixel data, or 66.95 of the Sony compressed raw file.

7z with lzma2 and a large dictionary + ultramode gives me 51.176.749 bytes when I compress the original (compressed) ARW file.

Funny enough, it gives me 47.061.852 bytes when I compress the original raw monochrome pixel data.

What I think is even more interesting - and a bit depressing - is that JPEG XR (the old 2009 Microsoft HD Photo format) which is actually meant to be used inside cameras can compress the raw data to 43.156.396. Lossless still. JPEG XR is also an open specification, and is based on jpeg math with a bit bigger matrices and memory requirement. But it compresses and decompresses almost instantly on my system, should be way more easy to implement in fixed-function processing chips that already use jpeg, and comes close. I never understood why it was never used. The compression-gain-vs-processing-speed is still best in class after all these years, supports lossy and lossless, up to 16bit per channel.

Anyway, If you really want to archive smaller files, save a small DNG or a preview JPG and 7zip the original RAW file. You’ll get the easiest compression that is quite good.

If you expect camera makers to use any algorithm you think is nice inside a RAW file, don’t get your hopes up. And I prefer speed and simplicity in the camera. A bigger storage card is an easy solution.

There are external tools to compress RAW files, but you have to uncompress them again to use them in any software (I believe, not sure). In those cases, just 7zip the ARW and be done with it.

Maybe someone has the time and energy to write a tool to split a raw file into it’s channels, use an external tool to compress them, and then can reverse the process. But it will require some knowledge of all different raw formats to get them back bitperfect.

Compressing to a DNG would be possible, but Adobe already does that, so :man_shrugging: .

Being annoyed with the difference between 12bit vs 16bit per pixel, or 14bit vs 16bit per pixel of uncompressed data… there are bigger issues in the world to think about.

1 Like
18

Not 4x more - 2 blank bits per 14 useful ones.

Yes a waste of space, except that in many cases, loading/saving aligned data is much faster. Crossing alignment boundaries is a great way to kill performance.

1 Like