Compressed DNG support?

70MM13 · January 8, 2018, 2:57pm

It would be of great benefit to those of us doing cinema work with rawtherapee if you enabled support for compressed DNG files. It would not only save hundreds of gigabytes of storage space, but also remove the extra step of having to decompress the source footage just to load into rawtherapee!

Thank you!

Morgan_Hardwood · January 8, 2018, 3:49pm

@70MM13 please upload a sample raw file to https://raw.pixls.us/ or https://filebin.net/ and link it here.

70MM13 · January 8, 2018, 4:21pm

Uploaded sample to your database, but am unable to provide link. I tried unsuccessfully to find a way to do so. Sorry, it may be a phone browser issue. I’m sure you will be able to find it! I described it as something like “compressed dng sample frame”

Thanks.

Morgan_Hardwood · January 8, 2018, 4:33pm

Thanks for uploading @70MM13, @LebedevRI found it:
https://raw.pixls.us/getfile.php/2204/nice/Canon%20-%20EOS%205D%20Mark%20III%20-%2014bit%20(2.3471882640587).dng

Morgan_Hardwood · January 8, 2018, 4:42pm

Opened issue #4285

Jed_Smith · July 9, 2018, 1:42am

I would love to see compressed cinema dng support added to RawTherapee as well. I’ve added a few more example images, and some information that might be helpful on the issue:

github.com/Beep6581/RawTherapee

Support for JPEG-compressed DNG files

opened 04:41PM - 08 Jan 18 UTC

closed 05:29PM - 29 Nov 18 UTC

Beep6581

scope: file format type: enhancement

Request from: https://discuss.pixls.us/t/compressed-dng-support/6370 Raw fil…e: https://raw.pixls.us/getfile.php/2204/nice/Canon%20-%20EOS%205D%20Mark%20III%20-%2014bit%20(2.3471882640587).dng The DNG file seems to come from Magic Lantern and uses JPEG compression. ```text -TIFF:ExifByteOrder=Little-endian (Intel, II) -TIFF-IFD0:BitsPerSample=14 -TIFF-IFD0:Compression=JPEG -TIFF-IFD0:Make=Canon -TIFF-IFD0:Model=Canon EOS 5D Mark III -TIFF-IFD0:Software=MLV App -TIFF-IFD0:DNGVersion=1.4.0.0 ```

Thanks!

70MM13 · July 9, 2018, 1:49am

Yes!

It’s badly needed!

paperdigits · July 9, 2018, 2:47am

I wonder if @LebedevRI has an interest in supporting this in rawspeed?

LebedevRI · July 9, 2018, 6:17am

Unlike GoPro, this would be kinda trivial.
Basically, you would “only” need to change

github.com

darktable-org/rawspeed/blob/e9eafa1ab3c8eaee852a968fa625391f8d4dbc90/src/librawspeed/decompressors/LJpegDecompressor.cpp#L193

    
      
          
          
copy_n(predNext, N_COMP, pred.data());
          // the predictor for the next line is the start of this line
          predNext = dest;
          
          
unsigned x = 0;
          
          
// For x, we first process all full pixel blocks within the image buffer ...
          for (; x < fullBlocks; ++x) {
            unroll_loop<N_COMP>([&](int i) {
              *dest++ = pred[i] += ht[i]->decodeNext(bitStream);
            });
          }
          
          
// Sometimes we also need to consume one more block, and produce part of it.
          if /*constexpr*/ (WeirdWidth) {
            // FIXME: evaluate i-cache implications due to this being compile-time.
            static_assert(N_COMP > 1 || !WeirdWidth,
                          "can't want part of 1-pixel-wide block");
            // Some rather esoteric DNG's have odd dimensions, e.g. width % 2 = 1.
            // We may end up needing just part of last N_COMP pixels.

That is prediction, where the previous pixel is the only pixel used for prediction, i.e. mode 1.
CinemaDNG uses different prediction scheme, mode 6. (last time i checked?)
See Lossless JPEG - Wikipedia for the modes explainations.

The downside of that is combinatorial explosion…

So TLDR: yes, but not yet.