Darktable unable to process DNG from Panasonic LX1

LebedevRI · June 19, 2018, 6:19pm

To be noted, native LX1 raws should work just fine.

virtual void rawspeed::LJpegDecompressor::decodeScan(), line 101: Tile component width (17) is not multiple of LJpeg CPS (2)

Hm, that is unusual one.
Does the current version of dng converter still produce such strange dng’s?

github.com

darktable-org/rawspeed/blob/55b7a49b3d59a7c5c2485bab04cfba5e14413c89/src/librawspeed/decompressors/LJpegDecompressor.cpp#L98-L102

    
      
          const auto tilePixelBlocks = mRaw->getCpp() * w;
          if (tilePixelBlocks % frame.cps != 0) {
            ThrowRDE("Tile component width (%u) is not multiple of LJpeg CPS (%u)",
                     tilePixelBlocks, frame.cps);
          }

LebedevRI · June 19, 2018, 6:34pm

@Luemmel please open an issue https://redmine.darktable.org/projects/darktable/issues/new

LebedevRI · June 19, 2018, 6:37pm

@Luemmel the current adobe dng converter does not produce such broken raws.
Can you confirm that your raw originates straight from ADC, and was not modified otherwise?
Please contribute it to https://raw.pixls.us/, otherwise i will not be able to do anything
(yes, ignore what it says about no DNG files created with Adobe DNG Converter)

LebedevRI · June 19, 2018, 6:52pm

Correction: current ADC does produce such raws, so we’re good.
But i still want a bugreport.

Luemmel · June 19, 2018, 6:54pm

@LebedevRI I have a lot of these files. They were converted by Adobe Bridge CC 2018 straight from the original raw. I don’t understand, what is “strange” in the file, but I have this problem with all DNG converted from LX1-RAW since 2006. They were sometimes converted by Adobe DNG Converter, sometimes by Lightroom and since 2 years by Bridge. Example 2006: 2006-09-24-1348-41.dng (6.9 MB)

I will upload the file and open an issue. Thank you for taking care, it is the last barrier to switch completely to dt

LebedevRI · June 19, 2018, 6:55pm

Hm, maybe i should think this through before commenting, shouldn’t i

@Luemmel better idea: can you contribute a normal raw from that camera to https://raw.pixls.us/
The current one is not under CC0, which is really bad.

Strange as in, this failure mode was not present in any other sample present on RPU.

Luemmel · June 19, 2018, 7:00pm

Too late, already uploaded. Take it as CC0

LebedevRI · June 19, 2018, 7:02pm

I have seen the uploaded DNG sample, but we really really want a normal RW2 sample under CC0, too.
Reason being, i would like to have a matching pair of the original RW2, and a converted DNG with this issue.
And i can’t do that since the existing RW2 sample is not under CC0…

Luemmel · June 19, 2018, 7:03pm

OK, I will shoot one. But it isn’t rw2, that came with LX2. It is RAW.

LebedevRI · June 19, 2018, 7:04pm

Sure, not the point

Luemmel · June 19, 2018, 7:24pm

OK, I uploaded unmodified RAW straight from the camera and DNG saved by Bridge, both to redmine. Should I upload them elsewhere, too?

LebedevRI · June 19, 2018, 7:26pm

To the https://raw.pixls.us/
Only that one, not the DNG. I’ll take care of DNG one.

Luemmel · June 19, 2018, 7:28pm

Done. Again, thank you! I promise to not convert to dng anymore

LebedevRI · June 20, 2018, 2:15pm

github.com/darktable-org/rawspeed

LJpegDecompressor: support degenerate raws with width % 2 != 0

committed 01:38PM - 20 Jun 18 UTC

LebedevRI

+72 -28

This is needed e.g for DNG's converted from Panasonic DMC-LX1 raws. I have no w…ords. What were they (Adobe) thinking. They discard padding columns, that were garbage, and end up with such an odd raw, where the tiles (LJPEG-compressed) still have padding, yet the DNG image sizes specify that it has to be discarded. What is even worse, the image width is not multiple of two, and the LJPEG's components-per-pixel is 2, so we have to very-special-handle this case, to not completely discard that last required block (then we'd be producing nondeterministic garbage in those last few columns), but also not fully decode it, because we don't have anywhere to write it, We could, of course, introduce something like "internal padding columnts", in fact we already pad the rows to be 16-byte aligned, but that padding is used for ASAN overflow detection. So right now i'm not seeing any more reasonable fix. :/ Fixes https://redmine.darktable.org/issues/12239

Luemmel · June 20, 2018, 7:18pm

Thank you for fixing it. Can’t wait to try it.

darix · June 20, 2018, 10:53pm

is this in DT master already?

LebedevRI · June 21, 2018, 6:23am

github.com/darktable-org/darktable

RawSpeed submodule update: LJpegDecompressor: support degenerate raws with width % 2 != 0

committed 02:13PM - 20 Jun 18 UTC

LebedevRI

+1 -1

This is needed e.g for DNG's converted from Panasonic DMC-LX1 raws. I have no w…ords. What were they (Adobe) thinking. They discard padding columns, that were garbage, and end up with such an odd raw, where the tiles (LJPEG-compressed) still have padding, yet the DNG image sizes specify that it has to be discarded. What is even worse, the image width is not multiple of two, and the LJPEG's components-per-pixel is 2, so we have to very-special-handle this case, to not completely discard that last required block (then we'd be producing nondeterministic garbage in those last few columns), but also not fully decode it, because we don't have anywhere to write it, We could, of course, introduce something like "internal padding columnts", in fact we already pad the rows to be 16-byte aligned, but that padding is used for ASAN overflow detection. So right now i'm not seeing any more reasonable fix. :/ Fixes #12239

darix · June 21, 2018, 11:09am

So @Luemmel can test with our master snapshot packages

Luemmel · June 21, 2018, 11:53am

I would love to do so, @darix, but the snapshots are Linux only, aren’t they? I’m on Windows, I thus will have to wait until next release, I guess.

Luemmel · June 21, 2018, 2:07pm

Just set up a virtual machine with Ubuntu and tried the snapshot: Works great! Thank you, @LebedevRI und @darix. Can’t wait for the Windows build.