error reading a nef file

masmys · August 23, 2018, 4:20pm

I am running daraktable (2.4.4-1) on debian testing.

In a batch of photos taken with my Nikon D750, darktable fails to load an image with the following message:

[temperature] failed to read camera white balance information from `dsc_0059.nef'!
[temperature] `NIKON CORPORATION NIKON D750' color matrix not found for image
[temperature] failed to read camera white balance information from `dsc_0059.nef'!

The same file can be opened with rawtherapee. In a batch of 100 or so, I have seen this happen with about five or six. This is a recent occurrence. But for those few files, all others open correctly.

Tried attaching the file here. Got an error message telling that the extension is not allowed! The file is available here: image

Jacal · August 23, 2018, 5:09pm

Same thing with Windows version.

Claes · August 23, 2018, 5:11pm

Same thing with Linux Mint 19 and dt 2.4.4

heckflosse · August 23, 2018, 5:17pm

Though RawTherapee opens the file, it also reports in console:

dsc_0059.nef: Corrupt data near 0x1c7823f

Jacal · August 23, 2018, 5:25pm

No problems with Nikon - D750 - 14bit compressed (Lossless) (3_2).NEF from raw.pixls.us, so it looks like a very specific or somehow corrupted file.

paperdigits · August 23, 2018, 5:55pm

Perhaps time to format the SD Card and/or run a bad blocks test on it.

masmys · August 24, 2018, 12:25am

Will try that. IIRC, I had done that. I will try again and shoot random images and check whether I get any corrupt images.

Jacal · August 24, 2018, 12:39am

(I have been able to convert the problematic photo to DNG (via Adobe’s software) and open it in dt or RT without issues. Just in case, as a possible workaround.)

masmys · August 24, 2018, 2:03am

Was about to post something similar As rawtherapee opens the file despite the error, could not dt do the same? If as the message suggests, it is only the temperature data that is problematic, personally I see no problem in dt allowing us to edit/process the file.

Is it possible for you to upload the converted file?

Jacal · August 24, 2018, 2:30am

I can’t give you a more useful answer, but the two programs are using quite different tools in the background. (Nufraw, on the other hand, reports the same error as RawTherapee, perhaps since both are using some of dcraw abilities.)

Tobias · August 24, 2018, 7:07am

@LebedevRI could you look at the files, you are the dt expert for this.

LebedevRI · August 24, 2018, 7:48am

It’s [2/3] Regression: huffmantable is bad for last decode · Issue #128 · darktable-org/rawspeed · GitHub plus It seems BitStream/BitPumps are broken, too · Issue #132 · darktable-org/rawspeed · GitHub

TLDR: the raw compressed data is a stream of 1 and 0, plus some padding
because of the data endianness/packing. For performance reasons, we don’t
just consume one bit(!) at the time, but use some abstractions, that consume
several bytes at the time, cache them, and thus improve performance.

But that only works if that cache actually can consume the amount of bytes it wants to…
And in this case, it can not, there isn’t enough padding. [un]fortunately, that happens rarely.

Possibly this can be solved by pretending there is some zero-bit padding when reading, i.e. in

github.com

darktable-org/rawspeed/blob/e011a20dce52a1833a3d879a4203c55065905c71/src/librawspeed/io/BitStream.h#L113-L156

    
      
          inline void fillSafe() {
            assert(data);
            if (pos + BitStreamCacheBase::MaxProcessBytes <= size) {
              uchar8 tmp[BitStreamCacheBase::MaxProcessBytes] = {0};
              assert(!(size - pos < BitStreamCacheBase::MaxProcessBytes));
              memcpy(tmp, data + pos, BitStreamCacheBase::MaxProcessBytes);
              pos += fillCache(tmp);
            } else if (pos < size) {
              uchar8 tmp[BitStreamCacheBase::MaxProcessBytes] = {0};
              assert(size - pos < BitStreamCacheBase::MaxProcessBytes);
              memcpy(tmp, data + pos, size - pos);
              pos += fillCache(tmp);
            } else if (pos < size + Cache::MaxGetBits / 8) {
              // yes, this case needs to continue using Cache::MaxGetBits
              // assert(size <= pos);
              cache.push(0, Cache::MaxGetBits);
              pos += Cache::MaxGetBits / 8;
            } else {
              // assert(size < pos);
              ThrowIOE("Buffer overflow read in BitStream");

This file has been truncated. show original

heckflosse · August 24, 2018, 9:42am

The last line of the image is corrupted as you can see in this screenshot where border is set to zero and demosaic is disabled.

LebedevRI · August 24, 2018, 11:12am

That’s why that border exists…
Normally it isn’t zero (in case of that raw it’s supposed to crop two last rows).

heckflosse · August 24, 2018, 11:30am

It’s perfectly fine to use zero border in undemosaiced mode to see where the data corruption is.

LebedevRI · August 24, 2018, 11:42am

Perhaps we mean different things by last line of the image is corrupted.
I’m not talking about that line/image pixels being unusable.
i’m talking about data corruption in the sense of bit flips/file truncations.
What i was saying is, this does not seem to be data corruption.

heckflosse · August 24, 2018, 12:12pm

Why do you think it is not data corruption?

Here’s what I get when reporting all errors, not just the first one:

dsc_0059.nef: Corrupt data near 0x1c7823f
dsc_0059.nef: Corrupt data near 0x1c78242
dsc_0059.nef: Corrupt data near 0x1c78244
dsc_0059.nef: Corrupt data near 0x1c78248
dsc_0059.nef: Corrupt data near 0x1c7824b
dsc_0059.nef: Corrupt data near 0x1c7824e
dsc_0059.nef: Corrupt data near 0x1c78250
dsc_0059.nef: Corrupt data near 0x1c78252
dsc_0059.nef: Corrupt data near 0x1c78255
dsc_0059.nef: Corrupt data near 0x1c78257
dsc_0059.nef: Corrupt data near 0x1c78259
dsc_0059.nef: Corrupt data near 0x1c7825b
dsc_0059.nef: Corrupt data near 0x1c7825e
dsc_0059.nef: Corrupt data near 0x1c78260
dsc_0059.nef: Corrupt data near 0x1c78263
dsc_0059.nef: Corrupt data near 0x1c78265
dsc_0059.nef: Corrupt data near 0x1c78267
dsc_0059.nef: Corrupt data near 0x1c7826a
dsc_0059.nef: Corrupt data near 0x1c7826c
dsc_0059.nef: Corrupt data near 0x1c7827a
dsc_0059.nef: Corrupt data near 0x1c7827d
dsc_0059.nef: Corrupt data near 0x1c78280
dsc_0059.nef: Corrupt data near 0x1c78283
dsc_0059.nef: Corrupt data near 0x1c78285
dsc_0059.nef: Corrupt data near 0x1c78287
dsc_0059.nef: Corrupt data near 0x1c78289
dsc_0059.nef: Corrupt data near 0x1c7828b
dsc_0059.nef: Corrupt data near 0x1c7828d
dsc_0059.nef: Corrupt data near 0x1c7828f
dsc_0059.nef: Corrupt data near 0x1c78291
dsc_0059.nef: Corrupt data near 0x1c78293
dsc_0059.nef: Corrupt data near 0x1c78295
dsc_0059.nef: Corrupt data near 0x1c78298
dsc_0059.nef: Corrupt data near 0x1c7829a
dsc_0059.nef: Corrupt data near 0x1c7829c
dsc_0059.nef: Corrupt data near 0x1c7829e
dsc_0059.nef: Corrupt data near 0x1c782a0
dsc_0059.nef: Corrupt data near 0x1c782a2
dsc_0059.nef: Corrupt data near 0x1c782a4
dsc_0059.nef: Corrupt data near 0x1c782a7
dsc_0059.nef: Corrupt data near 0x1c782a9
dsc_0059.nef: Corrupt data near 0x1c782ab
dsc_0059.nef: Corrupt data near 0x1c782ad
dsc_0059.nef: Corrupt data near 0x1c782af
dsc_0059.nef: Corrupt data near 0x1c782b1
dsc_0059.nef: Corrupt data near 0x1c782b3
dsc_0059.nef: Corrupt data near 0x1c782b6
dsc_0059.nef: Corrupt data near 0x1c782b8
dsc_0059.nef: Corrupt data near 0x1c782ba
dsc_0059.nef: Corrupt data near 0x1c782bc
dsc_0059.nef: Corrupt data near 0x1c782be
dsc_0059.nef: Corrupt data near 0x1c782c0
dsc_0059.nef: Corrupt data near 0x1c782c3
dsc_0059.nef: Corrupt data near 0x1c782c6
dsc_0059.nef: Corrupt data near 0x1c782c9
dsc_0059.nef: Corrupt data near 0x1c782dc
dsc_0059.nef: Corrupt data near 0x1c782df
dsc_0059.nef: Corrupt data near 0x1c782e2
dsc_0059.nef: Corrupt data near 0x1c782e4
dsc_0059.nef: Corrupt data near 0x1c782e7
dsc_0059.nef: Corrupt data near 0x1c782e9
dsc_0059.nef: Corrupt data near 0x1c782ec
dsc_0059.nef: Corrupt data near 0x1c782ee
dsc_0059.nef: Corrupt data near 0x1c782f0
dsc_0059.nef: Corrupt data near 0x1c782f2
dsc_0059.nef: Corrupt data near 0x1c782f5
dsc_0059.nef: Corrupt data near 0x1c782f7
dsc_0059.nef: Corrupt data near 0x1c782f9
dsc_0059.nef: Corrupt data near 0x1c782fb
dsc_0059.nef: Corrupt data near 0x1c782fd
dsc_0059.nef: Corrupt data near 0x1c782ff
dsc_0059.nef: Corrupt data near 0x1c78301
dsc_0059.nef: Corrupt data near 0x1c78303
dsc_0059.nef: Corrupt data near 0x1c78305
dsc_0059.nef: Corrupt data near 0x1c78307
dsc_0059.nef: Corrupt data near 0x1c7830a
dsc_0059.nef: Corrupt data near 0x1c78312
dsc_0059.nef: Corrupt data near 0x1c78315
dsc_0059.nef: Corrupt data near 0x1c78317
dsc_0059.nef: Corrupt data near 0x1c7831a
dsc_0059.nef: Corrupt data near 0x1c7831c
dsc_0059.nef: Corrupt data near 0x1c7831e
dsc_0059.nef: Corrupt data near 0x1c78320
dsc_0059.nef: Corrupt data near 0x1c78322
dsc_0059.nef: Corrupt data near 0x1c78339
dsc_0059.nef: Corrupt data near 0x1c7833b
dsc_0059.nef: Corrupt data near 0x1c7833d
dsc_0059.nef: Corrupt data near 0x1c7833f
dsc_0059.nef: Corrupt data near 0x1c78342
dsc_0059.nef: Corrupt data near 0x1c78344
dsc_0059.nef: Corrupt data near 0x1c78346
dsc_0059.nef: Corrupt data near 0x1c78348
dsc_0059.nef: Corrupt data near 0x1c7834a
dsc_0059.nef: Corrupt data near 0x1c7834c
dsc_0059.nef: Corrupt data near 0x1c7834e
dsc_0059.nef: Corrupt data near 0x1c78350
dsc_0059.nef: Corrupt data near 0x1c78352
dsc_0059.nef: Corrupt data near 0x1c78355
dsc_0059.nef: Unexpected end of file

LebedevRI · August 24, 2018, 12:21pm

I’m sorry, but i have no idea what that text means.