Nikon: a specific raw sample wanted.

rawspeed
rpu
darktable

(Roman Lebedev) #1

Hi everyone.

Pretext blurb

As you may or may not know, nikon cameras are rather interesting piece of work.
The compression is configurable (three different types), the bitdepth is configurable (2 variants).
That already results in a 6 different raw types.

If only it was that simple. :confused:

Under some unknown conditions, some nikon cameras, when configured to store lossy compressed raw files, actually produce some strange modification of raw files, that internally are split in two, with slightly different compression between two halves. I’m not aware of any official documentation about it, but i suppose it is most often refereed to as “lossy after split”.

That oddity does not seem to be stated in the EXIF, only deep within nikon makernotes, and, if decoded properly, there is no difference between normal lossy compressed raws, and these raws with split. Right now i’m aware of 1 J5 1 and D3400 2 producing such raws.

We always create our own problems, don’t we?

After the maintainership of the rawspeed library, which is used by darktable to load the raw files, was transferred to us late last year (2016-12), the RPU was created/resurrected right after that. And then the “cpr” on the library begun.

Now, here is the problem, due to the huge [unnessesary] rush, [which partially led to] and a gross code review mishap, the support for these NEF with split raws was broken. Since RPU was very new back then, and due to the [unnessesary] rush, there was no such sample, so the breakage was not detected. Later, it was finally caught by Wolfgang Goetz, and nowadays we have two 1 2 samples.

That is great, and i will finally be able to work on unbreaking that support. The one small caveat is that both of these samples are 12-bit.
It would be really nice to also find the 14-bit raw with the same problem.

HOW TO DETECT IF THAT IS THE RAW WE ARE LOOKING FOR

Now, as i have already said, i’m not aware of any way to auto-detect that raw easily using exiv2/exiftool. The following command can help you check compression and bitdepth of the raws.

$ find -iname \*.nef -print0 | sort -z | xargs -0 exiftool -compression* -nefcompression -bitspersample
======== name.nef                                                                                                                                                                                                                                    
Compression                     : Nikon NEF Compressed                                                                                                                                                                                                                         
NEF Compression                 : Lossy (type 2)                                                                                                                                                                                                                               
Bits Per Sample                 : 14

If the output matches this example, please try opening that raw in a fresh darktable (from git master, NOT 2.2.x or earlier). If it fails to load, and in console you see the following message, you have found it!

[rawspeed] (name.nef) void rawspeed::HuffmanTable::setCodeValues(const rawspeed::Buffer &), line 176: Corrupt Huffman. Code value 92 is bigger than 16

Please help! :slight_smile:


darktable 2.4.0rc0 released
darktable 2.4.0rc2 released
(Hevii Guy) #2

I’ve found quite a few (Lossy type 2, 14 bits per sample) but I’m not comfortable (experienced) with building a fresh darktable. If you can provide some detailed direction, I’d be happy to give it a shot. In fact, I’ve always wanted to be able to try the latest and greatest features before general release!


(Gord) #3

FWIW, the original photo for this PlayRaw is Lossy type 2, 14 bits per sample. I don’t have time right now to throw up a fresh DT to see whether it has the problem.


(Roman Lebedev) #4

Sadly (fortunately?), that raw is not one of these split raws, loads fine.

You’ll have to be more specific than that.
Building. For fedora/opensuse there are packages by @darix


(Pascal Obry) #5

I have found some on my hard drive, the good news is that all of those RAW are opening fine with current master.


(Roman Lebedev) #6

Thank you for checking! Hopefully someone will have worse news than that :slight_smile:


(Roman Lebedev) #7

Up. Still looking :slight_smile:


(Mike Romanov) #8

I think that such splited RAW are obtained when using a slow flash card (for example, Сlass 10) for long serial shooting. Maybe someone will try to?


#9

@LebedevRI-
I have 2.2.5 installed.

I pulled 2.4.0 RC2 from git master.

I have at least two images that are 14-bit lossy Nikon RAWs that fail to load using 2.4.0 RC2, but load and have been edited in 2.2.x.

The images were taken with a Nikon D5500.

Here’s an exiftool snippet:

======== ./DSC_0248.NEF
Compression                     : Nikon NEF Compressed
NEF Compression                 : Lossy (type 2)
Bits Per Sample                 : 14
======== ./DSC_0258.NEF
Compression                     : Nikon NEF Compressed
NEF Compression                 : Lossy (type 2)
Bits Per Sample                 : 14

Here is a snippet of the errors I saw with 2.4.0 RC2:

[rawspeed] (DSC_0248.NEF) void rawspeed::HuffmanTable::setCodeValues(const rawspeed::Buffer&), line 176: Corrupt Huffman. Code value 92 is bigger than 16
[temperature] failed to read camera white balance information from `DSC_0248.NEF'!
allocation failed???
[rawspeed] (DSC_0258.NEF) void rawspeed::HuffmanTable::setCodeValues(const rawspeed::Buffer&), line 176: Corrupt Huffman. Code value 92 is bigger than 16
[temperature] failed to read camera white balance information from `DSC_0258.NEF'!
allocation failed???

@mikrom-
The images were taken with a SanDisk Extreme Pro UHS-I SDXC, so I can at least confirm that it doesn’t only occur with slow cards.


(Roman Lebedev) #10

@lptech
Hi!

Oh great, sounds just like what i was looking for!

Could you please contribute at least one of such raws to the https://raw.pixls.us/, please? :slight_smile:


#11

@LebedevRI-

I uploaded one of the RAWs.

Do you expect to have this fixed before the 2.4.0 official release, as a 2.4.x patch, or in a later release entirely?

Thank you!


(Roman Lebedev) #12

Well, given that 2.4.0 has happened already… :slight_smile:

Thanks, but… I don’t suppose there is some other raw without people in it?


#13

@LebedevRI-

I don’t see any images in my library that appear to be 14-bit “lossy after split” Nikon RAWs that don’t have people in them.

While it perhaps isn’t the best for long-term public verification/testing (as I see the image uploader rule), could you use the image for development and hope someone uploads another sometime in the future?

Thanks!


(Roman Lebedev) #14

I’m afraid i can’t really give any credible time estimates. “Once it’s done”.

Ok then, this is not too bad, too.
I’m sure this will bite someone else and there will be more samples :slight_smile:

Thank you for this sample!