Darktable: Unable to open a number of .NEFs in darktable, rawtherapee but OK in LR and PS

@paperdigits, you raise a good point, but I’m so old and it has been so many years since I switched to Fuji that I can’t remember. It’s possible that I used some Nikon software to transfer from the camera, but by the time I took these photos I’m pretty certain I was using LR 1.0 (having been a Beta user) and already had a slow, el-cheapo USB SD Card reader (still in use!) which avoided the need for Nikon software. Further more, I seem to remember that I was using PhotoMechanic (early version) to ingest. Either way, I assume that Adobe had to make some modification in LR 1.0 to enable it to read any potentially corrupted NEFs which had been ingested.

But if anybody has a clearer memory about this I would be grateful as it may give a clue as to a work around.

The following image made available under the terms of the CC BY permissive licence.

DSC_2400.nef (5.1 MB)

I trust the ‘drag & drop’ post I made a minute or so ago gives the result people want ?

Looking at your file with a binary editor shows this:

    0000:0140 |                                        4E 49 4B 4F |             NIKO
    0000:0150 | 4E 20 43 4F  52 50 4F 52  41 54 49 4F  4E 00 4E 49 | N CORPORATION.NI
    0000:0160 | 4B 4F 4E 20  44 38 30 00  2C 01 00 00  01 00 00 00 | KON D80.,.......
    0000:0170 | 2C 01 00 00  01 00 00 00  43 61 70 74  75 72 65 20 | ,.......Capture 
    0000:0180 | 4E 58 20 31  2E 33 2E 33  20 57 00 02  32 30 30 38 | NX 1.3.3 W..2008
    0000:0190 | 3A 30 34 3A  32 34 20 31  30 3A 31 32  3A 34 36    | :04:24 10:12:46 

So it’ looks like at least some of those files were transferred with Capture NX and damaged in the transfer. But there are also a lot of mentions of Adobe, and things that look like processing instructions

Perhaps it’s worthwhile to try the utility mentioned bt @kmilos on a copy of the affected files?

Thanks for confirming the use of Capture NX; I tried the utility as suggested; it gives an error because the camera is not on the supported list. But it looks like a very convenient solution, - if it would work on my D80 Nefs!

Would you mind also sharing the dng?

As for the NEF, it doesn’t surprise me that LR and PS can handle this damaged file. Adobe is big enough to devote some of his programmers to help fixing Nikon damaged files caused by Nikon.
On the other way, it doesn’t surprise me that none of the raw editors I tried could open the file (darktable, Rawtherapee, ART, Photoflow, Filmulator, Rawproc, Gimp), for obvious reasons.
What really surprises me is that someone from the FOSS world indeed lost some time fixing the damage, for free, and for a considerable number of cameras (unfortunately, not the model you have).
Just some thoughts to put things in perspective - as I see it.

Agree; I have already been through this thought process my self - the Adobe action is the only explanation for LR and PS (directly or through LR) being able to open the file. the associated dng should be attached ,

The following image is made available under the terms of the CC BY permissive licence.DSC_2400.dng (2.0 MB)

1 Like

Thanks.
It feels like I’m editing a jpeg. The sky is greyish, the size is around 6 mega pixels (but the camera model is 10).
So, I would seek to convert to something as close to raw as possible, so you can get most information.
It seems you have used adobe dng converter, right?
I would open the nef in PS and try to export it as tiff, 16 bit color depth, a big color space (adobr rgb, linear rec 2020, not sRGB!), but I’m not an expert on this, maybe others can help.

Anyway, here’s what I get from a quick edit:


DSC_2400.dng.xmp (10.5 KB)
(darktable 3.2.1)

DSC_2400.dng is an already demosaiced file, not really raw…

Does this indicate it has also suffered some tone mapping?

image

No idea!

Thanks for this; better than my attempts in 3.2.1 because you clearly have more ‘darktable-hours’ than I. Yeah, I agree about the sky - it looks very much like a sub-compact point-and-shoot jpeg! I’m going to examine your history stack in detail.

Yes, your suggestion on exporting a tiff from a PS s what I had decide to do, not being all that impressed with what I could get from the dng. And, yes, I did use the latest version of the dng converter .

Yes, I think I vaguely understood that, which is why I have made very little use of dng in my workflow.

More or less. These guys don’t stop working and improving the software so I’m always pushed back to the beginners class :smiley:

Just make sure you can get as neutral as possible, don’t apply any effect in ACR and PS, and try to undo effects applied by default. That way, you will get a tiff as close to the raw as possible. And don’t forget about the tiff export settings regarding bit depth and color space (not sure if there are other settings you should also care about…)

Not necessarily, these are just part of the embedded ICC profile.

@LateJunction
Who knows what that old Capture NX software did to those NEFs, it doesn’t look like it actually contains any raw data at at all! Here’s the excerpt showing all image IFDs from exiftool -a -u -s -g1 and none of them is raw like in a normal NEF file:

---- IFD0 ----
SubfileType                     : Reduced-resolution image
ImageWidth                      : 320
ImageHeight                     : 213
BitsPerSample                   : 8 8 8
Compression                     : Uncompressed
PhotometricInterpretation       : RGB
Make                            : NIKON CORPORATION
Model                           : NIKON D80
StripOffsets                    : 34450
Orientation                     : Horizontal (normal)
SamplesPerPixel                 : 3
RowsPerStrip                    : 213
StripByteCounts                 : 204480
XResolution                     : 300
YResolution                     : 300
PlanarConfiguration             : Chunky
ResolutionUnit                  : inches
Software                        : Capture NX 1.3.3 W
ModifyDate                      : 2008:04:24 10:12:46
DateTimeOriginal                : 2007:10:18 10:34:55
TIFF-EPStandardID               : 1 0 0 0
PreviewTIFF                     : (Binary data 204696 bytes, use -b option to extract)
---- SubIFD ----
SubfileType                     : Reduced-resolution image
Compression                     : JPEG (old-style)
Orientation                     : Horizontal (normal)
XResolution                     : 1
YResolution                     : 0
ResolutionUnit                  : inches
JpgFromRawStart                 : 238930
JpgFromRawLength                : 3273332
JpgFromRaw                      : (Binary data 3273332 bytes, use -b option to extract)
---- SubIFD1 ----
SubfileType                     : Full-resolution image
Compression                     : JPEG (old-style)
Orientation                     : Horizontal (normal)
XResolution                     : 1
YResolution                     : 0
ResolutionUnit                  : inches
OtherImageStart                 : 3512262
OtherImageLength                : 1884562
OtherImage                      : (Binary data 1884562 bytes, use -b option to extract)

There is a thumbnail and some two JPEGs in there…

Going further, it looks like this full-resolution OtherImage in the NEF is indeed the demosaiced one that gets saved in the DNG (notice the same number of bytes) as:

---- SubIFD ----
SubfileType                     : Full-resolution image
ImageWidth                      : 2779
ImageHeight                     : 1853
BitsPerSample                   : 8 8 8
Compression                     : Lossy JPEG
PhotometricInterpretation       : Linear Raw
StripOffsets                    : 200632
SamplesPerPixel                 : 3
RowsPerStrip                    : 1853
StripByteCounts                 : 1884562
PlanarConfiguration             : Chunky
LinearizationTable              : (Binary data 1374 bytes, use -b option to extract)
BlackLevelRepeatDim             : 1 1
BlackLevel                      : 0 0 0
WhiteLevel                      : 65535 65535 65535
DefaultScale                    : 1 1
DefaultCropOrigin               : 0 0
DefaultCropSize                 : 2779 1853
ChromaBlurRadius                : 0
AntiAliasStrength               : 1
BestQualityScale                : 1
ActiveArea                      : 0 0 1853 2779

It’s saved as a compressed 8-bit JPEG, but the LinearizationTable and WhiteLevel tags give you a hint that the content is then expanded to 16-bits once decompressed.

So the DNG route is probably the best way to go forward with these, although the data underwent a (very?) lossy compression (global tone curve to fit into 8-bits as @gadolf suspected + JPEG), so you can’t ever expect the raw quality, that’s unfortunately gone for good it seems…

For the uber-interested, this is the contents of the LinearizationTable :wink:

image

What I find very weird, is the first “reduced-resolution” JpegFromRaw in the NEF of 3273332 bytes (3.12 MB), which is larger than the “full-resolution” one! That one seems to be gone in the DNG…

I guess it is the OOC (or Capture NX generated) JPEG? Here it is:

This is most interesting but, realistically, far above my intellectual peak (which I can never operate at anyway!). However, just a few comments:

  1. The images that I am able to mess around with in LR do not appear to be markedly different in quality to images taken at a later date which darktable can open, suggesting to me, in my relative ignorance, these two sets of images have similar levels of raw information. Certainly the image being managed within in LR is well within the range of raw images, from 3 different manufacturers, that I have been managing.
  2. The appearance of the jpeg which has bene included here seems to me to be the most faithful representation of the scene, especially in terms of the HSL characteristics of the brickwork, the grass and the copper beech tree in the foreground right. These is not Fuji jpeg but it look better, in my opinion, than that delivered by the dng with - supposedly - some raw data behind it.

As far as the linearization table - I’m pretty sure Nikon has the option of semi-lossy NEFs using nonlinear quantization. (Sony compressed ARWs are similar in that there is a nonlinear quantization curve in addition to the 11+7 strip quantization, the latter of which is notorious for causing some artifacts)

A reference “non-corrupted” NEF from the exact same camera might be useful.

Side track; I’ve been developing my NEFs without applying the linearization table - what am I missing? I can find no prose on the internets to explain it…