Yes, I think I vaguely understood that, which is why I have made very little use of dng in my workflow.
More or less. These guys don’t stop working and improving the software so I’m always pushed back to the beginners class 
Just make sure you can get as neutral as possible, don’t apply any effect in ACR and PS, and try to undo effects applied by default. That way, you will get a tiff as close to the raw as possible. And don’t forget about the tiff export settings regarding bit depth and color space (not sure if there are other settings you should also care about…)
Not necessarily, these are just part of the embedded ICC profile.
@LateJunction
Who knows what that old Capture NX software did to those NEFs, it doesn’t look like it actually contains any raw data at at all! Here’s the excerpt showing all image IFDs from exiftool -a -u -s -g1 and none of them is raw like in a normal NEF file:
---- IFD0 ----
SubfileType : Reduced-resolution image
ImageWidth : 320
ImageHeight : 213
BitsPerSample : 8 8 8
Compression : Uncompressed
PhotometricInterpretation : RGB
Make : NIKON CORPORATION
Model : NIKON D80
StripOffsets : 34450
Orientation : Horizontal (normal)
SamplesPerPixel : 3
RowsPerStrip : 213
StripByteCounts : 204480
XResolution : 300
YResolution : 300
PlanarConfiguration : Chunky
ResolutionUnit : inches
Software : Capture NX 1.3.3 W
ModifyDate : 2008:04:24 10:12:46
DateTimeOriginal : 2007:10:18 10:34:55
TIFF-EPStandardID : 1 0 0 0
PreviewTIFF : (Binary data 204696 bytes, use -b option to extract)
---- SubIFD ----
SubfileType : Reduced-resolution image
Compression : JPEG (old-style)
Orientation : Horizontal (normal)
XResolution : 1
YResolution : 0
ResolutionUnit : inches
JpgFromRawStart : 238930
JpgFromRawLength : 3273332
JpgFromRaw : (Binary data 3273332 bytes, use -b option to extract)
---- SubIFD1 ----
SubfileType : Full-resolution image
Compression : JPEG (old-style)
Orientation : Horizontal (normal)
XResolution : 1
YResolution : 0
ResolutionUnit : inches
OtherImageStart : 3512262
OtherImageLength : 1884562
OtherImage : (Binary data 1884562 bytes, use -b option to extract)
There is a thumbnail and some two JPEGs in there…
Going further, it looks like this full-resolution OtherImage in the NEF is indeed the demosaiced one that gets saved in the DNG (notice the same number of bytes) as:
---- SubIFD ----
SubfileType : Full-resolution image
ImageWidth : 2779
ImageHeight : 1853
BitsPerSample : 8 8 8
Compression : Lossy JPEG
PhotometricInterpretation : Linear Raw
StripOffsets : 200632
SamplesPerPixel : 3
RowsPerStrip : 1853
StripByteCounts : 1884562
PlanarConfiguration : Chunky
LinearizationTable : (Binary data 1374 bytes, use -b option to extract)
BlackLevelRepeatDim : 1 1
BlackLevel : 0 0 0
WhiteLevel : 65535 65535 65535
DefaultScale : 1 1
DefaultCropOrigin : 0 0
DefaultCropSize : 2779 1853
ChromaBlurRadius : 0
AntiAliasStrength : 1
BestQualityScale : 1
ActiveArea : 0 0 1853 2779
It’s saved as a compressed 8-bit JPEG, but the LinearizationTable and WhiteLevel tags give you a hint that the content is then expanded to 16-bits once decompressed.
So the DNG route is probably the best way to go forward with these, although the data underwent a (very?) lossy compression (global tone curve to fit into 8-bits as @gadolf suspected + JPEG), so you can’t ever expect the raw quality, that’s unfortunately gone for good it seems…
For the uber-interested, this is the contents of the LinearizationTable 
What I find very weird, is the first “reduced-resolution” JpegFromRaw in the NEF of 3273332 bytes (3.12 MB), which is larger than the “full-resolution” one! That one seems to be gone in the DNG…
I guess it is the OOC (or Capture NX generated) JPEG? Here it is:
This is most interesting but, realistically, far above my intellectual peak (which I can never operate at anyway!). However, just a few comments:
- The images that I am able to mess around with in LR do not appear to be markedly different in quality to images taken at a later date which darktable can open, suggesting to me, in my relative ignorance, these two sets of images have similar levels of raw information. Certainly the image being managed within in LR is well within the range of raw images, from 3 different manufacturers, that I have been managing.
- The appearance of the jpeg which has bene included here seems to me to be the most faithful representation of the scene, especially in terms of the HSL characteristics of the brickwork, the grass and the copper beech tree in the foreground right. These is not Fuji jpeg but it look better, in my opinion, than that delivered by the dng with - supposedly - some raw data behind it.
As far as the linearization table - I’m pretty sure Nikon has the option of semi-lossy NEFs using nonlinear quantization. (Sony compressed ARWs are similar in that there is a nonlinear quantization curve in addition to the 11+7 strip quantization, the latter of which is notorious for causing some artifacts)
A reference “non-corrupted” NEF from the exact same camera might be useful.
Side track; I’ve been developing my NEFs without applying the linearization table - what am I missing? I can find no prose on the internets to explain it…
Good question - I don’t have Nikon cameras myself, so my knowledge of lossy NEF is “Jim Kasson described this behavior in comments on dpreview.com, and I trust his analysis because he has generally shown that he knows what he is talking about.”. My knowledge of Sony lossy ARW is a bit better because I’ve read the source code of ufraw, dcraw, and a few other RAW converters when trying to track down an issue that plagued me with ufraw long ago (I gave up, although many years later I think I know what the “issue” was, but it’s no longer relevant since I use RawTherapee now)
As I understand it, lossy NEF (where the lossy algorithm is nonlinear quantization) is optional, so maybe you turned that option off? I’m not sure why it would save a linearization table in that case though.
Please add a license to this image.
Nothing, your raw file decoding library applies the linearization automatically, if the NEF is of the lossy compression kind.
@Entropy512 There is background on lossy NEF compression on both Emil Martinec and Bill Claff’s sites.
NEFs with lossy compression still provide raw CFA data, just differently quantized, and that’s it (with lossless Huffman encoding on top). The weird NEFs shared here that underwent Capture NX munging are no longer CFA but demosaiced, have a similar quantization approach, but with a big difference of another very lossy JPEG compression step applied on top (and quite possibly 4:2:0 chroma sub-sampling).
2 Likes
Sorry about this.
The following image is made available under the terms of the CC BY permissive licence
_DSC4013.NEF (9.5 MB)
Thanks. Funny when reading posts in various places about this, I just didn’t connect the dialogue about compression with the linearization table. I perhaps skim too much… 
I wasn’t sure how low-level @ggbutcher had gone - e.g. if he was using a library that automagically applied the table when needed, or doing lower-level file decoding operations. 
True that! I did make an assumption there since I recalled him mentioning using LibRaw at some point on the forum… If you’re doing your own low-level compressed NEF decoding, you absolutely need to linearize the data yourself using the contents of NEFLinearizationTable.
Yep, libraw.
I have never shot compressed NEFs, so I apparently haven’t run into the situation. Now that we’ve discussed it, I’m going have to do it, so I can consider what libraw does/doesn’t do, and then how to handle such in rawproc.
Ha, I’ve just found out that I’ve been shooting lossy compressed for years, because I didn’t fully understand the menu selections. Now, I can only assume this is dealt with in the bowels of libraw, as I haven’t had to deal with it, but I can’t find the code that does this…
Follow libraw_colordata_t.curve[] 
1 Like