Thanks for this; better than my attempts in 3.2.1 because you clearly have more ‘darktable-hours’ than I. Yeah, I agree about the sky - it looks very much like a sub-compact point-and-shoot jpeg! I’m going to examine your history stack in detail.
Yes, your suggestion on exporting a tiff from a PS s what I had decide to do, not being all that impressed with what I could get from the dng. And, yes, I did use the latest version of the dng converter .
More or less. These guys don’t stop working and improving the software so I’m always pushed back to the beginners class
Just make sure you can get as neutral as possible, don’t apply any effect in ACR and PS, and try to undo effects applied by default. That way, you will get a tiff as close to the raw as possible. And don’t forget about the tiff export settings regarding bit depth and color space (not sure if there are other settings you should also care about…)
Not necessarily, these are just part of the embedded ICC profile.
@LateJunction
Who knows what that old Capture NX software did to those NEFs, it doesn’t look like it actually contains any raw data at at all! Here’s the excerpt showing all image IFDs from exiftool -a -u -s -g1 and none of them is raw like in a normal NEF file:
Going further, it looks like this full-resolution OtherImage in the NEF is indeed the demosaiced one that gets saved in the DNG (notice the same number of bytes) as:
It’s saved as a compressed 8-bit JPEG, but the LinearizationTable and WhiteLevel tags give you a hint that the content is then expanded to 16-bits once decompressed.
So the DNG route is probably the best way to go forward with these, although the data underwent a (very?) lossy compression (global tone curve to fit into 8-bits as @gadolf suspected + JPEG), so you can’t ever expect the raw quality, that’s unfortunately gone for good it seems…
For the uber-interested, this is the contents of the LinearizationTable
What I find very weird, is the first “reduced-resolution” JpegFromRaw in the NEF of 3273332 bytes (3.12 MB), which is larger than the “full-resolution” one! That one seems to be gone in the DNG…
I guess it is the OOC (or Capture NX generated) JPEG? Here it is:
This is most interesting but, realistically, far above my intellectual peak (which I can never operate at anyway!). However, just a few comments:
The images that I am able to mess around with in LR do not appear to be markedly different in quality to images taken at a later date which darktable can open, suggesting to me, in my relative ignorance, these two sets of images have similar levels of raw information. Certainly the image being managed within in LR is well within the range of raw images, from 3 different manufacturers, that I have been managing.
The appearance of the jpeg which has bene included here seems to me to be the most faithful representation of the scene, especially in terms of the HSL characteristics of the brickwork, the grass and the copper beech tree in the foreground right. These is not Fuji jpeg but it look better, in my opinion, than that delivered by the dng with - supposedly - some raw data behind it.
As far as the linearization table - I’m pretty sure Nikon has the option of semi-lossy NEFs using nonlinear quantization. (Sony compressed ARWs are similar in that there is a nonlinear quantization curve in addition to the 11+7 strip quantization, the latter of which is notorious for causing some artifacts)
A reference “non-corrupted” NEF from the exact same camera might be useful.
Side track; I’ve been developing my NEFs without applying the linearization table - what am I missing? I can find no prose on the internets to explain it…
Good question - I don’t have Nikon cameras myself, so my knowledge of lossy NEF is “Jim Kasson described this behavior in comments on dpreview.com, and I trust his analysis because he has generally shown that he knows what he is talking about.”. My knowledge of Sony lossy ARW is a bit better because I’ve read the source code of ufraw, dcraw, and a few other RAW converters when trying to track down an issue that plagued me with ufraw long ago (I gave up, although many years later I think I know what the “issue” was, but it’s no longer relevant since I use RawTherapee now)
As I understand it, lossy NEF (where the lossy algorithm is nonlinear quantization) is optional, so maybe you turned that option off? I’m not sure why it would save a linearization table in that case though.
NEFs with lossy compression still provide raw CFA data, just differently quantized, and that’s it (with lossless Huffman encoding on top). The weird NEFs shared here that underwent Capture NX munging are no longer CFA but demosaiced, have a similar quantization approach, but with a big difference of another very lossy JPEG compression step applied on top (and quite possibly 4:2:0 chroma sub-sampling).
Thanks. Funny when reading posts in various places about this, I just didn’t connect the dialogue about compression with the linearization table. I perhaps skim too much…
I wasn’t sure how low-level @ggbutcher had gone - e.g. if he was using a library that automagically applied the table when needed, or doing lower-level file decoding operations.
True that! I did make an assumption there since I recalled him mentioning using LibRaw at some point on the forum… If you’re doing your own low-level compressed NEF decoding, you absolutely need to linearize the data yourself using the contents of NEFLinearizationTable.
I have never shot compressed NEFs, so I apparently haven’t run into the situation. Now that we’ve discussed it, I’m going have to do it, so I can consider what libraw does/doesn’t do, and then how to handle such in rawproc.
Ha, I’ve just found out that I’ve been shooting lossy compressed for years, because I didn’t fully understand the menu selections. Now, I can only assume this is dealt with in the bowels of libraw, as I haven’t had to deal with it, but I can’t find the code that does this…