AFAIK (older?) Nikon cameras had black levels already subtracted in the NEF, and they also use “white balance pre-conditioning”, i.e. the channels are already scaled somewhat. I think both factors play a role here. However, I don’t really know the details, sorry – if @ilias_giarimis is still around here he should know better.
Edit: possibly also @JackH?
1 Like
@stefan.chirila Thanks for the samples! Some initial findings. I plotted the cumulative fraction of pixels with a certain raw value (without distinction between RGB pixels) for the different ISO’s. Since you provided two series without LENR, I simply averaged them together for slightly better statistics.
This is what you get:
The dashed vertical line marks a black level of 512. The dotted lines are a few values below and above. The dashed horizontal line marks the 0.5 mark. Ideally, and assuming symmetric noise, all curves should have their inflection point at the intersection of both dashed lines.
For all ISO’s this is pretty much the case, at least somewhere in between 511 and 512. The exceptions are ISO 200 and 400, where the inflection point is more between 510 and 511. Not sure why, but I don’t think setting the black level to either 512 or 511 will make a big difference for regular images.
We can also look at the separate RGB channels to see if something is going on there.
I would say that there is also nothing out of the ordinary here. It’s harder to see the actual inflection points here, but at least all intersections with y = 0.5 lie between, either 510-511 or 511-512. You might say that sometimes not all channels behave equally (like for ISO 400 R is a little lower, for ISO 800 B is a little lower and for ISO 1600 G is a little higher), but I think these are all pretty negligible considering this graph is heavily zoomed in.
I haven’t looked at LENR, but at least this little investigation doesn’t seem to indicate anything really wrong with the current black level for all channels on 512.
1 Like
My Nikon D7000 requires no black subtraction. Nikon Z6 requires subtraction of 1008.
If one had the time and inclination, they could download NEFs from DPReview’s comparison collection and grep exiftool output for Black tags, and probably find a Rubicon…
So it seems, also according to, @Thanatomanic that there is, at least from a technical standpoint, nothing wrong with the files? Or did I misunderstand.
I guess at the end of the day, the fact that magenta seems to find its way into shadows, (like the dirt in this image) as well as even blues that are close to mids, sort of bothers me. Also why are the greens so incredibly saturated? I get that perhaps some people would enjoy overly amplified greens, that also tend to lean towards blue on their own; however I’d consider doing that in an edit, not straight out of camera. In my understanding Raws aren’t meant to be this subjective, they are meant to show what the sensor sees 
@stefan.chirila If the problem is not with the absolute black level, it could still be something else. Maybe there is something going wrong in RawTherapee specifically, because the image @Matt_Maguire produced from darktable seems quite all right to me.
I’m not sure how else we could quantify your magenta cast. Shoot a color target at different exposures and measure deltaE vaues?
2 Likes
As @agriggio mentioned the D700 does dynamic black subtraction before writing data to the raw file, so from then on its BlackLevel is zero and moving the slider to the left would have not effect.
On the other hand on the Sony moving the slider to the left would add an offset to the raw data, effectively reducing the contrast ratio/DR.
1 Like
I would tend to agree that it is a white balance issue on a severely underexposed image I assume combined with a ‘filmic’ curve. The in-camera white balance renders it with an unusually low tint of 0.766 in RT.
Try this: Neutral profile, +4EC, in-camera white balance, tint = 1. The path looks neutral enough to me. For my taste I would lower the color temperature a bit. If you feel that the path was even more neutral than that, white balance off of it with the relative color picker.
I am wondering whether Sony Raw files have the DCP profile for the given setting the camera is on saved/embedded into them.
Here I downloaded someone’s set of dcps for the sony a7ii and the “faithful” setting seems to give me the least amount of grief with greens; making them look closer to what my Nikon would have outputted.
RT ships a dcp file for D700 which was made by @Morgan_Hardwood, me and a bottle of wine using my D700. Maybe that’s the cause the D700 colours look better 
1 Like
Maybe there is some discoloration in the shadows because the high iso.
My edit without touching white balance
S7_07667.ARW.pp3 (13.5 KB)
1 Like
it very well may be why 
1 Like
You know, RawTherapee actually ships with a custom DCP for the a7ii.
@Morgan_Hardwood, the commit on March 9 is well documented, but do you remember what changed for the commit on April 14? It’s two years ago though…
2 Likes
So if I understand well the RawTherapee team put together a custom profile for this camera in order to avoid the magenta shift …which means the magenta shift was even worse then than now 
? wow
That or perhaps my version of RT doesn’t use it? Any way to check?
I am using RawTherapee AppImage 5.8-2437-gea485e4
The way to check is to toggle between ‘Camera standard’ (which uses the dcraw matrix) and ‘Auto-matched camera profile’ (which uses the DCP). It will also enable the options in the DCP block. Only have ‘Base table’ checked (but YMMV with Tone curve and Look table as well).
Edit: the profile is in RT for two years already, yours should support it.
1 Like
i guess it uses it; there is a slight change in lightness but nothing radical in colour.
Wouldn’t it be more correct to use Custom instead of Auto-matched to check this? This would exclude the possibility, that, for whatever reason (bug) RT picks the wrong dcp accidentially.
1 Like
Here’s the rawproc screenshot, with the metadata-suppled black subtract, libraw-supplied dcraw primaries, and a filmic curve (in the parameters pane):
The magenta cast is about not doing the black subtract, or doing it with an incorrect number; that is just part of the raw processing to compensate for dark current in the sensor that keeps black from being 0.
The dcraw primaries yield an acceptable (well, to me, anyway… 
) color rendition.
It takes a steep filmic curve to pull the image out of the shadows , although it’s worth noting that the bits of sky in the top-right are blown.
So, I don’t think there’s anything extraordinary to regular camera default processing needed for this image. S’why I like rawproc; all that is done to the image from the very start is displayed in unambiguous and explicit order in the tool chain… however, do not download it, it’s too hard to use. 
1 Like
In my very quick edit, the greens appear heavily saturated because I probably didn’t push the exposure enough, and the tone curve is causing the underexposed image to appear heavily saturated (local contrast probably adds in a bit of saturation too) Here is the same image I edited before, but pushing the exposure compensation up to +4EV.
Just as a point of comparison, here is the image pushed to the same exposure, with the camera-suggested white balance:
Just FYI, here is “what the sensor sees”, taking the raw tristimulus values and dropping them straight into an sRGB colorspace, no white balance or tone curve, just demosaicing and scaling:
So a raw editor is always going to be applying some sort of “editing”, at a minimum to map the sensor values in a sensible way into a standard colour space, adjust the balance between the channels, and apply some sort of tone mapping. Particularly the tone mapping will affect the look of the image, and different editors take different approaches here. In darktable, I used filmic for tone mapping, but I could have used base curve for a different result, and RT even uses some sort of histogram equalistion process to make the raw image look more like the OOC JPEG that the manufacturer creatively processes inside the camera. So, Raws are maybe not quite as objective as one might first suppose.
2 Likes
I’m sorry I guess I was being a bit overly dramatic there
Thanks for the feedback though! much appreciated!
@heckflosse do you guys use an open source program to modify the dcps? Would I be able to modify the one that is there for the Sony a7ii and make one with more tame greens ?
…or, perhaps, can I dare request one being added?