The OP wanted something (in darktable) to see how much ‘histogram the data is using’. The easy answer is already given : use another tool that displays a raw histogram.
While shooting film negatives I wanted to know how much I can raise the exposure without clipping the filmbase. I used dcraw then, set to no white balance adjustment and ‘raw’ color profile to export to a linear tif. Open that in photoshop and look at the channels (or the histogram for each channel).
AFAIK this is the closest you can get to ‘unaltered data with only mosaicing’.
This is also how I figured out different exposure settings for each of the channels, to get as much DR for each channel from a film scan.
… After all that I just use my dedicated scanner again
Noise comes from a) photon/shot noise [random], and b) read noise (camera electronics) [patterned]. It doesn’t come from ISO. ISO is just an amplifier - it amplifies whatever is there. If whatever is there is a lot of noise, that’s what gets amplified. This is what happens in low light situations. Because there is low light, people boost ISO. Then they see images with a lot of noise and think ISO is to blame. But the real problem was not capturing enough light.
Raising iso (often) raises the gain on the amplifier, boosting the signal but also boosting the noise that was already there.
My point still stands that slower shutter speed (while keeping the same exposure value) also reduce noise, the shot noise (or ‘measurement accuracy’) and if an otherwise well lit scene at low iso values gives weird red specks I’m thinking that’s the cause… Or something weird in the demosaicing phase… Or a really noisy sensor.
Yes, but not necessarily proportionally for both signal and noise as you think. For some sensors/cameras there is an advantage in shadow SNR when shooting at higher ISO, as @Matt_Maguire pointed out. Again, Emil Martinec provides an excellent explanation of ETTR misconceptions and ISO use here.
For the interested, you can check you camera’s behaviour at Bill Claff’s indispensable site, I’m linking and example for the A7R4 mentioned, indeed showing a step at ISO 320 giving you an extra 1/2 stop in the shadows if your highlights are not a priority.
I’m interested in running some tests of my own based on this discussion. Can someone tell me how I can review the raw histogram in Darktable… is it just a matter of selecting the original image in the history, or is there more to it than that? If it’s not possible, then what’s the best way outside of DT?
You cannot. The histogram always shows the output of the pipe; even if you go back the the original image (at the bottom of the stack), white balance, demosaicing, input colour profile and output colour profile will be applied.
If you really want to get a feel for how much headroom you have with your jpeg histogram, start bracketing your exposures in camera. It isn’t simple math, such as “always add 1.75 stops,” but if you start bracketing you’ll start to get a feel for how much you need to add.
My rawproc software shows the histogram of the tool selected for display. But, you can select any tool in the chain for display, including the opened raw file before any processing.
Need to keep in mind though, the rawproc internal data is floating point in the range 0.0-1.0, but the ingested raw data is converted in proportion to its container from the file, unsigned 16-bit integer. So, the max value of a 14-bit raw image, 16383, is about 0.25 in internal rawproc. Later, the user has to insert a tool to scale the data to fill the 0.0-1.0 range; either exposure compensation or blackwhitepoint will do the trick.
Really, RawDigger is probably the best tool for the job. You pay for it, but that compensation supports the open-source libraw project, which is probably the most comprehensive and current open source raw conversion library out there.
Thanks, that’s exactly what I’ve done, but what I needed was a way to assess the point at which I’ve clipped in raw. It sounds like Rawtherapee will allow me the means to make that rough estimate.
I just ran through my series, and the DT raw indicator tracks fairly closely with RT, a little more aggressively because of its clipping threshold.
Interestingly, the threshold of clipping that I observed in RT is a nearly exact match with the highlight indicator on my Canon LCD display, which would imply no additional headroom. Of course that’s a data set of one, but does that makes sense to you?
After some tests, I have found that my Panasonic GX80 is very conservative at finding “overexposed” areas. It’s generally safe to have +1EV every time and treat this setting as base, like it was 0EV. And default “factory” zebra is set at only 80%.
If most of the scene is well lit - fits in Addams’ 6th or 7th zone, like penguins on snow, I can safely add another EV to my “base” +1EV setting. Still no clipped areas and no need to reconstruct. “RAW exposed indicator” mentioned by @paperdigits, when set to 1.0, is everything you need to check if your image is overexposed on “data level”. And to determine “safe margin” for your camera.
For my Panasonic, +2/3EV (no matter what metering mode) is absolutely safe until I try to shoot sunrise or something that contrasty. For colleague’s Canon 7D it’s about +1/3.
Have you ever compared it to Faithful as noted here with contrast and saturation reduced? As mentioned above in this thread by @Phil_Smith, both Neutral and Faithful are recommended for this purpose of making the JPG more closely match the RAW so I wonder which one seems more accurate to you? At least in my limited testing so far, Faithful with 0, -4, -4, 0 seems to still show more clipping on the Canon LCD than is actually present in the RAW so I’ll try Neutral next and see how that compares.
Neutral and Faithful were pretty close to one another, and quite honestly, there was only a slight difference with Standard. I’ll try with the reduced settings and see how it plays out. But it seems I’m already close to a match already
I can show people what I’ve come up with if there’s interest
I’ve been reading more online and it sounds like the 50D is pretty well-known for having banding issues in shadows even at low ISO and the solution is to apply chroma noise reduction. Following the manual for denoise (profiled) I took the image of the column above and tried to correct just the chroma noise using wavelets and HSV color:
Original:
One instance of denoise (profiled) removing chroma noise:
I know that you can now do both chroma and luminance noise reduction in a single instance of the module, but I applied two here so I could make the chroma noise reduction quite a bit stronger. This cleans it up a lot.
It’s really helpful to me to understand that even with a proper exposure and low ISO, it’s still possible to see this type of noise in shadows with this sensor (and that this type of chroma noise reduction can remove a lot of it).
As a matter of curiosity: what if you no not use denoise (profiled) at all
— and invoke two instances of the contrast equalizer instead
(first instance = preset denoise, second instance = preset denoise chroma)?
