Histogram difference - camera vs darktable.

Software darktable
1

Hi,

There is an old discussion Looking for tips on manipulating the histogram about the Histogram.

I am using Canon 70D. Within the settings there is something called “D+”
As per the manual - it gives priority to the highlights. To my understanding less chance of clipping them.

image
image727×227 41.9 KB

What I am observing is what the previous discussion points to.

When I look at the picture on the LCD - the histogram displays similar to this
image

When I look in DT it is like this
image

In essence - significantly pushed to the left.

Questions - when cameras display histogram - is it based on the RAW or on the processed embedded JPG preview?

Is the difference expected?

I remember discussion here about ETTR
https://docs.darktable.org/usermanual/4.0/en/overview/workflow/process/

However - since the type of photos I am taking are extremely frequent changes - travel / vacation - it is impossible to control the scene without a significant slowdown.

Choosing D+ appears to be the easier approach - preserving the highlights more aggressively so clipping is avoided but it came as a surprise to me how dark the shots look in DT.

If I am to bring the DT displayed histogram similar to on camera displayed histogram - I have to adjust significantly
+2.5 EV and ease a bit on the blacks.
image

I feel I am missing the point - how to relate the on camera Histogram to Darktable.

Can you help me please to understand how these 2 are related? On camera LCD Histogram vs Darktable Histogram?

In my desire to avoid clipping highlights - would it be better if I am to abandon D+ and perhaps change to 1/3 or even 2/3 underexposed pictures on purpose?

2

Always JPEG unless you’ve flashed an alternate camera firmware like Magic Lantern.

You’ll likely need to experiment (or hear from another canon user) whether D+ is useful for a raw ETTR approach.

I have Nikon and Fuji. With the Nikon, setting the in camera jpeg to a flat profile results in the best raw exposure. I generally have 2/3 stop head room when ETTR’ing.

On the Fuji, they have something call “natural live view” which on its face sounds like it’ll be helpful as it turns off any in camera film simulations… But it turns out that the histogram is then base on what is shown on the LCD screen, which means I clipped almost every shot by about 1/2 to 1+ stops. Absolutely useless. What ended up working best for me was finding the flatest film simulation (pro std neg) and setting the jpeg profile even flatter.

Sorry that doesn’t directly answer, but hopefully it is somehow helpful.

3

I am on the manufacturer’s firmware…
I intended to include it but looks like I missed it.
The profile is “Neutral” and it is “0, 0, 0, 0” basically - it should not apply contrast / brightness etc.

I think I will take your advice on 2/3 underexposure. DT handles well the denoising so the images are not lost at all but they can be better…

Thank you…

4

This guy goes through the process of actually working out the DNR of his camera… interesting read for some if you have not ever come across it…

1 Like
5

I am probably not going to be able to answer your question very well and defer to more knowledgeable people on the forum. I was surprised to discover a few days ago on this forum that the histogram is actually based on the JPEG rather than the RAW data in the camera. But I approach the problem of working with the camera’s histogram by looking for clipping on either side. If all the data is within the histogram bounds then I believe I have captured all the data and can do what I want with it in processing with DT or any program. Some scenes contain a high dynamic range of lighting and either I choose to accept some clipping or I do multiple exposures to cover the range and combine them later (separate topic how to best do that). However, if the data can all fit in the histogram I prefer to skew towards the right to minimise noise. This is very important when I choose to shot with a compact camera while travelling.

For many years I really focused on the histogram in processing images in programs like Lightroom and GIMP. I still pay a lot of attention to the histograms in these programs. However, now that I predominately edit in DT I pay little or no attention to the histogram. On one video about DT the presenter referred to using the histogram as painting by numbers where photography is about the look of the image. So now I have placed the histogram on the left of the screen where I can ignore it. Also, I don’t often use the clipping indicators while processing. I just pay attention to the look of the scene and trust filmic to keep all the pixels within range. However, I do look at the RAW clipping indicator to decide if highlight reconstruction is needed and if all channels or just some are clipped. I hate reconstructing highlights so exposing in the camera to avoid that is a great idea. I believe this is where highlight tone priority may prove useful if it applies to the raw exposure and not just the JPG (someone else might know the answer to this).

I am sure you will get some helpful replies on this great forum. Good luck.

6

This is what I did my best to achieve. The issue is that while the LCD Histogram on the camera claims everything is within range - when actually in DT it is significantly pushed to the left.

My camera is using a crop sensor (and it is also not in its prime any more). When shadows are boosted there is more noise within them so there is more work that needs to be done to bring them up.

Since I don’t do any manipulation to the embedded jpg - it is flat - as far as I can say - I was hoping it would have a reasonable representation to the raw data.

It would be interesting to hear from some canon shooters with similar models if they utilize D+ at all or they opt for something like -2/3 EV during shooting.

I just did a test an hour ago with D+, vs -1/3 vs -2/3. It looks to me that the manual correction is going to do a better job in preserving the shadows but I didn’t have a very bright scene. Often my pictures are under exactly such conditions - bright daylight. They are not art - just family photos. But still I’d like them to be as good as it can be.

7

I look forward to following this post to hear about the experiences of others. There are some very pro-ETTR people and very anti-ETTR views amongst the posters to this forum. That may because difference cameras respond in different ways. But, I teach my students that exposing to the darker side is easier to fix, but at the cost of noise. The cardinal sin for me is clipping highlights which are nothing less than a pain in the arse to fix and usually produce disappointing results despite the best efforts. My internet is so slow this morning that your file still has another 7 minutes of download time before I can get a look at it. No now six minutes.

8

The mystery deepens. I just opened your image in my version DT which is the latest weekly build for windows and the histogram is fine. I activated the raw clipping indicator and there is no clipping. I would be happy to work with this exposure in DT.

I also opened your image in Lightroom and the histogram is skewed to the right. See lower image posted here for LR histogram.

image

image

9

Btw, I use Magic Lantern for my 6D and 7D. It is not a firmware but an add-on booting from the memory card. Way easier with Magic Lantern histogram than the one from Canon.

Installation tutorial for 70D, but read the whole thread about 70D first before you do anything Magic Lantern Nightly Builds

10

Log vs Linear display… and it wouldn’t be rec2020 assuming that is your histogram profile…

DT

image

Also likely reflects a gamma…so adding unbreak in DT would get you closer…

I think at least…

1 Like
11

I had forgotten about switching between log or linear display in DT. I do it all the time in GIMP, but in DT I really ignore the histogram most of the time. Still, I am not sure why the original posters histogram is so different to mine.

12

I thought that image came from a different post ??

13

@vbs would need to confirm if the original posted histogram is from the supplied RAW file. I presumed it was and do not understand why it looks so different to when I open it in DT.

14

When I look at the picture on the LCD - the histogram displays similar to this … [centred histogram] … When I look in DT it is like this … [histogram skewed to the left]

I expect the camera is showing the values that encode in (non-linear) sRGB space, where darktable is showing values in the working colorspace, which is probably linear.

Remember that 50% in sRGB is approximately 18% in linear, so values in a linear space will be skewed to the left, compared to sRGB.

Don’t confuse this with software that has a log or linear vertical scale to histograms.

15

I don’t have a Canon - so I’m guessing here - but on my Sony, using ‘DRO’ (dynamic range optimizer I think) has the effect of lowering the exposure, then the camera pushes the shadows and midtones back up.
So the jpg looks like a normal exposure, but with more highlight detail. But the RAW in dt is darker, as dt obviously doesn’t recognize Sony’s exposure tinkering.
Maybe your Canon’s D+ is doing something similar?
I don’t know much about the histogram in dt…

16

Hi

My understanding of natural live view is that it provides a optical viewfinder like view (DSLR if you will) in the electronic viewfinder. Quite useful if you are working in very low light, otherwise not (in my view, YMMV). Film simulations are not turned off and any other tweaks (higlight tone, shadow tone etc) are as set by you in settings.

The nearest thing you can get to a raw histogram (and raw blinkies) is to set up a custom profile as below:

Settings > IQ settings > edit save custom settings
DR100
D range priority off
Film sim Pro neg std ()this is a very flat simulation)
Grain OFF
Highlight -2
Shadow -2
Colour 0
Sharpness whatever
NR whatever

Hope this helps.

17

Not sure where you got that idea from, but Fuji’s own literature suggests differently:

Choose whether the effects of film simulation, white balance, and other settings are visible in the monitor.

OFF: The effects of film simulation, white balance, and other settings can be previewed in the monitor.

And

Turning Natural Live View ON disables the visual effects of in-camera JPG processing. This includes film simulation, white balance, shadow/highlight adjustments.

18

Yes, in the viewfinder. Not what’s recorded in the file - that keeps film sim, and other adjustments. Apols for not making that clearer. From the X-H1 manual in front of me.

image
image1120×744 98.2 KB

I can’t see how (in effect) artificially boosting shadows and back lit subjects is going to help you get a more accurate result, since the raw file will not have those areas artificially boosted.

Another Fujifilm oddity you may not have noticed. With natural live view off (which is where it should be almost all the time) the only time you have true what-you-see-is-what-you-get (in terms of exposure) in the EVF is if you are in full manual mode. If in full auto or aperture or shutter priority the EVF shows you what the camera thinks you probably want.

19

This… A thousand times this.

Almost always the ‘embedded JPEG’. i.e., what your camera would call ‘the final picture’. Like others have said, maybe with alternative firmwares, but I don’t know of a camera that has a RAW histogram out of the box.

My easy - and harsh - answer is: ‘you cannot relate’. And maybe you shouldn’t even try.

Things like that ‘D+’ setting you talk about. Some cameras have modes like this that actually alter the exposure and compensate in the ‘developing the picture’ step (so they actually shoot darker than you think, but compensate the final JPEG file. This way you preserve highlight data, also in your RAW file). Others just adjust the curve they use to make the JPEG (so the tone curve they apply will leave more detail in the highlights, but the shot itself is not technically different, so the RAW file should be the same, which means you still need to be careful not to clip your highlights).

See it that your camera has a ‘built in RAW converter’, and the histogram you are seeing is for what the camera is doing ‘in the edit’. Darktable is another RAW editor, so will do different things to your captured light-data, so will display a different histogram.

I always walk around at -1EV or even more by default on sunny days. But I know that I’m more allergic to clipped highlights than to noise, and my cameras doesn’t add a lot of noise in the shadows.

Your D70 is one of the older Canon’s that’s notorious for noise added to the shadows, even at low ISO’s. So what you might want to do depends on how allergic you are to noise or to clipped highlights. That’s something only you can decide.

But that D+ mode is a typical ‘make shot too dark, compensate for it in the edit’ from what it seems like. So it’s only natural that you have to do the compensating yourself when you open the RAW in Darktable (because the camera only did the compensating in JPEG, not in RAW so to speak).

I would happily use something like that, but as I said, I don’t mind a bit of noise, or I’m not afraid of it. I can’t decide if that’s true for you, or how big the noise problem is on your camera, etc…

Try it out, is all I can say. Experiment a bit.

20

The image you chose at least as far as I can see is from here ??

Not this thread but I may be missing something in a historical reference???