out of the box RAW rendering not up to JPEGs

[anarcat]

So I mentioned this problem briefly in another thread, but I’ve often have trouble working on some pictures in Darktable, especially when compared to the original JPEG generated by the camera. Here’s an example.

This is the JPEG as rendered by my Fujifilm X-T2:

And this is the same picture, as rendered from the RAF file by Darktable, after only minimal modification (a null crop, basically):

Notice how the bridge is completely different: the colors are just completely off. I’m not even sure how to describe that problem, but to my eyes, it’s purple-ish in the original JPEG while in the rendered one, it’s deep blue. Also notice how the deep blue color matches the blue on top of the building to the left, and that blue is consistent with the original JPEG.

More generally, I find that the default Darktable rendering is more “flat” than the original JPEG from the camera: the images lack contrast, colors are weird and everything has this “tame” look. I had a similar feeling when working on RAW files from my Canon Powershot G12, so maybe that’s just expected. I know that RAW files are different, out of the box of course. But I feel that the second I open that darkroom tab, I have already an uphill battle to reach some basic levels to get back at the original JPEG to actually start doing what I need. My experience with other RAW rendering software (can I mention Lightroom here without getting beaten up? :)) is quite different there…

So this all means I often work directly on the JPEG instead of the RAW files, which Darktable is obviously not really tailored for. (There I particularly have trouble with the fact that photo stars are not sync’d between the RAW and JPEG versions, and that only the RAW is shown when “grouping” pictures. So to work on JPEGs, I need to “ungroup” pictures and skip the RAW files… )

I’d understand if the response was “Darktable isn’t for you, clueless newbie”: maybe I just don’t understand basecurves or RAW rendering enough to be productive with Darktable… But I want to learn and I’d really like to understand what’s going on when that kind of stuff happens (which is not always!).

Here’s the original RAF file and sidecar files:

DSCF1110.RAF (22.3 MB)
DSCF1110.RAF.xmp (2.8 KB)
DSCF1110.JPG.xmp (1.3 KB)

[paperdigits (Mica)]

@anarcat

More generally, I find that the default Darktable rendering is more “flat” than the original JPEG from the camera

This will be true of any raw processor (that doens’t auto apply a bunch of stuff when you open your file). In general, your raw has less contrast than the jpg, as the raw files is generally trying to preserve shadow detail and then let the rest of the scene fall as it may.

The jpg is processed by your camera decision making.

So with minimal editing (only a crop as you say) I don’t think you’ll realize the full potential of the raw.

I’d at least do the following in DT:

1. white balance (looks different from your raw and jpg)
2. pick a good base curve
3. use the tone curve to control global contrast and saturation
4. use the equalizer (luma) for a pop of local contrast

[LebedevRI (Roman Lebedev)]

I can only confirm your findings, the dt defaults are just some defaults to produce viewable picture.
Other software (especially manufacurer’s one, camera jpeg’s) is much more edited by default.
You are supposed to go rest of the way by yourself (i.e. edit to your liking)

See also end of colour manipulation with the colour checker lut module | darktable

[paperdigits (Mica)]

Here is a quick try:

DSCF1110.RAF.xmp (22.7 KB)

[anarcat]

@paperdigits

1. white balance (looks different from your raw and jpg)
2. pick a good base curve
3. use the tone curve to control global contrast and saturation
4. use the equalizer (luma) for a pop of local contrast

So see this is where i get off the rails. What’s a “good base curve”? Darktable automatically picks the “Fujifilm” base curve - am I supposed to pick a different one? Or just start editing the curve directly? Why isn’t the “Fujifilm” base curve from darktable a good default? Same with the white balance, I guess… I tried messing around with that, but that changes the darks and browns, the bridge stays bright, deep blue…

The tone curve is another tool I have a hard time using: it’s way too hard to “clip” out of range and get obvious aberrations. But I get it: I can fix the contrast and get some of the “flatness” of the RAW file fixed there.

In summary, apart from the white balance, I think you’ve outlined three tools I seem to be incapable of using. I guess I need to read up on those and watch a bunch of youtube videos to catchup!

[anarcat]

@paperdigits

Here is a quick try:

See that’s nice, and it fixes a bunch of the “RAW” feeling that I was desribing. However, it doesn’t fix the colors on the bridge at all, which still seem completely off.

Look: I know that the camera does a bunch of processing and that RAW files are just that, raw material. I get that. I’m fine with messing around with contrast, saturation and so on, I do it all the time with most pics in DT. But it seems there’s something just completely off with the colors on that bridge! That can’t just be some contrast issue, can it?

[paperdigits (Mica)]

@anarcat

So see this is where i get off the rails. What’s a “good base curve”? Darktable automatically picks the “Fujifilm” base curve - am I supposed to pick a different one? Or just start editing the curve directly? Why isn’t the “Fujifilm” base curve from darktable a good default?

Sorry, I didn’t mean to cause further frustration. Picking the “good” base curve is entirely subjective. I think because you have a night shot, that your base curve might differ a bit from a regular day time shot. The fuji base curve is a fine starting point and renders nicely in most situations. For this particular photos, you might look at the Leica base curve, as it renders the scene a bit darker, which allows your more room to play with the highlights.

You can also just edit the base curve yourself… again, there is no “right” answer, just what you think looks good.

@anarcat

The tone curve is another tool I have a hard time using: it’s way too hard to “clip” out of range and get obvious aberrations. But I get it: I can fix the contrast and get some of the “flatness” of the RAW file fixed there.

Yes, the tone curve operates in LAB color space, which means a small adjustment goes a long way!

@anarcat

In summary, apart from the white balance, I think you’ve outlined three tools I seem to be incapable of using. I guess I need to read up on those and watch a bunch of youtube videos to catchup!

You’re not incapable, you’re just not a pro yet Time playing around with images will fix this! Rome wasn’t built in a day, and all that

[LebedevRI (Roman Lebedev)]

@anarcat

But it seems there’s something just completely off with the colors on that bridge! That can’t just be some contrast issue, can it?

Seeing how it is bright blue in the dark, i’m going to guess it’s again saturated blues problem.
In input color profile module, set gamut clippling (! NOT the color profile itself) to rec2020.

[anarcat]

@LebedevRI

Seeing how it is bright blue in the dark, i’m going to guess it’s again saturated blues problem.

In input color profile module, set gamut clippling (! NOT the color profile itself) to rec2020.

That doesn’t seem to do anything. But you may be onto something here, because changing the input color profile itself (yes, I know what you said ) does change the colors on the bridge (even if gamut clipping doesn’t).

[houz]

I can confirm that the blue bridge goes all squirrely with the default matrix. With Rec2020 as input profile it looks way better.

[Carmelo_DrRaw]

Here is what one can achieve by using the “Fujifilm X-T2 Adobe Standard.dcp” profile (available from CameraRaw), plus a slight contrast curve:

I had to use another image editor as DT does not support DCP profiles… anyway, one can see that in this case the DCP does really a good job!

I wonder if the magenta cast on the bridge in the camera Jpeg corresponds to reality, or if the DCP result is more realistic.

[anarcat]

@houz

I can confirm that the blue bridge goes all squirrely with the default matrix. With Rec2020 as input profile it looks way better.

Here’s what it looks like with linear Rec2020 RGB input profile here, for what it’s worth:

It does look slightly better, but still completely blue and not magenta. I’m now left to wonder how the camera figured out this was magenta in the first place. That bridge is a (huge!) multimedia installation and I know some of those people that made it (the installation, not the bridge of course), so technically I could try to figure out what color it was supposed to be if that’s useful…

[anarcat]

@paperdigits

Sorry, I didn’t mean to cause further frustration.

And just for the record: absolutely no harm done here. I am positively amazed at the response I’m getting here. Incredible support, really positive and constructive comments. I am sorry if my comments come out as frustration… I think I’ve been using Darktable for about 5 years now (according to my storage anyways), so I’ve had that issue bug me for a while… I’ll try to be a little more positive next time.

[garrett (Garrett LeSage)]

Here’s the darktable X-Trans III Velvia preset + a rec709 input profile workaround in action:

DSCF1110_01.RAF.xmp (8.6 KB)

It’s not exact, but pretty close. Switching to rec709 muted the colors a bit (not surprising, as it changed the input color space), so I bumped them back up with the “velvia” module to compensate. Between the input profile (rec709 setting) and velvia modules, the colors shifted a tiny bit, so I nudged the hue of the bridge until it looked similar your JPEG.

This is a tricky scene: It’s dark overall and the bright lights of the bridge are way out of gamut. I’ve got to say that Fuji cameras are quite good at making JPEGs in most situations. (One exception: Waxy skin when ISO is cranked way up. It doesn’t affect raws thankfully.) Still, you’ll want to keep shooting raw. You could, for example, work on top of the raw of this scene to change white balance, recover a bit more detail in the darker parts, and so on. The JPEG would start to fall apart if you wanted to do that.

But I think this proves that you can have your RAWs and your Fuji colors too. Even with my short amount of time with darktable, I can say that it’s usually much closer to where you want it under more optimal settings.

[anarcat]

@garrett

It’s not exact, but pretty close. Switching to rec709 muted the colors a bit (not surprising, as it changed the input color space), so I bumped them back up with the “velvia” module to compensate. Between the input profile (rec709 setting) and velvia modules, the colors shifted a tiny bit, so I nudged the hue of the bridge until it looked similar your JPEG.

That is quite impressive, I got to say. It’s almost exactly what I expected… Your mention of “velvia” reminds me that I did choose a specific “landscape” film emulation (called “Velvia” too! coincidence?) on the camera for that shot.

I see you used the “color look up table” early on in the history stack: what does that do? It seems to be the key change the fixes the color on the bridge to restore the original JPEG… or maybe the history is not ordered the same way here?

@garrett

This is a tricky scene: It’s dark overall and the bright lights of the bridge are way out of gamut. I’ve got to say that Fuji cameras are quite good at making JPEGs in most situations. (One exception: Waxy skin when ISO is cranked way up. It doesn’t affect raws thankfully.) Still, you’ll want to keep shooting raw. You could, for example, work on top of the raw of this scene to change white balance, recover a bit more detail in the darker parts, and so on. The JPEG would start to fall apart if you wanted to do that.

Yeah. I think most of the problems I had in that space were night shots with odd colors like this. Day shots are way more faithful to the original scene. And for sure, RAWs do allow much more flexibility, but the amount of work required just to get to a basic “normal” scene seems a little prohibitive in those cases…

@garrett

But I think this proves that you can have your RAWs and your Fuji colors too. Even with my short amount of time with darktable, I can say that it’s usually much closer to where you want it under more optimal settings.

Definitely. I’ve had great experience recovering lost dark zones or compensating for exposure with DT, don’t get me wrong. But this makes me think I should definitely look at the styles uploaded in that thread:

dt styles for arangast's "New Fujifilm XTrans film simulations"

@arangast posted really nice HaldCLUTs for Fujifilm film simulations. I created darktable styles based on https://www.youtube.com/watch?v=m_cLCL5PJk4 so that these LUTs can also be used in darktable. Hope they work fine for you Fuji_XTrans_III_dtstyles.zip (31.4 KB)

Maybe they could give me a better basis to work from without too much work?

And yeah, I obviously have a lot more to learn. But thankfully, there’s an awesome community here (and elsewhere!): that helps tremendously.

It would be great if the rendering was better out of the box. I still can’t quite fathom why the colors were so out of whack on that bridge: it’s not just that it doesn’t match the original JPEG’s color, there’s just something weird with the overexposure or something. Maybe, as you said, it’s something to do with the gamut: those are LEDs, after all, on that bridge, and those might show up weirdly on the sensor or something. If only I could remember what the actual color was with my eyes now…

Anyways: thank you so much everyone, that is a really impressive response!

[anarcat]

@anarcat

Maybe they could give me a better basis to work from without too much work?

Turns out, they do help quite a bit:

That is the “Fujifilm XTrans III - Velvia” style out of the box. Of course some more work would be required on that shot, but it’s already an improvement over the original DT rendering, in my opinion.

It would be great if we could have camera-specific styles shipped with Darktable, so that we get a little better experience out of the box. Those could be auto-detected to match the settings used on the camera: it’s kind of too bad to have done some “editing” work done on the camera only to have it dropped when imported in DT… We already detect stuff like basecurves and white balance from the camera, why not add those styles as well?

[anarcat]

Okay, here’s another example of the issues I found with this rendering. I’m probably nitpicking like crazy here, but here we go.

Here’s a crop of the bridge of the original:

And here’s a similar crop of the image as beautifully rendered by @garrett:

What strikes me now is not much the difference in colors: that’s pretty close, and anyways, it’s just close enough. What seems to differ wildly now is the level of detail: the original JPEG is so much cleaner, so much sharper… I know that RAWs are less sharp, but I can’t figure out how to make it that clean. I see @garrett did some sharpening in the end to try and resolve that, but try as I might I cannot figure out a way to reproduce this mix of sharpness and softness in the bridge’s superstructure.

What is it about that JPEG that makes that picture so special? Shouldn’t I be able to get more detail out of the RAW than the original JPEG, if anything?

Looking at the “original” RAW rendering from DT, it sure looks like some data is just completely clipped off:

DSCF1110.RAF.xmp (3.8 KB)

Of course that looks much better with the linear Rec709 RGB input profile:

DSCF1110_03.RAF.xmp (5.2 KB)

but even there I can’t quite get the level of surreal precision of the original JPEG…

Thanks again for all the suggestions and comments!

[anarcat]

@anarcat

It does look slightly better, but still completely blue and not magenta. I’m now left to wonder how the camera figured out this was magenta in the first place. That bridge is a (huge!) multimedia installation and I know some of those people that made it (the installation, not the bridge of course), so technically I could try to figure out what color it was supposed to be if that’s useful…

Oh and for what it’s worth, I started asking around for that, just to see what the original color was. Stay tuned!

[Wocket]

[quote=“anarcat, post:17, topic:6669, full:true”]
What seems to differ wildly now is the level of detail: the original JPEG is so much cleaner, so much sharper…[/quote]

The detail is there, IMHO. The shodows in the ooc jpeg are just a little brighter. For example, you can use the shaodws and highlights module to bring up the shadows a bit more.

Also activated the local contrast module and used the equalizer to go for a little more sharpness.

[houz]

@anarcat

That is quite impressive, I got to say. It’s almost exactly what I expected… Your mention of “velvia” reminds me that I did choose a specific “landscape” film emulation (called “Velvia” too! coincidence?) on the camera for that shot.

So you used an Instagram filter in your camera and wonder why darktable’s output looks different?

I see you used the “color look up table” early on in the history stack: what does that do? It seems to be the key change the fixes the color on the bridge to restore the original JPEG… or maybe the history is not ordered the same way here?

The order of the entries in the history list if just reflecting in what order the user has used the modules. The processing order is always the same and can be seen on the right side of the screen: All active modules are processed from the bottom (“raw black/white point” for raw files) to the top.