RAW sharpness with darktable

That may be true, but my reply was about clipping due to sharpening. I maintain that that’s not a risk for the highlights (unless the sharpening module you use doesn’t play nice with others).

E.g. with filmic, the exposition module is used to correct the midtones. It’s not rare that the highlights are pushed far over the edge, but they are not “clipped”, in the sense that the maximum value is is not bound. So after exposure correction, your clipping detection would show a lot of over exposed areas, which are then brought back in range in filmic with no loss of details. The details may be hard to see due to the tonal compression, but they are there in the data. Don’t forget that floats have 24 bits for the manitssa, and compare that with the 8 or 16 bits available per channel in e.g. jpg, tiff, png.

In the shadows, it’s a bit different depending on how the sharpening handles the changes. If it’s a multiplcation, you shouldn’t have too many problems, if it’s an addition, you can indeed clip data (negative values need to be clipped or adjusted before feeding them into filmic).

1 Like

When one starts on the journey with photo tools that can read RAW files, the comparison with the OOC Jpeg is an informative pastime. Initially frustrating as you search for darktable modules to transform RAW to something as good as the OOC jpeg. But I assure you, after mabny frustrating months, and several suicides (i.e the times when you just give up cos you are not making any headway or improvement, and ressurections, when you go back to dark table and try again, you wil discover that darktable allows you, when you know how, to far exceed the quality of the OOC jpegs. - Far exceed.

So with darktable you have a powerful tool, like a nuke - a nuclear weapon, it can power a submarine for years, or destroy a city in minutes., and you are in charge of the red button.

As you have seen from the earlier responses, there are so many ways to achieve results in darktable, so many.

A recent somewhat biased review on youtube accused darktable of being complex - which is partialy true, and while I still have some aspects such as transforming sky colours to more realistic like my eyes see, when it comes to sharpening - that is not an issue with darktable.

There is truth in the fact that unlike lightroom, many of the modules in darktable have many parameters, and it can be a bit difficult working with them.

When you speak of sharpness, may I interpret this to mean edge recognition, the ability to distinguish shapes and patterns.

In other words, if you took a photograph of a plain white sheet of paper, unless you did this with a microscope, or some reasonable enlargement lens like macro or magnification tubes or bellows, with typical camera lenses, there would be absolutely no features to be seen. Such that even with vignetting you really could not distinguish any curves, lines, nothing. And if you took a photo of any other single color object same.

So permit to use the term contrast - loosely, by this I mean micro contrast at the level of a few pixels, to define edges of objects and patterns, and global contrast to define the variation between the darkest parts of you image and the brightest parts.

I think you could also define regional contrast of the kind that occurs between sections of the brightness/colour spectrum, such as between dark green and light green or between brown and green, or between dark tones and mid tones. i.e the darkest and lightest tones may not change, but using various tools, such as an S-curve in any of the curve bending modules such at tone curve, base curve, or rgb curve, you can enhance areas of contrast in between the micro contrast and the global contrast.

Global contrast can be modified using lots of method, using the exposure module to adjust the whitest whites and over all brightness or the darkest darks and how much of each of these you want, i,.e the extremes. The Contrast brightness saturation module also has a slider for contrast.

The sharpness you seek, is a combination of all these 3 forms of contrast. Global, regional, and micro, and regional (maybe I use macro instead for this cos its not really about regions, but about regions of brightness variation).

So over time, with experience, and lots of round trips, and weeks of learning and sometimes giving you eyes and brain a break for a few days (highly recommended), you will achieve 2nd nature like driving a car or a bicycle, from all the mistakes you have made, and learnt from (sorry no other way to learn this except my making a real fool of your perceptions and learning how your eyes may occasionally deceive you, without anything else to compare with), in picking teh right tools for addressing each area of contrast.

I will go directly to address micro contrast, edge sharpness. I find the following method the most efficient.

Using the highpass module.

Adjust the sliders in the highpass module to define how much change you want in sharpness., then using the uniformly button (the circle), in normal mode these sliders behave like a mask selection set of controls.

Change the blend mode from normal to overlay, to apply this sharpness mask on the image.

Or change the blend mode from normal to softlight, or any of the others in the same group as overlay and softlight. See which works best for you.

You can tone down the strength of this change using the opacity slider in the highpass module.

This is the quickest and easiest to control, method of sharpening an image in darktable. Other method are valid, but this is the most workflow friendly, cos it has only two main sliders. Yes it does have a twist such as changing from normal blend mode to overlay. But once I knew about this approach - and you will see lots of examples on Youtube, I do not use any other method for micro sharpness,

Be gentle cos a little goes a long way, otherwise you will over do it and your images will become very digital looking.

The key with darktable is finding the shortest path to your desired result using tools whose impact is immediately visible and preferably not subtle, so that your creativity is immediately rewarded.

I must also add, with respect to images looking like your ooc jpegs, there are so manu it depends so many.

Yoru RAW file needs a curve to transform the boring contrast reduced image to one that looks more like real life.

In tools like Adobe, they have 25+ years of history with learning how to apply automatically a curve that represents something similar to the curve that is applied to create the jpeg ooc. This is one of the key challenges of using darktable, not a criticism, just a statement of fact. The development focus is on predominantly the features that transform an image, before and after this curve is applied. darktable provides a method of applying this curve, but the curves provided are somewhat generic, unlike Adobe where they have internally provided curves which you cannot access or tweak. So adobe and darktable apply what darktable calls a base curve, but in Adobe lightroom, that “base curve” is called something else - like profile and you cannot edit it - only pick what you want - most people who use Lightroom are not aware of this transformation.

Darktable like Adobe Lightroom, then goes further to also give you a tone curve(similar to what is available in Adobe) and an rgb curve. For additional user driven control.

I rarely use the native base curves provided by darktable. I spent about 2 months creating a set of custom base curves, which allow me to quickly change the overall look and feel of a raw image, far more powefully than using a combination of different modules in darktable., I have about 15 of them which span the spectrum from relatively bright and contrasty, to dark and not so contrasty, and then all the way to dark and still with contrast. These curves came about cos the default base curve assigned by darktable, based on camera model/brand did not give me a similar look to my ooc jpegs. Not the fault of darktable, this is a very difficult feature to develop - i.e ensure you have a curve for every single digital camera ever made.

It is not an exercise for the faint hearted, and I do not think it is one i would want to do again. and it means I must always keep these curves, saves as base curve presets, also backed up so I can restore them if my computer fails or hard drive fails, and I cannot recover data from it.

Applying the right curve, and in many instances, this will need to be a custom one, took me to the point where I no longer ask the camera to generate any standalone jpegs to accompany the RAW version. (partly also because I can extract the Jpeg version - as this is embedded in the RAW version already) Lots of open source tools to extract the embedded Jpeg from the RAW, if I ever needed this.

Regrettably I have not heard of others who went this far to tailor darktable base curves to improve their workflow, so its the evidence of only one person - me. I’m afraid, so you just have to trust me on this one. This was the 80/20, that weaned me completely off jpeg comparisons. Now I have a new problem. Running through 5 or 10 base curves with each new set of similar photos, to decide which of my custom base curves or the native ones provided by darktable, look the best with that image - a good problem to have cos typically I end up with at least 3 different base curves that do justice to the image and I know have to whittle it down to one or two. Almost like using presets, and these custom base curves are extremely powerful presets, that reduce the amount of work that I need to do in other modules.

There is another approach which instead of using base curves, uses technical LUTS, to get you very quickly to an image that looks good, similarly you develop a bunch of these and overtime have a small preset of 10 to 15 of the ones that work best with almost any pic.

So about 60% of the transform is done using a LUT preset or base curve preset
10% is achieved by adjusting exposure sliders.
10% is achieved by changing white balance
10% is achieved by additional dynamics changes and toning using Tone curve or RGB curve, if needed

The remaining 20% of effort is finalising using any other tools., such as the highpass for sharpening, Contrast Brightness Saturation module as needed - less is more, Denoising using one of the denoising modules,

Before all this - it does help if the photo was well taken, properly exposed, with a good super clean lens without excessive issues like glare, with proper focussing - it takes more effort but I now go manual focussing - and allow the camera to control shutter speed - in aperture priority mode - and by checking each photo, I can retake by adjusting exposure compensation - in camera, on the next shot. If the intended sharp areas in the original image are not tack sharp, for a variety of reasons including camera shake, there is no point attempting to improve it in Darktabe. The challenge with auto focussing is you are sometimes not sure what areas are in focus. and therefore sharp.

With manual focussing and using focus peaking - before I take the shot, I know exactly what aspects of the photo will be sharp, and when I get into darktable, its exactly as intended sharp as per intent, and easier to sharpen further using the highpass module as described above.

I must add - you could also using masks of various kinds limit the areas of the photo to be sharpened. You see - if everything in the pic is sharp, then nothing is really sharp. sometimes it helps to sharpen specific portions only - typically the areas in focus, and this makes the entire pic appear even sharper, cos some areas are more in focus than others, in a similar way to how the eye sees - not everything is in focus with the natural eye - even when we look at a landscape we do not focus on everything at once - never, Our gaze is only centered on one part of the view at a time., and unless one is doing photos for certain kinds of commercials, good to have a realistic target for our sharpness intention.

I am confident all of the above has enough for you to consider and act upon with few constraints

5 Likes

Welcome aboard…lots to chew on in that post.

Harry Durgin did videos a number of years back on constructing custom base curves…
I think there is a tool for it as well no???

Thank you.

1 Like

Just to clarify, the several weeks of torture that led me to create custom base curves, was because the default base curve assigned was clearly not matched with my ooc jpegs.

I have read of a method that uses some open source software that is a part of the darktable distribution in linux, as well as a product with colour blocks, to measure and create a custom curve for each camera. i.e you take a picture of the color card and it compares it with some other thing, to create the curve.

The process I followed for creating based curves was painful, all done by hand, about 20 to 30 points per curve, you create teh 1st one and try to get it as close as the result of a jpeg ooc image.

Then you do the same for some other image, e.g taken with different conditions like brighter light or less light.

And then over time you create variations of these, tweaking aspects of the dark parts, mid tones, and highlights. All done by hand. No automation.

and eventually you whittle down the set to about the top 20 that you find most useful, over time you tend to settle on about 4 o 5 favorites that give you the desired result quickest. You need something to brighten mid-tones, you kinda know exactly which one does this best.

Over time during and after creating your base curves, you discover and remember which ones to use (and you could go back to rename the base curve presets - using these observed transforms)

What makes it the more difficult is that in teh normal linear scale display, especially in the darker regions, base curve points are so close to each other, they are really hard to tweak. So the secret is to use the base curve in logarithmic mode during editing.

So setting a base curve after your library of hand tweaked base curve presets is ready, is simply a case of selection. you have a hunch which one is likely to work, and you try a few to get the closest to your final intention for the image.

Typically I never tweak any of these base curve presets, cos they are like set in stone, and reused across many images, so leaving them unchanged allows me to edit an old image, and ensure that darktable can “tell” me which preset I chose for that image. If I were to do any edits to the base curve in an image, this association by darktable would be broken.

If I had to do this again, it would take me about a week or less, devotig an hour or two each day. Much faster than the several weeks it took me when all this was new to me.

I had no cl;ue it would be such a game changer in my workflow, I was simply experimenting with how to solve what I saw as an issue with darktable’s workflow.

What I also found was that when you have a diverse set of custom base curves, I also changed cameras, from a Fuji to a Sony )both mirrorless) and the base curves were still just as productive a workflow accelerator with the new camera. So its really not about getting a perfect curve with one camera, but more like getting a set of candidate curves, that would work in pretty much any camera.

Typically any finetuning of base curves is done in the tone curve module. either using one instance or several. Why? darktable allows you to turn on and off a module, and sometimes I want to try out an incremental set of possible changes to the result of the base curve. Most productive manner is to break up these curve based tweaks into multiple instances of the tone curve, which I can turn on and off individually.

I wish I could have a tone curve and an rgb curve module, that enabled me to turn off or on any points added to the curve, this way I could assess the contribution of just one or two points on this curve., without having to turn on or off the entire tone curve module or rgb curve module.

In the same way that LUTS have really taken off, and Presets in Lightroom have also become quite a thing. I’d like to hope that in the near future we can have collections of Custom made Base curves, which we could publish as compilations and share with others to improve their workflow and accelerate this aspect of the workflow in darktable, I’d be happy to share a few of mine - as much as I feel quite precious about them, in the spirit of supporting the superb work of the darktable developers. One current challenge with darktable presets is they come in a long list., so it may be unwieldy to have too many, which is why I limit mine to about no more than 20 custom base curve presets. Maybe based on subdirectories, we could have darktable modified so that the presets are also grouped into sub categories, based on the directory structure in which they were “imported”, to avoid cluttering up the preset menu into one long list.

3 Likes

I’m sure many would appreciate them. I too am rather surprised custom curves haven’t become a thing, as they avoid the interpolation of luts.

It sounds like you are using the display referred workflow. Have you tried scene referred? Most here will use filmic, which comes with its own curve. Recently, I have taken to using filmic only as a tone mapper to control Black and White points, whilst making the curve straight. This allows one to use a custom curve in another module. I’ve been doing that either with either tone curve or tone equaliser. So is there any reason you chose the base curve module over those?

I also find it useful to Compare my custom curve to some other Presets, as there is nearly always a preset that does it better - but rarely the same preset each time. The Presets I’ve been using are the film emulation luts from rawtherapee, which are downloadable from rawpedia. That folder contains hundreds of luts. I’ve narrowed it down to about 30 favourites which produce the goods regularly. With more use I hope to narrow it down to 10 or less. But I wonder how easy it would be to extract curves from these luts and save as curve Presets?

This is a particular topic of interest for me at the moment (albeit off topic from original discussion).

Unless I am very much mistaken, unless you have a strictly parametric (mathematically well-defined) curve, interpolation happens regardless. The basecurve in darktable is not parametric afaik.

In any case, we’ve drifted off topic quite a bit here. Also, I notice that I’m getting increasingly irritated when I see people vehemently wanting to replicate their in camera JPEGs. If you like them all that much for how they look, why jump through hoops in editing the raw yourself? :man_shrugging:
Anyway I’ll try to keep my personal issues to myself… :slightly_smiling_face:

2 Likes

TBH I do not get it, why. I must admit, I did not check all vids from @s7habo, but I saw some and what I remember does not explain to me the contr-EQ stacking.

I downloaded the xmp from @anon41087856 and in a duplicate I swtiched off two of the contr EQ and increased the first one accordingly, to do it all in one. Is it my eyes? I did not see that much difference. Too subtle IMO to spend the energy (or do I still miss something?)

1 Like

You know, I asked that question a few times, and never got an answer…

1 Like

Suggest we create a new thread to discuss this further here :slight_smile:

To avoid derailing the current thread

Link to new thread below

Darktable - Workflow approaches - Custom Base curves, LUTS, Filmic

But the answer is quite simple. People like some aspects of their OOC jpegs, but they don’t want to lose all the possibilities of raw, like for example being able to recover shadows/highlights or tweak WB - basically to fix some occasional mistakes or difficult conditions.
If you like how OOC jpegs generally look like, but don’t want to lose all the possibilities of raw - I feel like it’s a very reasonable request to know how to make your raw generally look like OOC jpegs, maybe just as a good starting point for editing or just for fixing problematic things (e.g. shadows).

3 Likes

Regardless the motivation, I am very happy about all what I have learned here

replying to my self…
What I did is:

  • set one contrast euqlizer
  • snapshot
  • attenuate mix down to 0.333
  • duplicated times
  • compared to snapshot

There is an effect visible. Donno why, but it is like this…

That presupposes that users pick their camera (at least in part) for the jpg rendering. I know I didn’t, and I don’t like the Sony basecurve at all (look at what it does to the highlights).

And I think it’s quite unreasonable to expect any raw development program to be able to produce a look identical to the OOC jpegs of brand X, which is usually what is requested/complained about (after all, that’s what “replicate” means). One of the problems there is the multitude of scene settings modern cameras have (be they “smart” or not"), and even the possible settings for the “standard” renderning…

If you like the general look of OOC jpegs, you can use the basecurve approach, and pick a basecurve that you like, no need to stick with the curve corresponding to your camera brand. But that will not give you jpegs corresponding to the OOC rendering.

Perhaps the difference is just that x × 1.3333^3 > 2 x (it’s ~2.37 x )

Because in OOC jpg-s have a great color grade and look but you still have to pull out some shadows/highlights, correct exposure, retouch some parts of the image, correct perspective, crop, lens correct for third party lenses. dehaze or do something else etc.
The deal with the ooc jpeg is that it has a great contrast and colors if only it lacks a bit of saturation sometimes. But there are many other corrections that need to be done or creative editing if you want to do that.

1 Like

My feeling is that people view the OOC JPEG as a starting baseline. Put a little differently, if I can’t do better than my camera settings then why bother with raw, or what am I doing wrong?

1 Like

Interesting write-up. I came across this advice from @anon41087856 from at thread at Tactica.com easy ways to sharpen skin with darktable (hopefully I haven’t broken any cross-posting rules), where the sharpen module is actually exactly an highpass filter blended in overlay mode, and furthermore raising the right most node in the contrast equalizer falls back to a highpass filter.

So, it would seem that you could do everything and more in the contrast equalizer - plus new insights into stacking CE instances - if you can predict the results from the settings. For me, it might help if I had a better understanding of what’s going on under the hood of the CE module and how it relates to the others

1 Like

Interesting. Would you believe I have never heard of or used the Contrast Equaliser module in darktable

Did not know it existed. Using the highlight module has become so 2nd nature, and really quick, and also easy to tweak to taste, I suspect even after I have studied other candidate modules that can achieve sharpness, I most likely will continue using the highpass. cos its so quick and so easily tailored, using the opacity I can quickly reduce its effect. Now that is one gem in darktable, pretty much all modules have the opacity, so when you;ve gone overboard, so easy to reign it back, without changing the sliders in the module itself.

This is probably the piece de resistance of darktable, and I cant find this in any alternative tools - masking and Opacity in almost all modules. kinda like having Lightroom and a bit of Photoshop in the same image editor. un beatable.

1 Like