A plea for more guidance in the darktable manual

Take care when reading this: I am potentially the most dangerous man you will encounter. This ‘way beyond 6-sigma’ claim is founded on the well-established principle of “a little learning is a dangerous thing”. My little learning in general is especially applicable to darktable, into the understanding of which I have invested a slow lock-down year of my time. The trouble is that when I have a hard-won nugget of learning in my hand and start to examine it in detail, it soon begins a process of catastrophic dis-integration, transforming itself remorselessly into particles small enough to merely slip out between my fat, dumb fingers, leaving just a ‘WTF’ moment behind.

Let me give you an example: learning a functionally comprehensive module in darktable. Only because it is what I am trying to learn right now, I’ll pick the Tone Equalizer (fear not, I haven’t forgotten color calibration or filmic RGB or the magic new retouch module, but one step at time, even if it is backwards into grue-infested twisty little passages, all different, in my mind. My simple mind says this is a powerful module, constructed by a powerful mind(s): just look at how many options, parameters, sliders and charts it has; why, it barely fits in the right hand border on my 4K monitor without vertical scrolling, when I find the necessary scroll-bar…… “Ignore the quality; just feel the width”, as they say – anyway it’s fun to switch repeatedly from masking tab to advanced tab to see where the mask is now (wait… what ?).

So I’ve watched a number of tutorial videos on the Tone Equalizer by generous, well skilled and well informed presenters and think I now have a nugget of learning. I return to the manual (3.4 markdown version), which, it has to be said loudly, without equivocation and right away, is pretty damned good. The author(s) is/are obviously well-educated, well read, logically thinking, articulate, fluent in colloquial English and properly informed about the subject matter in hand – maybe they are even subject matter experts – which is not always necessarily a requirement for a good manual writer (I used to manage such creators, once….). Sometimes manuals produced by great writers have this strange psychological property of bonding immediately with (most of) the intended audience, while being deeply impenetrable to the awkward few. It’s not the fault of the writer, but the reader (dyslexia for example). I am positioned prominently in the awkward few.

An illustration of the process of disintegration of learning, after reading the manual, to which I previously referred, lies in trying to understand when and why to choose a particular option for, or to set a variable in, this module. It is here, in my opinion, that the manual has a problem. Yes, I think I have, at the very least, an initial understanding of what the options, sliders, presets, sampling pipettes etc. in this module do, but under what circumstances, and to what extent, do I apply them? That part of the manual is either indistinct in my reading, or suffers from Microsoft’s latest entry in the ‘shoot self in foot response to having shot ones-self in the foot’ competition with printing support (q.v.) in Windows 10 and is thusly missing altogether.

Example 1: What characteristics of my image would drive me to select a particular luminance estimator? In fact I choose it by what appeals to me on the screen - totally defeating the intention of the developer and doing despite to all the thinking and mathematics he has put into the development. There must be a more definitive guide to option selection, surely?

Example 2: As for Example1, but with ‘preserve details’ replacing ‘luminance estimator’

Example 3: What characteristics of my image would drive me to set the various options for mask edge feathering or mask ‘opacity’ (i.e. filter diffusion, smoothing diameter, edges refinement/feathering, mask quantization or mask compensation)? Here, the technique of using the blindingly stupid approach of ‘what looks good’ doesn’t really work since these are all about effects on the mask, not effects directly on the image pixels. So, I need to know what sort of image characteristics require sharply defined masked edges, or require a dense rather than a diffuse mask, or require a uniform rather than piebald mask, and so on. Today I don’t have a clue what to look for that might drive the setting of all these mask properties.

I want to make a special mention of the need for more ‘recipe’ or ‘prescription’ based documentation to help with the difficulty I have in understanding how module parameters should be set in relation to image aspects which have the property of ‘frequency’ – whether in association with noise or tone or sharpness or what, but especially in connection with wavelets. With my strange relationship with words, I tend to think of frequency as the number of cycles of value that a property goes through in a unit of time. Aside from physical effects beyond my understanding, I think the colours shown by the pixels on my monitor don’t have a ‘frequency’ in the sense that the blue, for example, continually cycles back and forth between, say, Prussian Blue (whatever that is) and Wedgwood. No, in time, is is always the same blue.

So this frequency being talked about isn’t time based. I conclude therefore is is ‘place’ based (if I remember the difference between Lagrangian and Laplacian analysis from my maths courses in the late 50s). So, if I were sub-pixel sized and were able to venture across the surface of my image data-set, it might be that I have to journey across a number of pixels before I see any difference in a specific property of the pixels. But what degree of difference is significant? If these pixels exist in a 16-bit world, am I looking for a change of 1 in 65 thousand or am I looking for changes of all 16 bits - between 16 zeros and 16 ones - to include in my ‘frequency’ count? Is frequency therefore related to the distance between different pixels as a proportion of all the pixels in a given dimension, or is it related to the size of the changes in pixels’ values? Or is it a combination of both?

The bottom line of all this is that if the manual says apply parameters of ‘this’ type if the image has high frequency aspects or ‘that’ type if there is low frequency, then I’m lost: all the understanding has run away through the holes in my hands.

Is there any salvation?

6 Likes

I understand your concerns and certainly share some of them. I too have trouble knowing how to apply some of the module options and often resort to trial-and-error. Sometimes, though, trial-and-error combined with “what looks best” is the only valid solution because all of the possible use cases can’t easily be described. If someone can come up with some guidelines I’m more than happy to include them in the manual but actually coming up with that is a bit beyond me.

In terms of the meaning of “frequency” it might be worth looking at the wavelets section of the manual. Generally high frequency is a way of referring to the detailed areas of the image (like the individual hairs in an animal’s fur) and low frequency to the flat areas (a clear blue sky). The section I’ve referenced shows a single image broken down into high frequency (fine) and low frequency (coarse) layers.

Hopefully the fact that the manual is in a more approachable (markdown) format now, means that we can get more contributions from people who understand this stuff :slight_smile:

… I tend to think of frequency as the number of cycles of value that a property goes through in a unit of time.

Yes. Suppose you have a photo of a green bush, lit by the sun from the side. Trace your finger across the image at a constant speed. There is high-frequency change as your finger passes over individual leaves, and low-frequency change as you slowly pass from the shadow side to the sunlit side.

Speaking more scientifically, we have temporal frequency which is the normal meaning of something changing over time, and spatial frequency where something changes over space.

Frequency is relative, and there is no absolute meaning of high frequency or low frequency, merely higher or lower.

Put this another way, from our photo of a bush we can make two images, representing the low and high frequencies at any threshold so that when we add those filtered images, we get the original back.

This is a most helpful analogy - a little glimmer of understanding appears. Thank you.

Wow, that makes me feel a lot more comfortable as I have spent years blundering about in every form of photo manipulation software, really judging things by what looks acceptable (to me) and only later realising that the image turns totally red when I enable the clipped highlights indicator!

1 Like

I want to say what has happened with me, recently. I hope this is not a bad topic in which to talk/ask about it.

At first, Color Calibration seemed simple. But I kept getting weird results. So, I went to the module’s documentation and tried to learn to use it correctly and systematically. I started getting much better results.

But for the last couple of days, I am still using it the way I learned, but I’m getting weird results sometimes, again. I don’t think I’ve strayed from what I was getting better results with. An example was the mountain stream photo, yesterday (I’ll add a link after I finish this post). The CAT swatch seemed to show a very good white swatch, but the photo itself ended up very blue/green.

Then, when I forced it to have warmer coloration, the CAT swatch looked quite green, but the result image looked great. And the module complained that my temperature was “invalid”. I am one confused puppy.

I guess the lesson is that the input can be strange, making the tools act strangely. Maybe.

Think of the CAT swatch as the color cast of the image that’s being subtracted to get you to a “white balanced” image. I sampled the whole image, the grey rocks, the white water, and got pretty much the same values. So, somehow when that photo was taken, the camera was fooled or detected something to force a green cast to the image. When I click on the original image in the history stack, I definitely see green.

As for the invalid label it just means that the detected white balance doesn’t “fit” into the a “normal” daylight range (i.e. sunlight usually isn’t green :smiley:). The label says “invalid” and gives you the custom sliders to control it. You can also get a “normal” result that maybe says “daylight”, then click the drop down and select custom, then manipulate the sliders to change the color balance to something else.

The problem with image processing is that the input possibilities are infinite so the best the software can do is gather all the information it can from the image and make a best guess. If you shot the image, then you know more than the software, so you can make some different choices. If you’re using a play raw, then you’re pretty much in the same boat as the software as you evaluate the image and take your best guess.

I’ll give you an example about color calibration. I shoot high school sports. When I shoot football it is at night under really BAD lights. They are incandescent and flicker when your are shooting in burst mode so some of your images are color correct, some are green and some are red, with varying shades in between. When I open an image that has tinted, I use what I know about how the image was shot and select incandescent from the drop down. That usually corrects the image, but sometimes I have to use the dropper and select the whole image to get a better adjustment. Trying to do it with the old temperature adjustment was impossible.

3 Likes

Thanks very much. That does help me to understand it better.

We are certainly missing other types.of documentation, such as quick starts, specialized workflows, and conceptual information mingled with examples.

I find the User Manual to be reasonably complete, and i wouldnt expect any major additions… The user manual isn’t an end-all be-all manual. The goal of it is not to document every single little darktable bit.

That being said, we need more youtube videos and more conceptual information with examples. @s7habo has been doing great with posts here in the forum explaining things, but he is only one person. We need more.

2 Likes

At the risk of swerving off-topic…

This is a most helpful analogy…

The analogy is exact. The pixel values can represent the position of a loudspeaker cone at points in time, tracing the x-axis. For example, here is an image:
toes2snd
We can run a finger, I mean the loudspeaker, along the first line, then along the second line, all the way down.

I made the image and the sound file with these commands:

%IMG7%magick ^
  toes.png ^
  -colorspace Gray ^
  ( +clone -flop ) +append +repage ^
  +write toes2snd.jpg ^
  -crop "x1^!" +append +repage ^
  +write toes2snd.tif ^
  -depth 16 gray:toes2snd.u16

"%SOX%sox" --rate 11025 toes2snd.u16 toes2snd.wav

Incidentally, I couldn’t upload the .WAV file to the forum. It says:

Sorry, the file you are trying to upload is not authorized (authorized extensions: jpg, jpeg...

That’s a great shame: I find this idea absolutely fascinating, so would have loved to listen to this image! By the way, does the mini-player shown not play because this wav file is not available, or is there some setting in my browser which I haven’t breathed life into yet?

The forum shows a mini-player for me, too, but it won’t play for me either.

I uploaded the sound file to my website and pasted the URL to the file, and the forum shows the mini-player. In Firefox, if I right-click on the player, I can “Save Audio As…”, which will download the file toes2snd.wav to my laptop, so I can play it. It sounds maybe like a helicopter taking off, with weird echo.

Consider the image and sound file to be https://creativecommons.org/licenses/by-nc-sa/4.0/

Tim, the selection is usually the right hue but can often be too strong for that reason if I don’t initially like the look I go immediately to custom in the drop down box. This does nothing at first but it gives you a hue and chroma slider…Drop the chroma to 0 and its like the module is not applied so usually I find I am dropping the chroma a bit to reduce the strength in a sense of the color cast or wb correction…I don’t think of cooler or warmer…I just find this to be the nicest way to fine tune this module…give it a try and see…in your example just dropping the chroma until there is enough correction applied will get you there…

1 Like

@LateJunction My ah ha moment with the tone eq was to use two or three instances and to keep the adjustments of a reasonable magnitude. Even with a pretty good mask random dodging and burning I find messes things up. I often use one instance just to target darker regions and then add more instances for contrast or other adjustments…I found in this way I did not have to rely on the mask settings. I pretty much just use the simple tone curve preset. That seems to land my histogram in the middle even sometimes and then I just make a small adjustment. For bigger global changes I try to use blend modes to help. When I first started with the TE I saw some videos doing random adjustments but this rarely worked for me…someone may read this and say you just need a better mask …but I think I have heard others taking the same approach…thats my shared experience for what it is worth…Understanding what you want to achieve goes a long way and may be one of the hardest things (its hard for me) …For example @s7habo recently demonstrated how to mimic a LR preset. He explains how he looks at the image and how he attacks it and most importantly he reorders the modules to achieve that. When you take this sort of informed logical approach your are not constantly pushing and pulling image tones and colors …you attack one part and then the other and you reorder the module so that they don’t interfere with the process…However this is not the sort of thing you will ever get from a manual…Check this out…How to use Darktable to acheive stylized image effects

Back to the topic of documentation:

There are four kinds of documentation that need to exist to cover all the bases.

  • learning-oriented tutorials
  • goal-oriented how-to guides
  • understanding-oriented discussions
  • information-oriented reference material

This forum provides understanding-oriented discussions thanks to the people who author the tools, and the manuals for darktable and RawTherapee are very information-oriented for when you want to know what something does, but the other two are lacking.

I know Pat started PIXLS.US itself hoping to make a place for more tutorial type content, but most of the activity has been concentrated on the forum side of things rather than the website.

Goal-oriented guides are missing overall. There need to be lists of desired end results, like “My red flower petals are turning nuclear, how do I fix this?”, each with potentially several ways to achieve the result.

In Filmulator’s case I rely on the fact that the tools are few in number to help guide the user, but the other more powerful applications need to provide more guidance for a user who has an idea of what they want but not how to get it.

5 Likes

Writing is hard and time consuming. We welcome submissions.

We get a lot of good tid bits in here, but even organizing them is time consuming.

Also videos are good here.

1 Like

I keep meaning to redo my intro to Filmulator video but I can’t seem to get through it in one take and don’t want to edit together multiples ugh…

Perfection is the enemy of done.

9 Likes

The ‘invalid’ (non-)issue has been discussed several times. It’s not an error indication. Some examples:

1 Like

I think this is key. The darktable manual right now is very much like the manual you get with your camera. The camera manual tells you what all the settings do, but it doesn’t help you with how to approach your photography and take better photos. Similarly, the darktable manual doesn’t really help you learn how to approach editing of a particular photo. Arguably, that’s not the job of the manual, but these kinds of resources need to exist.

If we look at some of the commercial software out there, there are lots of tutorials on specific techniques: “How to sharpen in x”, “How to make your photos pop in x”, “How to colour grade” etc. Then there are tutorials on specific genres of photography: “Portrait photography in x”, “Create stunning landscapes in x”, “Better night photography…” etc. These goal-oriented how-tos are wildly popular and very useful (the good ones at least).

The darktable series by Bruce Williams is excellent and essential viewing for new users. But it by and large still takes a similar approach to the manual, i.e. “this is what each module does”. Bruce has done some more workflow-based videos to help with a particular photography style, but there are not many of these tutorials out there in the wild.

So why aren’t there? I have a feeling it’s simply that there aren’t enough experts at darktable. I spend a lot of time on here and on the Reddit and Facebook darktable groups, and almost everyone is still learning how to use it. After two years of using darktable myself, I still feel I could only do basic tutorials, and even then it would probably be a more “drag this slider until it looks good” approach, because I couldn’t explain when to use Bilateral grid or Local Laplacian Filter, for example. I simply have no idea.

Which brings me to my next point, which is the UI. The addition of more tooltips in recent builds is a big step forward, but they are still very technical, and many tools don’t have them. darktable is excellent for the sheer amount of options it offers the user, but of course this can be overwhelming. I would welcome more goal-oriented tooltips of the type “Adjust this to boost x”, “Turn this down if you see halos” etc. I know this could be very tricky, but I’m just wondering how we can help demystify some of the more obscure labels like “RGB Euclidian Norm” and “Bias correction”.

Incidentally, i wonder if the “Invalid” label in the color calibration module should just be removed. It seems to always scare people until they come across one of the threads that tells them not to worry.

5 Likes