New Sigmoid Scene to Display mapping

Indeed, I didn’t read @OK1’s super verbose posts because of their length. It is a good policy in any conversation to keep it in digestible chunks. Otherwise, you would risk alienating people and giving the impression that you are the most important person in the room.

If you must write a wall of text, be sure to summarize it at the top or hide it with this handy code:

[details=hidden title] hidden text [/details]

It would look like this:

hidden title

hidden text

Carry on.

4 Likes

Why? whats the benefit using different names for well defined terms? It’s already hard enough to translate terms into different languages without losing precision.
But feel free to provide a simplified_terms.po file to check if using different terms is really a benefit :wink:

Thanks for trying it out OK1!

For your input, I will see if I can break it down into something replayable, it’s a lot to chew at once.

Lens coating thing, no it should never be part of any global tone mapper or as you wrote std. Why? It is a local effect that is easy to simulate in scene-referred space without any other dependencies.

Not sure what you want to say with the example images. I have yet to figure out any set of reasonable presets but I feel quite good about the default as that is close to the ACES RRT+ODT (See earlier posts on the subject if you are unfamiliar). As for SOOC jpgs, I think these should be possible to recreate with ease in the raw developer as they are a valid edit, but I also see how unreasonable it is to deliver perfect matches as presets. To many brands, cameras, styles, and options to every make that feasible. Would you say that the sigmoid makes it easier or harder for you to recreate the jpg style? That would be interesting feedback!

  1. darktable is very much a dodge and burn tool, especially with the new scene-referred workflow. You are losing ALOT of potential if you do not want to do that in darktable. Pretty much all images I have posted in this thread are based on simple dodge and burn using masked exposure/tone equalizer before the sigmoid module.

  2. I think it is unavoidable to become technical when this much control is exposed, the reason Lightroom can use friendlier words is because they have fewer things to turn on. I’m open to suggestions though! I think the most important thing is to stay consistent within the software, that is why I used the same naming for display max and min luminance as in filmic f.ex. I’m actually considering renaming contrast to compression as it changes both contrast and saturation…

I know plenty of very technical women, no need to use that sort of generalization here. The module is still under development, so the topic of discussion is of course focused on the math and science behind it. There is no point in merging the module if the math doesn’t work out.

  1. Middle grey control in filmic is the same as an exposure change. Just use the exposure module to get the same effect. Do not use the shadows and highlights or color zones module modules for this as they don’t do proper scene-referred math.

a. Mock ups and suggestions are welcome, this is my best effort so far.
b. Of course, but later, not now.
c. Kind of what I’m using this thread for, more should follow as things stabilize.
d. Really no use to figure out presets yet as they will just break a few weeks from now because I made a change in the code. We have to figure out the math first.

  1. No, absolutely not, I have no interest in doing anything like that. I think you are vastly underestimating how much work it is to maintain software like darktable. And, I have no problem with the sigmoid not begin merged. I do all of this because it has been an awesome platform to learn more about digital image editing, not to start a new competing darktable version.

Thanks a lot!
Hope to see more images and testing coming from you (but maybe in smaller chunks as that makes it easier to answer properly :wink: )

5 Likes

Played around with it a bit and could at least reproduce the level of the filmic edits. I think the main problem in this image for darktable-users is that the highlight reconstruction is quite poor which leaves most of the sky wonky in this picture. There is information in there to make a much better highlight reconstruction than just clipping all channels over 1 though. This would indeed be a interesting topic to tackle at some point but nothing I can do atm. So the sigmoid module won’t be the hero in this case, it would rather be a proper scene-referred linear-RGB update of the highlights reconstruction module! (This is partly what I think is exciting about darktable development, there are still so many lose threads in the new scene-referred workflow that have to be figured out).

2 Likes

I haven’t had time to fully explore I was messing around with a bunch of filmic parameters including setting the latitude as low as it would go when I decided to try blending filmic in lightness blend mode. Not much changed in the couple of examples I tried but the sky for some reason is less desaturated maybe but I actually liked the look. I will experiment more and summarize…

EDIT: I tried it as I often do this if I use dehaze and it shift the color too much so I just thought what would this look like with filmic…

Most professional photographers are using Lightroom and they’re getting great results very fast in Lightroom. I think we should acknowledge this fact.

Thank you for highlighting (:face_with_hand_over_mouth:) this. Good highlight reconstruction is a must in any raw converter for me. Sadly, it’s the only thing holding me back from using darktable. I’m all RawTherapee 5.8 (personal projects) and Lightroom (work) at the moment. I’m not a developer, but having something like this in dt would be amazing. BTW, I’ve been experimenting with Sigmoid builds of darktable though and I was very pleasantly surprised. Generally I’m getting better results than with Filmic RGB with less time consumed. Win-win. Great work @jandren!

Do you say that as a developer, or a user (or both)? :slight_smile:

3 Likes

True, and my reply could be seen as a generalization itself, although I did preface that by saying “mostly.” But whether pro photographers use it or not, it’s still true a lot of commercial software, including lightroom, targets mass consumption, and simplifies some things to achieve their aim. Not all pro photographers are pro editors or have the time to be, so they rely on this simplification.

1 Like

Thats done - If Lightroom is satisfying the demand of a professional photographer then there’s no need to change the tools. There’s no ambition for darktable to take over as much users from Lightroom etc. as possible.
darktable scope is to provide the options Lightroom doesn’t give to you.
If you’re demanding just a free Lightroom clone then darktable is simply the wrong place to search for it.
It’s like the green rectangle on your camera - there‘re plenty of user for this but at least ambitious users prefers to customize their tools. darktable is focused on the M mode

6 Likes

OK thanks for trying! I found the same, that RT or ART gave a much more pleasant look in the highlights (gentle roll off, no sudden changes in colour) but this was difficult to achieve in darktable which either desaturated the sky or generated strange oversaturated colours, depending on which chroma preservation mode you chose in filmic. It may be user error on my part, but I tried lots of things including increasing the white point value in filmic as advised.

2 Likes

That was a nice example of how much better highlight reconstruction can be! But remember that the best is to protect your highlights at shooting time, any reconstruction has to make assumptions in the end!

As a developer :stuck_out_tongue:

As for your comment about Lightroom. I think speed due to fever options is a big reason for its popularity. I’m pretty sure few developers want a Lightroom clone as @MStraeten puts it but I also think we should pay close attention to the speed of the darktable workflow. From import, sorting, and tagging to reducing the number of clicks and parameter iterations when doing the edit.

Time is probably the number one constraint for many photographers so it’s nice to hear that the sigmoid module gets you good or better results in less time!

3 Likes

Modules/tools may not be aplenty, but, for being so fast to work with, I think Lightroom packs a lot of power (and polish). Take this little detail at 10m20s, the fact that you can move a masked gradient around and the position of the mask is kept in place. It’s a great example of what I’m trying to pin down.

I demand nothing from these fantastic tools I get to use for free. However, I wish more professional photographers (and retouchers) would use them. That’s why I’m interested in why darktable has trouble reaching outside the world of FOSS. It can’t only be a matter of marketing resources and lazy/uneducated users (which is somtimes mentioned as a reason around here)? darktable doesn’t have to be FreeLightroom™️, but that doesn’t mean there’s nothing to learn from LR.

I don’t know if I’m fine with the green rectangle metaphor. Yes, there’s an Auto Tone feature, but you don’t have to use it. I dare to say most people don’t. Lightroom has become quite sophisticated considering its ease of use. Selection tools are fine now that range (parametric) masks are available.

If it helps, I can tell you I’m not having a lot luck either. :slightly_smiling_face: I would approximate I’ve spent at least 60 hours learning darktable (probably more) and I just can’t get highlight reconstruction right.

Of course. Highlights are more easily protected in genres of photography where time is plenty (compare still life studio work to photo journalism or street photography).

That’s the answer I was expecting :grinning:

I read this article yesterday and had a hard time not thinking about darktable and Lightroom:

Complexity is like energy. It cannot be created or destroyed, only moved somewhere else. When a product or service becomes simpler for users, engineers and designers have to work harder. Norman writes, “With technology, simplifications at the level of usage invariably result in added complexity of the underlying mechanism.”

But also:

Norman writes, “Whether something is complicated is in the mind of the beholder. ” Looking at a workbench of tools or a digital photo editing program, a novice sees complexity. A professional sees a range of different tools, each of which is simple to use. They know when to use each to make a process easier. Having fewer options would make their life more complex, not simpler, because they wouldn’t be able to break what they need to do down into individually simple steps. A professional’s experience-honed conceptual model helps them navigate a wide range of tools.

Yes, this. It’s one of the reasons I use Lightroom for work. Another one being that the rest of the studio is on the Adobe suite, so I can send DNG with embedded XMP from Lightroom and they can continue the work in ACR when needed. Lightroom is boring but works, darktable/RT truly excites me! :slight_smile:

4 Likes

After I had posted the long read, the lingering question was - why do we have to put so much effort into revising one raw processor or the other or choosing between them?

Is it not possible to take the best bits of each, without having to, re-write the code and merge them into a single darktable module.

My other side is musical, and in that world, the digital and analog processing there does not think only in terms of this processor or that processor, when you can have a bit of this and a bit of that. You could plug any digital processor anywhere in the chain, In series, in parallel, like Shakespeare said - as you like it… For example, you can have several equalisers in the chain, each doing what its best known for. Top end (high frequency) enhancing, another processor for the bottom end. (bass). By the time we listen to most music, its been through several equalisers. And in that world, its normal.

In that world, plugins are the norm, taken for granted, and it all works together, from different developers, seamlessly without any committees to decide. I am not the first on this thread to publicly express a wish that darktable would be plugin enabled. Who knows, one day, one day…anything is possible…

We are familiar with HDR, Focus stacking, Panorama composites, combining images.

What’s stopping us, as end users, when we do not have the interest, responsibility or ability to code digital light benders(e.g. modules), from taking the output of one image processor and combining it with the output of another raw processor, in our own custom ways, in series, in parallel or both, not bound to whatever each processor design is capable of. And this can happen outside of darktable, or inside darktable.

So that’s how I have used sigmoid since my prior post, in combination with other raw processors, and each is chosen, by experience/testing, for their strengths on different areas of the same image. Colour rendition, highlight control, contrast definition, etc.

With this revised compositing approach, I’m 100% satisfied with sigmoid, just as it is, for my current use case, and appreciate the aspects in which it excels, moreso since I no longer have to choose one way or another, with only one image processor(or module), for all of the scene to display referred transform.

I’m now familiar with the sigmoid controls, so for me, no longer an issue.

Here is an image composited from s-t-d referred processing with sigmoid, and the s-t-d referred processing from another raw processor.

Some of my previous efforts on the same image here :

The raw file is in the other thread.

The article might have benefited from distinguishing between “simple” and “easy”. “simple” is the opposite of “complex”, while “easy” is the opposite of “complicated”. (to its credit, the article stays firmly within “simple”/“complex” , and does not mix metaphors)

I find this distinction particularly useful in the discussion of photo editing tools:

Luminar, for example, is trying very hard to be “easy”. Few sliders, much magic, very “artistic” names. There’s a slider called “drama”, and another for “punch”. It actually is quite usable that way. But the hard part then becomes predicting how that slider might behave and when it would be useful.

Darktable, to my mind anyway, is striving for a “simple” interface. One where every slider has a single function that can be easily understood. The hard part is more in how to effectively combine these tools, and less in understanding them individually.

The thing is that there probably is a particular combination of settings in Darktable that is a reasonable approximation of “drama”. But not vice versa. Filmic can not be emulated by adding “drama” to “punch”. On the other hand, it probably takes a bit of an analytic mind to enjoy the technical nature of Darktable.

To bring this back to the sigmoid module, it seems to reduce complexity without losing relevant functionality, thus making things easier and simpler. Perhaps the key word here is “relevant”. What I might consider irrelevant might be relevant to someone else. More importantly, I might consider something irrelevant only because I haven’t understood it yet

I don’t have answers. All I know is that the color preservation modes in filmic confuse the heck out of me, and sigmoid’s hue preservation looks like the rational (easy?) solution to that problem. Which I highly appreciate.

4 Likes

I don’t think it works like this. You’d need to re-code and maintain that code after.

This is essentially what darktable has with the pixelpipe, but since darktable is written in C, you must compile it all together, so we can’t just distribute plugins ad-hoc.

There is merit in the idea of plugins and there are several good examples where user-maintained plugins work seamlessly with the core software. Take Blender with add-ons, video NLE’s like PremierePro and DaVinci Resolve with e.g. Boris FX plugins and indeed the whole professional music industry with VST samplers, synthesizers, effects and libraries.
I can imagine @OK1’s idea quite well, where you would perhaps load a custom DLL, and then another module shows up at a designated spot in the pipeline of darktable. The biggest downside of such a model is of course compatibility.

2 Likes

I didn’t say it was a bad idea, rather that we basically have a plugin system already.

2 Likes

“plugin architecture” goes to the foundations of a particular software, rather hard to implement it retroactively.

That said, doing such to my hack raw processor may be the next thing I take on. It’s already organized to use such a mechanism (arbitrary workflow), and a plugin interface would let others do what they will, without by-your-leave from me. Every tool a plugin…

1 Like

I’m guessing the majority of OpenFX plugins will happily operate if you feed them only a single video frame.

Or: There’s already a fairly well established image processing plugin standard used by both FOSS and proprietary software.

That would handle most potential compatibility issues.

That’s a very personal wish.
I respect that, still, a very personal one.

Trouble to whom?
Again, a very personal opinion, clearly not shared by developers and those from the community closer to the development process.

With all due respect, I understand that newbies (in dt, in the community) may have these thoughts at start. What I don’t understand is why they insist and invest so much time on this discussion, even when all evidences show them it’s a void discussion. Why not spending that time learning more the tool, even sharing what they learn by producing tutorials, for example?

5 Likes

Just want remind everyone that this thread isn’t the place to discuss everything about darktable. Try to stay on topic with feedback on the sigmoid module, especially true for everyone trying to defend darktable. Just ignore or remind off topic comments about the topic of the thread rather than bringing in all the armor and weapons. I have no problem with lengthy discussions on the nitty picky details on the current sigmoid module but we have to stay on topic for this thread to be usable.

Examples that are off topic: plug in architecture, general darktable ease of use, any comparison to other softwares that isn’t focusing on their tone curve, Foss comments etc.

Rule of thumb: will this comment help Jakob improve his code such that the module gets good enough for a merge? Does this comment make Jakob smile such that he will spend more time on developing? Criticism about the module?
Yes? Please post here!
Uhh or no? You should probably post in a different thread or not at all.

Thank you :blush:

19 Likes