I get the feeling that we’re drifting from the aims listed here:
2 Likes
Thank you.
May I respectfully and cautiously extend the proposal to add.
Note, I do not know the intrinsics of scene referred and display referred, and how these are implemented, so I may be speaking utter nonsense, because I am ignorant of the implementation.
But thinking with an open mind, assuming it is possible.
Could any modules such as Sigmoid, which may implement such regional contrast adjustments controls, also have the possibility of turning off their global transformation
Why?
Today we have different modules, for adjusting aspects of luminosity, globally and all the way down to very granular detail, manually with curves. So you pick and choose what suits your intention best.
With the scene to display transform, should it not also be possible to have multiple modules perform aspects of this transform, so choose each for what you prefer.
e.g.
- Module 1 - Sigmoid - for Regional contrast.
- Module 2 - Filmic - for Global transform
So you can pick and choose a transform path that has the features of each which you like best for the image in question.
Options to achieve this.
-
If the global and regional changes are implemented in the same module, this could have the option to turn off or on, either of these changes.
-
A module like the current Sigmoid may have a companion module which addresses only regional aspects, and you use the companion module. Reminds me of rockets with primary and secondary stages. No need to have everything in one module.
So if I wanted to test regional gradient adjustments, independently of global transforms, I would have two sigmoid instances, and turn on only global transforms in one, and only regional transforms in the other.
There are lots of details I have left out, otherwise this becomes exhaustively long, such as in what order these would be sequenced, in the workflow, in dt. But these are implementation details. Best to partition requirements from design and implementation. I’ll stick to requirements, and let others worry about the best way to design and implement.
In simple terms, it would be great to be able to also turn on and off regional gradient adjustments, per region, or turn off all regional gradient adjustments., as an independent set of changes from the global scene to display transform.
Using rgb curves to achieve this not efficient - or needs multiple instances, and you cannot synchonize the cross over points between regions, across multiple instances of curves - not elegant. Best achieved in sigmoid or filmic.
1 Like
Thanks for your efforts!
Could you share the raw image you used as an example as well as maybe one more example with raw included? Helps a lot with my understanding to open your image with your edits.
When comparing the sigmoid and filmic, remember this: filmic preserve color = rgb ratio option in sigmoid. Filmic preserve no = crosstalk option.
The sigmoid module should be able to produce low contrast images as well but I can see how it lures you into high contrast as it is so easy to just push it a bit more.
Please find attached, with the xmp’s used for the images posted earlier.
For now, kindly park(suspend) any consideration of the “regional gradient” posts, cos the functionality I was considering, may have already been implemented in the color balance rgb module, using the luminosity slider for each region. It’s current implementation may not be as fine grained as described in my wish list for sigmoid, cos at this time, there is no specific control over the amount of crossover from shadows to mid-tones and mid-tones. I.e if I turn down luminance in the shadows, it still affects the mid-tones and highlights a bit more than I would have hoped for. It might be more profitable if I fed any suggested changes to the team responsible, as the module is 80% of what I had in mind as an enhancement for sigmoid. It was a “nice” idea, but I would not wish to suggest any duplication of features across dt modules - not efficient use of minds.
Nevertheless, being able to control the global contrast in sigmoid, is definitely a highly and most relevant sought after feature, or missing item of knowledge (if this is already attainable in the current version of sigmoid).
Raw Image
DSC02045.ARW (16.0 MB)
Cropped + Filmic
DSC02045_01.ARW.xmp (50.3 KB)
Cropped+Sigmoid
DSC02045_02.ARW.xmp (51.6 KB)
For whatever its worth, the out-of-camera jpg, is also here.
1 Like
If there was not enough control in the masking tab of the rgb color balance module I think you should start a new thread and get/provide feedback tips. I have not played with it that much but it seems highly controllable to me. Perhaps your use case will provide a source of discussion and indeed reveal if there is a deficiency or use case not being met by the current capabilities…
EDIT or you could take this approach and make presets with luminosity mask using the parametric masking in your module and create presets for the tonal ranges. Then every time you want to hit one of them you could invoke the preset for that range Luminosity Masks in darktable | darktable
You can setup as many levels as you want …this classic article does it in GIMP and creates three levels for dark, mid, and highlights GIMP - Luminosity Masks
I was doing some further testing/review/use in practice of sigmoid over the last day or so, and have a few more observations. I am following the “protocols” on this forum, so the details of each topic are “hidden”. Each forum has its own “culture”.
1. The Darkness. (not a horror movie!)
The impression I had formed and discussed earlier, about the tendency to “darkness”, which led me to try to find some way of resolving this.
If you turn on the clipping indicator (ideally set to clipping preview mode = Full Gamut), the images with this darkness, are those where settings in sigmoid have caused the dark regions to dip below the black point. So even aspects of the image, which are not 100% black still have some of the darkness applied.
All attempts to use other tools to resolve this (the new colorbalance rgb, rgb curves, etc) met with mixed success. Improved but not resolved.
When the clipping indicator hinted at what the cause of the issue may be, next step I took was to use the black point correction slider in exposure module, set to a negative value, which brings the black point back to where it should be and there are no more warnings in the clip indicator. But this black point adjustment is a broad one affecting more of the image than desired. so the entire image goes a bit dull. Not the most ideal solution.
The optimal solution, which I was satisfied with was to leave the exposure module as is, no adjustment of black point in the 1st instance, and use a 2nd instance of exposure module, which processes the image, after the 1st instance, and a parametric filter set to a grey value range that restricts any change to only the darkest parts of the image. In this 2nd instance of the exposure module, the darker parts of the image can be modified using the exposure slider in the exposure module. Positive EV, to brighten things up in the dark areas, and Negative EV, for the opposite.
Its my workaround, so I can us sigmoid with a lot more “latitude” i.e much more than I could using only the features in sigmoid alone. in some ways, this is not unusual, every tool will have its limits, or its own quirks, which need other tools to complement it.
I am not sure if this issue is something that needs to b fixed in sigmoid, or something that the users will accept and find workarounds for, as I have done, and possibly others may find my workaround something which helps them also.
I do not know what could be done in sigmoid, but if I were to express myself in layman’s terms, I would almost wish it was possible to move the target black indicator in sigmoid, to a value below its current minimum. Whether that is correct maths, or valid in image processing, I cannot say, as this not my area of expertise. Am only an end user, in this case.
Good enough, if it is fixed in sigmoid, or via a workaround as I have discovered, what’s important is there is a solution, and for me the concern I had with the dark tendency, is no longer a showstopper.
2. The Contrast
To a certain extent, yes sigmoid enables low contrast images, but the range of values on the contrast slider, within which one can lower contrast without turning the image to something undesirable, is a bit small, so one has to be quite delicate with the mouse or use numbers or other value adjustment feature with right click - can’t recall the name of this user interface feature, but I’m sure you know what I mean.
Somewhere around the midpoint of the contrast slider, in the current version of sigmoid, if one could have finer control, or maybe a 2nd slider that allows the use of the entire width of the tool area to make fine adjustments to the value of the contrast slider.
3. Houskeeping
Ultimately as the custodian of sigmoid, at this time, its up to you to decide when and how, a more formal structure will be established to coordinate the progress of sigmoid. At this time, its just you - one person. On one thread in pixls.us.
At some point in time, to obtain the maximum from the community, it would be helpful, to have the following, if it were possible.
a. A Roadmap. What’s the direction of the “project”. Without a very specific agenda, it becomes difficult for others to know exactly how to help, cos the direction is undefined, or loosely defined, or not yet communicated.
b. Repository. While sigmoid has a code repository, I hope you will agree that all the other items mentioned in this topic, also need to be stored in this central sigmoid repository, ideally also on github, as the current thread, may not be the ideal place to enable others to read, learn, and understand what sigmoid is about. The current protocol requires any discoverer to thumb through an entire thread, which is the only place where information on sigmoid can be found. Not very elegant. Or easy to do.
c. User docs. Sure if and when this sigmoid is formally included in dt, I am sure there will need to include documentation in the manual. There might be value in developing this now.
d. Project Team. Would sigmoid benefit from having a formal team of contributors, which can vary in composition as the need arises, who work with you a bit more closely, than this is possible via this forum thread alone, to augment your efforts? At such a time as you may consider you need this team, please establish.
e. Requirements. Each version, feature set/build will need a set of its own requirements, otherwise testing will not have the utmost efficiency, cos there is nothing to check against.
f. Testing. At this time, we may consider sigmoid as a “Proof of Concept”, so may not need rigorous tests, but at some point in time, there will be the need to have formal tests, and these tests are best achieved only when conducted against a clear set of deliverables, defined prior in the Requirements for each build/version. Would there be the need to collate a set of test raw images, against which all testing will be conducted. These images being a representative test of all kinds, overexposed, underexposed, saturated, out of gamma, etc, etc…
g. The build environment/Strategy. I would expect that most of those interested in sigmoid, would be running at least three environments of dt. In my case there is the version 3.4.1.1 (stable), then the latest dev version which I built about 2 weeks ago, and then there is the build derived from your github repository. Until sigmoid is integrated into dt, would it be possible to have a monthly refresh of your custom build, which is synced up with the latest dt dev code.
The sigmoid build, includes a version of dt dev which is several months behind the current code version. If your sigmoid build, could be synced monthly and refreshed with the latest dt code, then for many of us, we can keep only two instances of dt running, one with sigmoid which is no more than one month out of sync with the latest dev build, and our usual stable dt release. This allow us to test and experience the most recent developments in dt(no more than one month out of sync), with your sigmoid code, in the same build.
This text will be hidden
4. The DT pipeline may have some issues, which affect perception
I sincerely always wish it were possible to discuss sigmoid, independently of darktable’s other features, but this is not so, as much as we or you may wish it. Why? at this time, sigmoid relies on the DT infrastructure, to provide other services, on which sigmoid must rely.
DT provides demosaicing of the raw files, and allows us to display the image, etc, etc. export the image, and whatever we view or export is heavily reliant on the quality of the services provided by other DT features/modules.
While testing sigmoid, it is only sensible that I would compare its results with other methods of achieving the same thing, in DT or in other tools.
Whatever I say next, is a function of many things, for any other who reads this to be able to appreciate. In testing DT from different environments, we do not have the same displays, the same in-room lighting, and neither do we all have the same quality of eyesight(vision and colour appreciation), and for the kind of evaluation I have done, which is not based on any scientific tools, using only eyesight, any comments I make here, are impossible for me to verify with any measurement tools. I do not have the skill or experience to compare images scientifically, and can only describe what my eyes see. Others may have a different way of seeing these images.
When comparing images across different raw processors, I tried to exclude any variables, with the raw processor being the only variable. But this is impossible, except when using filmic, which shares the same pipeline as sigmoid, in DT. When using other non DT raw processors, which have their own sharpening and noise reduction, and colour science, I tried as much as possible to arrive at an equivalent way of comparing results with sigmoid.
It revealed something most unexpected. I either do not have the skills to get the best out of DT’s noise removal and sharpening modules, or DT’s noise removal and sharpening modules deliver results that also “colour” the final outcome, in their own unique way - a way which I was not especially pleased with.
In my comparative tests, I noted that the sharpening/noise tools in Lightzone (I’m using version 4.1.1 which I found to be the most stable and resource efficient on Windows) was IMHO producing images which looked quite visibly better than the results from DT.
To prove this conclusively, I exported an image from DT, processed via sigmoid, with:
- All sharpening and noise management turned off
- Exported as a 16 bit Tiff, using LinearProphoto RGB as colour space, uncompressed
- Imported into LIghtzone
- Applied Lightzone’s sharpening and noise reduction.
- Exported from Lightzone as tiff 16 bit keeping the Linear ProPhoto colour space
- Imported tiff from Lightzone back into DT.
The Linear Prophoto colour space may not be significant. I have not attempted using sRGB as the colour space during the transfer from darktable to Lightzone and back, but I do not think this should be of significance.
I was rewarded with the most impressive view, of what sigmoid is capable of, when the image has had an alternative processing to what I could achieve with the current sharpening/noise removal tools in DT.
I need not provide a photo, or image, cos this is science - hopefully, I have outlined the steps I followed and anyone else can check my assertion, by themselves.
For whatever the reason, my own skills or the excellence of Lightzone, processing sharpness with Lightzone, was visually more appealing, than with DT. I could have vouched that I had had a camera sensor upgrade.
I mention all this especially for anyone testing raw processors, as the raw processor is just one factor in the “environment”, and many other factors outside the control of the raw processor provide an objective or subjective difference to the end result.
If there was one word to describe the difference - it is resolution. With ligtzone’s sharpening, I was seeing so much more, almost as if the image had several times more megapixels, or my display had more megapixels, or my lens had developed a macro ability of microscopic vision. The level of detail I could see in the image was so much more… And its something everyone can try out for themselves. The image from DT modules alone is great, until you see an unsharpened image from DT, sharpened in lightzone, you do not want to go back.
I would recommend that while sigmoid is unavoidably tied to DT, anyone testing sigmoid, should adopt a process similar to that described above, so they can get a more revealing image of the end result of sigmoid, and can see so much more of what it does.
Mine is not to explain why, I have no clue of the sharpening code in Lightzone or DT, can only describe what my eyes observed. A pretty shocking alternative, and relevant to anyone testing any modules in DT.
5. The Colours
From values above 2.7 or thereabouts, the contrast slider, creates images which are definitely out-of-this-world, more like an effect, which some may value, but not a natural rendition. More like you turned on some artificial lighting on the image, and colours become vivid. Do keep this feature.
Had a go on your image and managed to get a result quite close to what you had with filmic. So I would not categorize sigmoid as always more contrasty.
DSC02045_03.ARW.xmp (76.4 KB)
You can easily make it even less contrasty by lowering the contrast parameter even more:
DSC02045_04.ARW.xmp (73.3 KB)
I noticed that you used quite heavy noise reduction even though there is almost no noise in the image (coming from a Canon 5D Mark II user who is used to a lot of noise). The result is that some parts of your filmic image will look much smoother than in the two above from me.
For some of your questions:
No, you would need binary masks that would be super easy to spot or you would need parallel processing pipes that are recombined as one result. darktable only supports sequential processing. What you are asking for is a so-called node-based editor.
There are also super duper highlights as there shouldn’t be any limit for how bright things we can edit. 
But a good observation, it doesn’t stop with just highlights. You can make this happen using the tone equalizer even though it’s a bit cumbersome for that application. Change the preserve details settings under the masking tab to “no” and you have this kind of control, kind of. The fixed 8 control points are a bit stubborn if you ask me but they get the job done.
You actually have a lot of control with the contrast + skew parameters in sigmoid as you can move the point of peak contrast to any luminance in the image. What you can’t do with the sigmoid module is to create wiggles in your tone curve. This is by design as wiggles create really weird higher derivatives which make the curve less smooth.
1 Like
The Darkness:
I think you need to show this in pictures, f.ex. screenshot with visible gamut check settings + raw + xmp. Because the curve is guaranteed to stay within the working gamut if you haven’t changed the display target settings.
The Contrast:
I think the darktable solution to this is to offer a tighter soft limit range. Something more feedback could help with. I think the right click feature in darktable is amazing though and kind of makes this setup ok.
Housekeeping (project management):
Heh, sounds like a boring large company doing that work 
This all started out for me as an exercise in learning the darktable source code and image editing. I don’t think any development of darktable has done anything like what you describe here. I think the next step for this module to really be merged is to get some proper feedback from the core darktable maintainers as they have the final say. They are in the mids of releasing darktable 3.6 so I’m calmly waiting for that to pass while putting more testing into what I have done so far.
Docs are a later project, the project team is basically this thread. One thing that I think had been super nice to have is an open test image database with a wide range of pictures to test on. That would also be helpful for every other image developer out there and be a great community contribution. It might be time though for a summary at the beginning of this thread such that you do not have to spend, let me check, 2 hours! reading through it all!
As for keeping the sigmoid branch up to date! Thanks for the reminder, it is definitely time for a rebase!
I think the example with Lightzone and sharpening is a good example of something that is off-topic for this thread. Sharpening probably needs work in darktable but fixing that is work for a different module and it is independent of any work done here. What would be interesting is how sigmoid compares with Lightzone’s tone curve. Break down the problem to its first principles to borrow a scientific word for it. Don’t worry about sharpening being different between darktable and some other software as at least I will be to tell the difference between the two effects.
The Colors:
Fun! It seems like you are testing on mostly vivid flowers so I’m not surprised that a contrast setting of only 2.7 will be very vivid. I have edited photos where I could crank up the contrast to 3 or even 4 and still have believable results. My feeling is that images have different ranges of usable contrast so you might find that the range you work with will be different with a different subject.
2 Likes
Lightzone I don’t believe has a tone curve…at least in the UI…it uses a module called the zone mapper ie a zone system for tone and contrast…
Short version : TLA, Three Letter Acronym. I like it.
Long version:
The Zone Mapper, Ligthtzone’s equivalent of a tone mapping tool, I found to be remarkably consistent with colours.
What sets it apart, from other tone mapping tools I have used, is the lack of any shift in the hue. All that changes is luminance of different areas of the image, based on what is modified via the controls within the zone mapping tool.
A purist implementation of virtual/digital lighting. Add light or take light away. Simple. Very predictable. No chromatic side effects.
The results might be a bit boring or unusual, for those familiar with the exciting color shifts, that are possible in other tone mappers.
Hint Hint, might be a good candidate for another tone mapping “Proof of Concept” in DT, after sigmoid has been embedded in the official DT roadmap.
darktable has a module with the UI much like this. Its been depreciated.
1 Like
That’s because you likely use it in the default luminosity mode…not going to impact color Nearly as much. Switch to RGB and you’ll get some shifts…Blend filmic in lightness and it doesn’t mess much with color either
This discussion is so in sync with my darktable endeavors. I finally found a module workflow that gives me the results I’ve been fighting to achieve.
From bottom to top:
- color calibration
- color balance (lower saturation to adjust for over saturation in step 5)
- filmic rgb (custom middle gray luminance and using white relative exposure and black relative exposure to produce a super flat image with plenty of room for shadows and highlights)
- rgb curve (local adjustments)
- rgb curve (cubic spline global tone curve with “preserve colors: none”).
- lut 3D
Saturation goes overboard when using “preserve colors: none” in step 5, of course. But I lower it in the color balance module (step 2). Doing it the other way around, using luminosity in rgb curve and upping the saturation in color balance produces super strange results.
I know, it goes against every good scene referred advice to go so hard on the rgb curve (or even rely on this module for a global tone curve), but I need more control over the curve than filmic rgb or color balance with “contrast” and “contrast fulcrum” adjustments gives me. 7 points or more on the global rgb curve is not uncommon. The tone equalizer feels like a curve in a straightjacket, I can’t put it better 
It is sadly too limited because I can’t shift the points on the histogram sideways or create more points. I’ve been dreaming about a scene referred replacement for the rgb tone curve, but I have no idea if that is even possible. Having it in Sigmoid may be what I’m wishing for? 
You beat me to it…
Most or many digital edits are over sharpened and often over contrasted from much of what I see. I think sharpness is another one of these subjective things. Even trusting the displayed image to be the true representation of sharpness might not always be safe as there can be adjustments that affect how it is rendered to the screen. I think RT does a good job by forcing you to go to 1:1 to see the sharpening. I thinks this helps to make it more targeted and reasonable but the second you zoom out you can’t see it so that can take getting used to.
I think a lot of images are actually out of focus and that leads to heavy handed sharpening to try to bring something that it did not originally capture.
In the end its what pleases the eye of the creator that matters
I think sharpness really shows up when you print but its a bit harder to gauge it on the display.
Perhaps the details threshold in masking will benefit DT. Opinions on that should start to appear as more uses get access to it in 3.6.
1 Like
Thank you. This grasshopper (me), has so much to learn. So much.
For contrast and details if you are in DT try local contrast with bilateral at ~150-200% then play with the other two sliders…I think this is an effect more like a jpg too…bit off topic but back to your contrast and sharpness questions…I’ve lost track maybe this has its own thread…
Yep, that might be the most common “heroic rescue” of a flawed image.
@mikae1 would you mind sharing two edits using this approach? Including raw and xmp.
My gut feeling is that step 3 - 6 is your way of creating a very complex tone mapper. Like stacking filmic and base curve plus a lut for the effect that one module should be able to give you. Would be interesting to see if it is a look that I can not reproduce the sigmoid and why in that case.
There will never be RGB curves in the sigmoid module, but I do see the potential for an upgraded rgb curves module that has support for full scene illumination range to complement the stricter tone equalizer. Remember for modules: single responsibility and separation of concerns. A tone mapper like sigmoid or filmic isn’t actually supposed to be able to do that much fine-tuning, we expect the usage of multiple other modules before the tone mapper to do all those special adjustments.
2 Likes