Scene referred, display referred, and the Colour Science of Canon/Fujitsu/Nikon/etc. in Darktable (filmic-rgb module in particular)

[priort (Todd Prior)]

On thing that many people fail to remember in these discussions esp when they go comparing software to something like an Adobe product is that this is open source software written by developers on their own time to suit first and foremost their needs, desires and vision not ours. We have the privilege to tag along for free or participate but we need to remember they are not marketing anything to others. Of course good ideas wherever they come from should always be welcome and considered and there is in fact a lot of code sharing at times where it makes sense.

Darktable has a philosophy to present the raw image with as much control and transparency of process as it can. In doing so out of the gate the image is “really” raw and often unimpressive when compared to commercial product. Then you go to work and craft your edit. The user should know and understand that and move on if that is not "their " expectation.

The current tool set works for you for some things and now you have found something to offset what doesn’t so that’s great. For me the benefit is not there and I will stick one program for the most part as that works for me the majority of the time. I also have Photoflow, Filmulator, RT, ART, Affinity Photo, GIMP ON1 photoraw Photoscape X and Picture Window Pro 8. From time to time I revisit all of these to see what changes have been introduced or if I am curious I try an image here and there but up to this point DT consistently seems to give me the best results in my hands…

[spidermonkey]

I think there is mental barrier for many people that if the starting point doesn’t look quite as nice as their OOC jpg, then they are dealing with a picture that is worse to start off with, so they don’t want to engage with it. There is also the belief that this is what came out of their camera so that is the ‘correct’ image. This is perhaps why commercial software developers invest so much time in making the defaults look like the OOC jpg, even if there are already lots of adjustments behind the scene. darktable developers, understandably, don’t want to spend their time doing this.

Would it be worth having a splash screen for new user installs that says that the starting point won’t look like the OOC jpg, and suggest that they need to adjust contrast, saturation, exposure and local contrast. Having those few controls on the quick adjustment panel too, would mean that they can easily adjust their images to taste.

It might mean there are less comments “darktable is broken, the images look fine in the lighttable but then they look all dull when I open them in darkroom, what is wrong?” It must get tedious for people having to repeatedly explain this.

I must admit, I fail to see what the problem is, after adjusting those few things, my image will always look better than the OOC jpg, after around 30 seconds worth of adjustments. Local contrast plays a big part, I think, to give images that punchy look.

There is also the fact that if somebody has paid for something, they will probably make sure they put more effort into learning how to use it. Can’t do much about that one.

[Thanatomanic (Roel)]

@OK1

What filmic needs is not anothr set of levers presented to the user, but a set of look profiles as presets, similar to base curve, as a starting point.

@OK1

darktable will have to take a back seat, cos the initial image presented by darktable, for very good reasons, deviates quite a bit from the image presented to you as a starting point in any of the proprietary closed source tools. Thats a fact, not debatable.

@OK1 Let’s put that to the test, shall we? I present to you four images: one is the thumbnail embedded in the RAW, one is the default processing from Filmulator, one is the default scene-referred processing from darktable, and one is the default processing from Adobe Photoshop 2020. I will not tell you which one is which, but ask a simple question: which one would you prefer as a starting point for further editing?

Option 1

Option 2

Option 3

Option 4

(edit: source of the image unknown, sorry for the lack of attribution…)

[betazoid (Anna)]

@Thanatomanic

which one would you prefer as a starting point for further editing

That’s a difficult question in this case I think. I can answer a different question more easily: I think the best out-of-the-box result is the last one. But I am not sure if that’s the best starting point for editing.
I am really curious about the solution.

[mikae1]

@Thanatomanic

which one would you prefer as a starting point for further editing?

With zero hesitation, the last one (number 4). There’s a very pleasing roll-off in the highlights and there actually seems to be something to work with there.

[Jade_NL (Jacques)]

@Thanatomanic

which one would you prefer as a starting point for further editing?

I personally like my starting point to be as bare bones as possible, so: Neither of the above. That being said…

If I do have to choose a starting point and the above four are my choices: Number 4.

@Thanatomanic

source of the image unknown, sorry for the lack of attribution

How to highlight the sky

IMG_7079.CR2 (24.0 MB) IMG_7079-art.jpg.out.arp (10.8 KB) [IMG_7079-art] This file is licensed Creative Commons, By-Attribution, Share-Alike.

Posted by @srgmro

[nosle]

Thats an interesting blind test. A couple of different scenes would make it very useful for devs?

I think there is a tendency to forget that huge amounts of effort has gone into “looks” over the years (including film) and that those are cultural not scientific. Our brains read images with the images we’ve seen before as reference. Cooking this from scratch isn’t really feasible for almost any user. Our software always makes some decisions for us but where to draw the line and what look is default is a huge part and draw of our tools.

[anon41087856]

@OK1

But its a democracy, if those in charge of darktable, consider it not to be a good idea, to have similar functionality in darktable, not a problem. We will wait. It’s only a matter of time. I am extremely confident that this idea will sit in the minds of all those involved, and its only a matter of time.

It’s not a democracy, it’s a docraty. Things happen because people make them happen, at the end the decision is up to the ones doing the job because that’s the only reward they will get anyway.

@OK1

Good ideas are good ideas, and it does not matter where they come from. I saw the results of filmulator and its a great idea, to have in a photo development tool. Any photo development tool should have something like this.

Ideas don’t solve problems, they are free and anybody can have them. They don’t even have to be relevant or realistic. Ideas are worthless until they become products. Different software cater to different needs and motivations, the only tool every photo editing app should have is gamut mapping. Trying to please all users at once is only going to displease them all, some of them have irreconciliable needs and expectations, pleasing all sides is going to end up in a convoluted software with too many modes that will be even more confusing.

darktable has an edge on any other software : the ability to affect chromaticity and luminance fully independently and advanced masking options capable of handling HDR with minimal distortion of hues. That gives a lot more freedom than any other soft around. But, yes, freedom is overwhelming when you are used to the leash.

[anon41087856]

@Thanatomanic

which one would you prefer as a starting point for further editing?

3 and 4 have washed sky degrading to cyan and close to no details in the clouds. That’s completely unacceptable.

1, 3, and 4 are over-sharpened.

As a working base, I prefer 2 for its neutrality and lack of opinion.

4 is definitely the Adobe look. I bet 3 is the OOC JPEC, 1 is filmulator, and 2 is darktable.

[OK1]

I will make one final - last ditch attempt to get all of us to see through the haze of sticking to a legacy approach, that does not work for everyone, and I am not alone in saying that we can do better, I am sure there are other voices amongst the darktable user community who share a similar opinion, but may not be as vocal.

1. The purpose of a tool, is to make life easier. Not more difficult.

2. Any additional functionality introduced into darktable will not take anything away from what is already there. Neither does it discredit any of the hardwork that it has taken to bring darktable to where it is today. Change and progress - that is life, and I would have hoped that a community of open source people and open source users, will also be a lot more open minded. I must be frank, the conversation and conservatism in this group from some has been dissappointing. there is a bigger world out there, which darktable can tap into for ideas to become a better tool. darktable’s origins have already borrowed lots of ideas from existing tools like Adobe Lightroom and Photoshop. What’s the problem with borrowing a few more good ideas. None.

3. I have read the conservative clamour for keeping things the way they are, from those who like darktable as it is, no issue, adding new functionality does not take away the legacy you like, just do not clamp down on new ideas that may improve darktable for others like me (and I am sure many others).

4. Aurelien and others speak of a cherished retention of providing “controls” so users can tailor to their hearts content in their own individual way. Implementing new functionality in darktable in a new module, will not take away the control they already have.

5. If you take a proper step back and look at darktable, you find right in the filmic module, and in many other modules, a button or icon, labelled “Auto”. Interestingly the english version of the darktable 3.4 manual mentions the word auto either on its own or in words like automatically - 146 times. That’s what darktable also does, automatically doing so many things for the users, and some of it can be by their choice.

In the filmic module - you have the auto button, so all this clamour about user choice is not taken away, by adding an auto feature. You can have darktable do certain things automatically by a user chosen choice to achieve something. In the case of the filmic module it does some analysis, obviously and then uses its in-built “Colour Science” to make a guess at ideal settings. Other modules in darktable have similar features on buttons or other controls like eye dropper tools, to make things easier. This does not in any way take away from the control features available. Within the filmic module, if you use the Auto function, you still have the ability to go ahead and improve or alter the automatically chosen settings.

So its not one or the other, you can have control and also have assisted “photo intelligence” in the same tools, or modules and the end user who should be the most important customer of the tool has a choice, but also has the option to take advantage of any assistance provided in the tootl - an option - you do not have to use it.

The way it has been managed in filmic is great, you have direct access to the settings automatically achieved by the auto button, andcan still change the results, to what you prefer.

I would suggest the new functionality that provides a ready to use image, be created in another module, so that filmic module development can remain independent according to whatever Aurelien had intended for it.

I suggest that the new functionality to address the concerns I have raised for a more end user friendly image as a starting point in darktable, be implemented as a module somewhat similar in approach to filmic where all the parameters that the tool uses to make an accurate “guess” of the best settings, are available as sliders, buttons, or whatever is the most suitable way of presenting this in the user interface, and all we need is an auto button, which when clicked results in the module which I will call “Polaroid” for want of a better name - but you get the idea - simple one click, and you get a nice image, but all the controls and sliders have been adjusted so you still have control, and much more control, cos you can go ahead and adjust to your hearts content.

Control is not taken away from you, actually it is enhanced, you have direct access to the intelligence behind such wonderful photo “colour science” that can inteligently analyse an image and adjust settings to present you a wonderful starting point. You do not have to use the auto button in this new “Polaroid” module, but its there for those of us who like to use it, same as you find in the filmic module. One click, you get a great image, if that is what you want, and you can then tweak the settings and use the other modules in darktable to complete your creative intention.

I reiterate, we have one good example of this is an open source tool - Filmulator, and if I could code myself, I would do this.

Maybe I need to challenge myself andt learn to code in C or C++ like Aurelien had to also learn, a few years ago, and like him, add this new module to darktable. It would be so much faster though if others who have far more abilities than me, saw the value in this and could achieve this much faster and more completely than I could.

The key 1st step would be to understand the code in Fimlulator and decide which variables are set by the “embedded” photo intelligence and how to expose these variables as controls, so that when the auto function in the new “Polaroid” module is activated, this sets the values of the exposed controls. Of course there could be default settings like in filmic. So you have three options, stick with the default generic settings, click on the Auto button to get enhanced settings of the controls, or start from manual and tweak the defaults to your liking without any assistance by and Auto “photo intelligence - colour science assistant” button. Pretty much like it already is in filmic, just more like what we have in filmulator, with the potential of instant high quality as a bonus.

Certain frustrations led to the development of filmic, and I’m making a genuine case for more. maybe some of the existing or other new developers will also catch my frustration and help out, to add much needed functionaility to darktable.

At the end of the day what distinguishes humanity is our persistent development of tools, and it does not matter who makes them, what’s important is that we move forward. Each of us can choose to play a part in progress or decide not to, its our choice. I would sincerely hope that we can move darktable forward. If I can think of these ideas, I am sure many others are likely to have thought similar thoughts, I only happen to be the one at this time kicking up a fuss about it cos I’ve seen this implemented, and moreso in an open source tool. We can do this and close some much needed gaps, in darktable.

It’s ony a matter of time, someone will add this feature to darktable - to provide instant hight quality starting points, similar to what is achievable in other tools Lightroom, Photoshop, Filmulator, and on Mac you have RAW photo processor which is similar - freware, but closed source(to the best of my knowlege), and we can either participate and contribute to making this happen.

Base curve - great starting point.
Filmic - even better.
“Polaroid” - next step. which will happen one day, its only a matter of time. “Polaroid” will combine the simplicity of Base curve, with some of the kind of user interface of Filmic, to combine simplicity and control, so you have the best of both worlds and above all a high quality auto finished product of a pretty decent image, if you so choose to click that “Auto” button, and still have all the control to modify the auto generated settings.

It will happen, its only a matter of time. Someone will do this.

[Thanatomanic (Roel)]

@OK1 I’m sorry, but despite your eloquence and visions of utopia, you are deadlocking any sort of reasonable discussion. You insist darktable misses the point and is conservative in its approach, but your only argument for this is that the starting point for editing is sub-optimal. You want presets, simple, automagical solutions. People try to explain to you why that is neither their intent nor desirable in general. However, you don’t engage further to understand them and seem as closed-minded about this situation as you claim the others are…

Case in point: you claim Filmulator and other software can prove that things can be better in darktable. Yet, when I provide a blind experiment to test your claims, you (so far) refuse to comment and instead write another 1448 word post rehashing your prior statements. In other contexts this would be considered trolling…

[priort (Todd Prior)]

@OK1

I would suggest the new functionality that provides a ready to use image, be created in another module, so that filmic module development can remain independent according to whatever Aurelien had intended for it.

You are asking for something that can’t be defined. A ready to use image for most professional colorists esp in video is as flat as a pancake. That is where they would like to begin with a mostly blank canvas with the most room to move. Others like yourself may want it more like what they expect the camera to produce. If that is a huge priority then the manufacturers software is the best starting point without all this back and forth. Open it there save a tiff and edit the rest elsewhere or in DT as you see fit or cobble together the tools you want.

You have written hundreds of words without any concrete examples or comparisons so for now all one can conclude is you like the look “you” can get in filmulator and not what you can get in DT and you believe it to be easier.

I believe in another post @CarVac pointed out why currently Filmulator as it functions now would not be compatible with the pipeline used by DT so you keep asking for this module but its not that easy to just introduce it.

You seem to have a vision and if it is that important then I am sure you will find the time and acquire the knowledge to fulfil it and if not you rely on the tools at hand.

[mikae1]

@anon41087856

3 and 4 have washed sky degrading to cyan and close to no details in the clouds. That’s completely unacceptable.

I’m almost shocked by that reply Number 2 has gigantic swaths of lost detail and unrecoverable highlights, while the clouds can easily be salvaged in number 4. Degrading to cyan? That would be nitpicking in this context.

Number 4… Just a simple curve pull will give this:

Number 2 gives me this:

This is not edited for pleasing aesthetics, but rather to show what’s there to work with (after being saved to 8 bit JPEG, of course).

I agree that number 4 is ACR and that number 3 is probably OOC JPEG. I have no experience with Filmulator, so I’ll leave out the rest of the guesses.

[gadolf (Gustavo Adolfo)]

My two cents, as a plain user, much like what @OK1 seems to be.
What you @OK1 don’t seem to understand is what lies behind any of these FOSS software: a community gathered around some interests, or values, that end up being instantiated in a product like darktable.
It’s important that you understand that what you’re really trying to do isn’t change the way the product should be (by means of ideas), but actually trying to change the community’s values, in this case, the darktable community.
I’m not sure if you realize that this is a bit pretentious - like someone who’s invited to a home and start saying how people should be and behave.
I don’t agree with any of the items you outlined above because none of them consider the community and the way it works. Those items clearly describe a commercial vision (making life easier, democracy, users, …).
As people said before, software like this starts from someone trying to produce something the way he/she wants, then gathering more people around the idea, and in the long run, of course there will be changes, but they probably will stick inside the original idea and vision. If a change needs to replace that vision, it’s a clear sign that another community is needed. And that’s the beauty of FOSS: you can fork it.
EDIT: Not to mention that I find it a bit disrespectful to Filmulator’s community: you seem to find it useful (means, it fits your vision), but you still insist in changing darktable. Why don’t you embrace Filmulator once for all and add value to its community, by start editing and contributing with compliments, bug reports, tutorials?

[Thanatomanic (Roel)]

Making literally one change in the GUI for Number 2 gives me this

For Number 4, also one change in the GUI:

[mikae1]

@Thanatomanic

Making literally one change in the GUI for Number 2 gives me this

Hooray, that looks much better! But I assume that change was made from raw data. All I had to make judgements from was a JPEG with lost details.

[paperdigits (Mica)]

Filmulator doesn’t work well in darktable says filmulator’s creator: Filmulator v0.11.0 released! - #26 by CarVac

We are lucky that we have a bunch of different applications that achieve different things. Smushing them all together means we loose that nuance.

[Roland_Rainer]

Isn’t it mostly just a matter of taste and time?
Personally I hate the OOC output of my cameras in about 90% of the cases. But if things have to go quick (n dirty) I have to confess - yes I sometimes simply took the OOC Jpegs and adjusted them a bit with gthumb and done.
Maybe I will take the time to optimize my cameras features and get more out of camera, but if not dt is always there as long as there is a raw.
In most cases I use the OOC Jpegs only to sort out unsharp or completely unusable images and then open my new shot raw images and enjoy all the endless options to mold it into either how I saw, felt or remember the scene, colors, contrast etc. or to create an impressive desirable look. An automatic OOC option for darktable may be desireable for some users - then maybe just open your jpegs with darktable to do the finetuning
The main reason to grab my dslm over the iphone is raw and darktable. When it comes to OOC images the phone is at least as good as the dslm, often better.

[CarVac]

@OK1

Please read my other comments - use filmulator - import any image, see for yourself, and compare with the workflow as we currently have it in darktable.

While I obviously prefer Filmulator’s results myself, there are obviously many who will not. Either you like it or you don’t.

In the dpreview comments many people didn’t see what the big fuss was about—and that’s okay. They don’t have to use it.

But it’s not self-evident that Filmulator is better. You cannot just say “see for yourself” and expect everyone to agree.

There are two confounding factors here, though, given that you’re talking to the darktable community.

One is that people get used to the way their tools render images, so hardcore darktable users (and devs) are not going to be inclined to see other editors’ results as better. I look at others’ people’s edits and sometimes think to myself “Filmulator would do that better”, and the people here might say the opposite.

The other is that the people who use darktable are the people who already liked darktable’s output. People like me who do think Filmulator’s default output is better already don’t use darktable.

[age]

@OK1

if you so choose to click that “Auto” button

Well the auto (based on what?) button is something that needs more discussion.

However i think darktable has got many good features over the last ten years, for me they are:

1. Amaze demosaicing
2. a good wavelet chroma denoiser
3. a good luma denoiser (I prefer the astrophoto denoiser)
4. rgb curves and rgb levels (with preserve chrominance set to no)
5. lch equalizer

There are something very basic missing like:

1. a not bugged highlights recovery module

2. a channel mixer that doesn’t produce ton of artifacts (the colorfulness and brightness sliders shouldn’t be there because they’re not based on how channel mixer works), the gamut compression slider is mostly useless

3. support for DCP profiles and dual illuminant matrices .

4. something like the “rawtherapee’s avoid color shift” after a lab adjustment Lab Adjustments - RawPedia