Darktable on macos: are there bugs with ICC profiles?

The report was closed considering that it was not a bug. The reason is that the raw file differs from the exported one in tiff, etc.

“It’s not a bug that raw pictures look differently in different editors, they are just processed differently. If you for example export this picture as TIFF and then use it as the test source, then it will look the same everywhere (e.g. in Preview, in dt darkroom view, after dt export), because in case of already processed image like TIFF or JPEG there’s no processing by default. For raw it’s not the case. Closing as not a bug.”

I didn’t even have time to add the answer that the same problem appears with the import of a normal tiff, or jpg. (Although, in the first post I said that the problem occurs not only with raw).

This is a very interesting image.
Same settings, except modern vs legacy chromatic adaptation (camera reference WB + color calibration vs as-shot WB):

BTW, even on Windows, the exported TIFF (imported back into darktable) does not match the darkroom view of the DNG (I used another XMP sidecar for that, though).
_MG_6068_01.dng.xmp (10.6 KB)

I’ll have to look again…I didn’t see that …they looked the same to me when I did it. I just looked at the meta data in the OP tiff file and the embedded xmp seemed to indicate that filmic etc were turned off. Indeed when I did that the images matched from what I could see as his darkroom view had the scene defaults applied… The DR display with those modules disabled match the output of the provided TIFF as least on my system… I will go back and look. I hadn’t tried the different WB setting as I was trying not to change things from the OP…I will have to go back and look at that… I did an export setting it to export with and srgb profile and adobe and re imported and on my system the all the images matched the preview state of the raw… so again I will have to go back it was likely late at night when I was doing it…

It’s all very cool, of course. But which xmp did you use? )) If I delete the picture a001.tiff (conditional name example) from the image library (lighttable), but leave the files themselves (tiff, xmp), and reload the picture a001.tiff into the program, I will see such a trash)) that it is better not to look at it. It will be very light gamma. As if we had set the gamma to 1 instead of 2.2 for some picture. ))
If I delete the xmp file and re-import it after, then everything kind of looks more sane.

I used the XMP I posted above.
(edited: quoted original question)

Why I noticed the gamma shift issue when I started using this program for the first time in my life.
I liked some of the tools in it that Davinci Resolve doesn’t have. When you change hue/sat (color correction) and others. But I work with video sometimes and I needed some way to get the 3dlut files out as a result.
But this program doesn’t know how to export such options. But I remember doing the same thing with Lightroom - I took and applied effects to hald files, and converted the hald files themselves into 3dlut files and everything was fine. Considering that I also converted the gamma of the processed photo (tiff image from Davinci Resolve), because lightroom didn’t know what gamma some Davinci had. Gamma and color space. and then I imported into Adobe Lightroom, processed such an image and applied the effect to the hald file, and so on.

Such a conversion profile is not difficult to create and apply. I took a file, opened it in Photoshop and converted it with my profile. That is, I didn’t put it in Lightroom or anything else. So that the programs would open the photo as usual.
I did the same with Darkroom.
But! When I copied the created style of the program for hald and saved such a hald, I saw a lot of artifacts, when I already converted hald to 3dlut and applied such 3dlut to video in Davinci Resolve.
And then I found out that darktable applies some kind of sharpening to all the images and whatever else is out there.
I turned those effects off, leaving only what I needed. So I kept the style and applied it again.
It worked out great and more! - There was no gamma shift, but! I saw a staircase. As in gradients. staircase on objects where there are smooth transitions of light and shade.
When applying similar 3dlut files created in 3dlut Creator, or in Lightroom (hald > 3dlut with 3dlut creator) I never saw such treshold effects.
That’s when it became easy for me to save a normal image from the darkroom, not a hald. To make sure that this program saves everything correctly.
Seeing the problem, I started googling various resources. I came across this topic and decided to bring it up.

In my hands I am seeing none of the issues except for sure it seems the legacy and modern WB are not the same as you noted and this has been noted before with the discussions around those D65 numbers for certain cameras…for me I have seen this before…its mostly in the highlights from what I can tell…if you place a bunch of pickers on the image and also look as you toggle back and forth between legacy and modern the image gets a bit brighter…not a ton but some. As for the effects seen on the Mac by the OP. I don’t see them and I just redid both photos that have been shared…mind you keeping it simple to look at the data …so no processing …no filmic no base curve nothing and legacy wb as shot…and when I export the image and bring it back in it looks the same. A set of color pickers placed around the image show mostly identical values…and I don’t see any case of extreme lightening mentioned by the OP.

I also don’t see any sign of the issue that was noted about removing and reimporting a file.

Going back to the original post it was clear that the provided tiff had been saved with different settings than we present in the darkroom preview used for comparison…that might have just been an oversight and the issue may be real on a mac…I am just not seeing any of this

I must admit its too early in the morning for me to follow that last post about all the lut conversions back and forth etc etc…

If you have issues with your monitor profile, the preview would look different / weird / wrong.

But loading a tiff file and not doing anything to it and saving it again in sRGB should look (almost) like the input.

Make sure you do NOT touch input profile module (leave the working profile to linear rec2020).

The output is only used if you want your files to be in a certain profile , but generally this should be sRGB.

All the soft proof settings should also be left at default to get started.

What i know from macos, is that the os is doing the color management for apps , unless an app really requests 'i know what i am doing , let me draw my own pixels '.

So just to get started , you should not touch any setting (and for sure not set anything to adobergb , because that has no place except for an output profile ).

Open a tiff file , that has an ICC profile embedded in it (and for giggles , make sure it is not a CMYK profile ). Make sure all Darktable modules are disabled that can be disabled .
In the export , save it as a jpg , set it to sRGB output and relative mode . Export it , compare it to the input tiff in a color managed program (like Photoshop ). They should be similar (gamut limits and such not accounted for right now ).

Note that i completely ignore how the file looks inside of Darktable. The input and output are being checked .

If you have a monitor calibration profile and have it set in macos settings , you should not set it anywhere in Darktable is my guess (but I’m not 100%) on this.

Of that input - output check is being done correctly and is NOT ok, it looks like therenis a bug with the build you are using. Darktable’s color management is done with a library internal to Darktable (lcms2, not using macos libraries AFAIK) so it should work the same as any other build using lcms2.

What i learned from using a 2017 Macbook pro , is that of a program is not pretending to be color managed , macos interprets it as being sRGB, so macos will take whatever the program is drawing , and interprets it as sRGB and makes sure it looks ok on the profile in the OS display settings.

Now, if a program advertises itself to macos as being color managed , it gets more choices and can draw in bigger gamuts than sRGB for instance , but it has to do a lot by itself suddenly . I guess this would mean you would have to select the monitor profile somewhere in Darktable. Note, this is monitor profile . NOT soft proof profile , not input profile , not output profile , not working profile .

Also note that i can’t imagine you have to select adobergb anywhere in monitor profiles or soft proof profiles. The only use case would be if your monitor has an adobergb mode you are using , and you have NO monitor profile selected on system settings.
The moment you profile/calibrate your monitor , you select that monitor profile, not Adobe RGB or something.
You also don’t select it as working profile , be a use Darktable usually uses linear rec2020 as working profile which is fine. You don’t use it as an input profile , unless you know the input file is adobergb but it has NO adobergb profile embedded. Of this is the case , comparing in other apps would go wrong anyway since they don’t know the input is adobergb and would interpret it wrong. And you don’t select it as output profile unless you want your output file to be in adobergb. And you would know if this would be required / desired for your use case.

If you see a shift in the histogram in Darktable, I’m guessing something in the Darktable pipeline is still changing the values or pixels. On the other hand , still confused about what the histogram is displaying in Darktable (i would suspect ‘working profile’ but there have been bugs and I have been wrong before , so I’m absolutely not sure what it is doing these days ). If it’s displaying your input file in the working profile , then yes, it would look different to what you expect , because it is converting the input to linear rec2020.

The preview on your screen is rendered from this linear rec2020 to whatever Darktable is using for drawing. ND if you export , that linear rec2020 is converted to the output profile (either set in the output profile module per file , or overridden in the export settings ).

It’s also not really clear to me what you are comparing with preview.app (the built-in viewer in macos , for the uninitiated). Input tiff vs exported tiff / jpg? Your monitor profile / soft proof settings / macos settings should have no effect.

If it’s raw file as input vs what you get from Darktable as export , then yes , that would be different , as preview.app uses a completely different pipeline for displaying raw data (a raw file has no 'true, correct look. Only interpretations).

If you are comparing a tiff input file to how it looks inside Darktable, then maybe there is more to discover . If you are comparing how it looks in Darktable vs how the exported file looks in other software , then also there might be more to discover ).

Yes, but this time the only difference is the highlight handling. Do you think that is due to the profile?

Here is a comparison of the darkroom view of a DNG and the TIFF exported from darktable, reimported into darktable (sorry about the crazy XMP):

(Check the arm/shoulder area.)

I’ll see if I can reproduce this on today’s master build on Linux.

Its hard to know and I think it has been further tweaked a bit over the last few days wrt highlights and some math…check the last few PR from yesterday/today…I have not built with those changes yet…

If I keep it simple and use legacy WB and no other editing I don’t see much of an issue with the exported Tiff I will look one more time… I tried to compare clipped highlights vs LCH that you had and I didn’t see much either …I will look at the region you mention… I used pickers and compared a snapshot of the raw with the exported and reimported tiff for my comparison above and there overall I did not notice much difference…certainly not a big global change in the brightness of the image which I believe is the issue for the OP with this set of images?? Its hard to say when the preview could be impacted by the display profile which could be a bit different for everyone and even in the specific OS if not setup correctly I guess??

Speaking of gamma, at one point, Apple/Mac displays had a different gamma. It messed up my prints because I would forget to compensate for the differences. They eventually standardized. Probably not the OP or @EsTaF’s issue, but putting it out there just in case.

Here is a split screen…so basically the image with only default filmic and remove exposure.

image
_MG_6068_01.dng.xmp (6.9 KB)

Left (snapshot) is DNG preview… Right is exported TIFF re-imported

No really major differences that I can see…

Edit:

However modern WB…not so good… and default filmic norm makes it worse…

Setting it to no is better

If you have issues with your monitor profile, the preview would look different / weird / wrong.

But loading a tiff file and not doing anything to it and saving it again in sRGB should look (almost) like the input.

There is another problem here. What will the original image look like in darktable and in another program that supports cms. I will see the difference already at this step. And so yes - if I open the source in preview.app and the exported file from darktable, which used to be the source, I also open in preview.app, then the source and the resulting file will not be different.

!= - gamut inequality
src.tiff+preview.app = output.tiff+preview.app;
src.tiff+preview.app != src.tiff+darktable.app;
src.tiff+darktable.app; != output.tiff+preview.app;
))

Make sure you do NOT touch input profile module (leave the working profile to linear rec2020).
The output is only used if you want your files to be in a certain profile , but generally this should be sRGB.
All the soft proof settings should also be left at default to get started.

  1. rm -rf ~/.config/darktable
  2. run darktable.app
  3. Open src.tiff in darktable
    ok :slight_smile:

What i know from macos, is that the os is doing the color management for apps , unless an app really requests 'i know what i am doing , let me draw my own pixels '.
So just to get started , you should not touch any setting (and for sure not set anything to adobergb , because that has no place except for an output profile ).

ok

Open a tiff file , that has an ICC profile embedded in it (and for giggles , make sure it is not a CMYK profile ). Make sure all Darktable modules are disabled that can be disabled .
In the export , save it as a jpg , set it to sRGB output and relative mode . Export it , compare it to the input tiff in a color managed program (like Photoshop ). They should be similar (gamut limits and such not accounted for right now ).

src.tiff+preview.app = output.tiff+preview.app;
src.tiff+preview.app != src.tiff+darktable.app;
src.tiff+darktable.app; != output.tiff+preview.app;
))

They should be similar (gamut limits and such not accounted for right now ).

Stop. In what ways should they be similar?
Gamma is both saturation and light.
Hue? Hue is similar. ok.

Note that i completely ignore how the file looks inside of Darktable. The input and output are being checked .

If you have a monitor calibration profile and have it set in macos settings , you should not set it anywhere in Darktable is my guess (but I’m not 100%) on this.

Most likely, darktable is friendly with cms mac os. Otherwise the topic would have been raised not only by me. If you google the question, it has been poorly raised, and the program (mac os port) is many years old.

Of that input - output check is being done correctly and is NOT ok, it looks like therenis a bug with the build you are using. Darktable’s color management is done with a library internal to Darktable (lcms2, not using macos libraries AFAIK) so it should work the same as any other build using lcms2.

ok

What i learned from using a 2017 Macbook pro , is that of a program is not pretending to be color managed , macos interprets it as being sRGB, so macos will take whatever the program is drawing , and interprets it as sRGB and makes sure it looks ok on the profile in the OS display settings.

Now, if a program advertises itself to macos as being color managed , it gets more choices and can draw in bigger gamuts than sRGB for instance , but it has to do a lot by itself suddenly . I guess this would mean you would have to select the monitor profile somewhere in Darktable. Note, this is monitor profile . NOT soft proof profile , not input profile , not output profile , not working profile .

Similar situation. the topic would have been raised by the community. it wasn’t raised, which means everything is fine here.
I don’t focus on everyone always, but in this case I need to be clear about the situation and analyze anything and everything. Otherwise I will go far into some wilderness.

Also note that i can’t imagine you have to select adobergb anywhere in monitor profiles or soft proof profiles. The only use case would be if your monitor has an adobergb mode you are using , and you have NO monitor profile selected on system settings.
The moment you profile/calibrate your monitor , you select that monitor profile, not Adobe RGB or something.
You also don’t select it as working profile , be a use Darktable usually uses linear rec2020 as working profile which is fine. You don’t use it as an input profile , unless you know the input file is adobergb but it has NO adobergb profile embedded. Of this is the case , comparing in other apps would go wrong anyway since they don’t know the input is adobergb and would interpret it wrong. And you don’t select it as output profile unless you want your output file to be in adobergb. And you would know if this would be required / desired for your use case.

Yes, that’s right. darktable uses rec2020 as input.
And yes - let’s localize the question without adobergb, otherwise I’m not the only one who dreams about it )) Just kidding) but we need to simplify the whole topic. To a sane state, of course.

If you see a shift in the histogram in Darktable, I’m guessing something in the Darktable pipeline is still changing the values or pixels. On the other hand , still confused about what the histogram is displaying in Darktable (i would suspect ‘working profile’ but there have been bugs and I have been wrong before , so I’m absolutely not sure what it is doing these days ). If it’s displaying your input file in the working profile , then yes, it would look different to what you expect , because it is converting the input to linear rec2020.

In my opinion, a histogram is not just some waves in a square. Right, wrong, and so on. But it is also a living part of the program system, working with the image as a whole. Everything has to be right here.
See. About rec2020 and others.
What does the input module do in the tools? Right, it interprets the picture and tells the program what kind of picture the program is dealing with. The program, in turn, shows us the picture as it is exposed in the input profile.
So what should we see in the histogram window? Well, the histogram, which represents what we see in the viewport.
That’s if we take the normal cases. Clearly, things may be different and we may need to see other options in the histogram as well.
But, imho, here is the first option should be implemented in the program by default. this is the case in davinci Resolve and in photoshop and 3dlut creator, as far as I remember.

The preview on your screen is rendered from this linear rec2020 to whatever Darktable is using for drawing. ND if you export , that linear rec2020 is converted to the output profile (either set in the output profile module per file , or overridden in the export settings ).

!= - gamut inequality
src.tiff+preview.app = output.tiff+preview.app;
src.tiff+preview.app != src.tiff+darktable.app;
src.tiff+darktable.app; != output.tiff+preview.app;

Again, the problem grew out of trying to copy darkroom processing settings to other programs.

  1. It wouldn’t be a bad idea to throw all the raw stuff aside. We need to localize the problem, in my opinion. I don’t want to point anything out to anyone, but if we experiment with raw now, it will complicate the topic ;(
    Leave only tiff in the work. Sources, samples received.
    Please ;(

(I will try to record a video on the subject of exporting processing settings from darktable to davinci resolve. About why all this was started in the first place).

I think you’d be surprised how little macOS users there are for Darktable, let alone technical enough to dive into ICC / gamma / histogram issues. Still, if there was a bug in the code I agree, it would probably have been talked abou before.

:+1:. Short answer: What is your problem, Darktable’s output file compared to an input file? Or Darktable’s output file (in other programs) compared to how it looks inside Darktable itself? Can you share your input file for testing, so we can follow along? (And see if your Darktable version behaves the same compared to others?).

Well… Let’s think about a regular sRGB monitor to keep things simple. What are we seeing? sRGB data. So gamma corrected RGB. But what data are we working with and are the calculations being done on? Linear rec2020. So what would you expect in the histogram? I would expect working data there, not display data (as in, the histogram should look the same on every computer you open it, independent of the display device used for previewing it). input data should not be there, because we are not working with it at all (it gets converted to working profile pretty early on).
Again, I have no clue what Darktable is actually showing in the histogram.

Short answer: What is your problem, Darktable’s output file compared to an input file? Or Darktable’s output file (in other programs) compared to how it looks inside Darktable itself? Can you share your input file for testing, so we can follow along? (And see if your Darktable version behaves the same compared to others?).

Yes, I can.

archive.zip (2.5 MB)

Well… Let’s think about a regular sRGB monitor to keep things simple. What are we seeing? sRGB data. So gamma corrected RGB. But what data are we working with and are the calculations being done on? Linear rec2020. So what would you expect in the histogram? I would expect working data there, not display data (as in, the histogram should look the same on every computer you open it, independent of the display device used for previewing it). input data should not be there, because we are not working with it at all (it gets converted to working profile pretty early on).
Again, I have no clue what Darktable is actually showing in the histogram.

Well the profile of the display in the histogram should not be taken into account. Of course. Maybe I wrote something wrong earlier.

https://youtu.be/jNpeRUOAbUo

Exporting color correction settings from darktable to davinci.
Thought, now I’ll take all such a handsome guy )) and show artifacts (staircase/searching treshold in the light transitions). But it’s not there and that’s it.

ahh.

You are using Darktable to apply some color-changes to a test-image. You save that as a style, then apply that style to a Hald-clut reference image, and export it from Darktable. So you created your own hald-clut. You use some other program (‘LUT Generator’) to convert the hald-clut to an .cube 3dlut.

When applying that .cube file in Davinci resolve (to the same input test image), you see an unexpected huge shift in what appears to be gamma (let’s just say, the image appears too bright and saturation / colors are off).

So this has nothing to do with monitor / icc calibrations in Darktable :slight_smile: .

It could be there are color-space mismatches between your test-image (tiff) and the hald reference/output image.

But it could also be a problem of LUT Generator, or it could be a problem for Danvinci Resolve how it applies the 3dlut .cube file (maybe it expects the .cube file to be for sRGB files so it applies some transforms?).

I don’t know if your darktable has the hald clut module if it’s still around? You could try applying the generated .png in Darktable itself to see if that works there. Or maybe try Rawtherapee to apply it to your test-image (tiff) to see if it works there.
Or use imagemagick (magick input.tif generated-hald.png -hald-clut output.tif) to test the hald-clut.

Try testing the generated hald png first to see if that one is working correct in other programs, that support hald luts. If that works, that means the problem sits somewhere in LUT-Generator and/or Davinci.