Confusion on color profiles

Ok I have a “theory”.
Please see the images attached.
It’s a 100%zoom comparison of the same file with different profiles.

1)One the left hand the DT with sRGB display profile selected; on the other LR, which, in theory, leaves no choice of option as it works with either AdobeRGB98 (or even ProPhoto). Colours are indeed very similar (not saying identical of course).

  1. Same comparison, but on DT I set AdobeRGB98 - very different Colors than LR

Now, I mentioned earlier that my “system profile” options is identical to the sRGB; my conclusion: it could be that indeed the “system profile option” is the right one (being AdobeRGB98), and that the sRGB is somehow failing to reproduce.
Otherwise how could you explain such a similarity (surely not identical, of course) in both programs?

While if I put on DT the profile AdobeRGB98 as “display option” I get totally different results than in LR.
More lost than ever.


But it does seem sad that for years now this problem has been known, and no-one other than @Carmelo_DrRaw seems to have come up with any solution, and that solution is a not an actual solution but a workaround.

This bug report says “something” was fixed (the bug was closed as fixed) , I wonder what that “something” was, as it apparently didn’t actually fix the real problem: https://bugzilla.gnome.org/show_bug.cgi?id=681784

From skimming through the bug report, it appears that Gtk/Cairo is pretty ignorant of Color Management, and on dealing with OS X has stumbled into problems. My conclusion is that there are three ways this could be dealt with in increasing level of difficulty:

  1. Make Gtk/Cairo render everything in native device space, which (I gather) would make it behave the same as the MSWindows and X back ends. You need to use the null transform trick to do this with recent OS X releases. Down side is that all the UI elements will be garish on wide gamut displays.

  2. Gtk/Cairo needs to distinguish between UI elements it is rendering and values that the application is handling. For the former it could use sRGB as the tag, and for the latter the application is expected to do the color management and Gtk/Cairo use the null transform tag with OS X for those elements.

  3. Gtk/Cairo support color managed back ends properly, by supplying the API’s to let the application tag all the color elements. That would mean implementing color management for the X back end, and using the MS CMM machinery to manage color for the MSWindows back end. The OS X back end would be pretty easy. Wide gamut displays are then fully supported.

(1) Would seem the most trivial to implement, but it could be that it’s not easy because Gtk/Cairo is hiding what display various things are on, and it is necessary to know which display is which when the application is doing color management or the null transform trick is used. This would be an issue if OS X allows rendering elements across multiple displays with a single call, and Gtk/Cairo is relying on such behavior rather than breaking up the rendering into separate calls for each display. (I wouldn’t have thought that was the case though, since in general you can’t rely on a back end supporting such a thing.)

[ Note that Wayland is in the “extremely difficult” basket to implement Color Management on. It basically can’t be done without introducing a color management extension, because of a fundamental blunder in the underlying assumptions made in Waylands design. (Lack of awareness of Color Management amongst graphics developers bites big-time!) And the Wayland developers seem to be actively hostile to introducing a Color Management extension. ]

2 Likes

Big reply to everyone in a single post ahead.

Well, kind of. The pipeline looks roughly like this:

For raw files, the data is in the native format (bayer, X-Trans, …) up to demosaic, after that it’s in camera rgb. Regular images are in their original color space (sRGB, AdobeRGB, XYZ, Lab, …).

In input color profile all colors are transformed to Lab and stay that way up till output color profile. There the image is converted to either the display profile or the export profile, depending on which pipe is running. In the very end the data is then converted from float to whatever the output medium needs.

So there are basically three “working profiles”, the one used for most modules is Lab, and the other two depend on the file and output. There are plans to replace the final parts of the processing with a fixed RGB profile, most likely linear Rec.2020, and then a final conversion to the output profile, but that’s just an idea so far.

That is what cairo currently tries to do and fails with on OSX since Apple broke their API.

2 and 3 are kind of the same, and are definitely what needs to happen some day. At least once

gets resolved. By then GTK+ needs a way to tag widgets with a color profile or alternatively tell it to leave areas of the screen unmanaged. But I don’t expect that to be a pressing issue for years, given how Wayland people deal with this.

That would be awesome. As a first step, could you try using CGColorSpaceCreateDeviceRGB() and then copyICCData() on the result and share the ICC profile with us? Please make sure that you have some display profile configured system wide that is NOT sRGB first so we can actually distinguish it.

This is exactly the thing I was setting up yesterday… I have run the following code snippet, with the display profile set to some wide-gamut colorspace in the preferences:
02


  CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
  if( !colorSpace ) {
    std::cout<<"Cannot get CGColorSpaceCreateDeviceRGB()"<<std::endl;
  } else {
    std::cout<<"Display profile: "<<CFStringGetCStringPtr(CGColorSpaceCopyName(colorSpace),kCFStringEncodingASCII)<<std::endl;
    CFDataRef data = CGColorSpaceCopyICCProfile(colorSpace);
    if( !data ) {
      std::cout<<"Display profile: cannot get CGColorSpaceCopyICCProfile()"<<std::endl;
    } else {
      CFIndex icc_length = CFDataGetLength(data);
      const UInt8* icc_data = CFDataGetBytePtr(data);
      cmsHPROFILE icc_profile = cmsOpenProfileFromMem( icc_data, icc_length );
      char tstr[1024];
      cmsGetProfileInfoASCII(icc_profile, cmsInfoDescription, "en", "US", tstr, 1024);
      std::cout<<"Display profile: "<<tstr<<std::endl;
    }
  }

  colorSpace = CGColorSpaceCreateWithName(kCGColorSpaceSRGB);
  if( !colorSpace ) {
    std::cout<<"Cannot get CGColorSpaceCreateWithName(kCGColorSpaceSRGB)"<<std::endl;
  } else {
    std::cout<<"sRGB profile: "<<CFStringGetCStringPtr(CGColorSpaceCopyName(colorSpace),kCFStringEncodingASCII)<<std::endl;
    CFDataRef data = CGColorSpaceCopyICCProfile(colorSpace);
    if( !data ) {
      std::cout<<"Cannot get CGColorSpaceCopyICCProfile()"<<std::endl;
    } else {
      CFIndex icc_length = CFDataGetLength(data);
      const UInt8* icc_data = CFDataGetBytePtr(data);
      cmsHPROFILE icc_profile = cmsOpenProfileFromMem( icc_data, icc_length );
      char tstr[1024];
      cmsGetProfileInfoASCII(icc_profile, cmsInfoDescription, "en", "US", tstr, 1024);
      std::cout<<"sRGB profile: ICC name = "<<tstr<<std::endl;
    }
  }

The terminal output from the above code is quite interesting:

Display profile: kCGColorSpaceDeviceRGB
Display profile: cannot get CGColorSpaceCopyICCProfile()
sRGB profile: kCGColorSpaceSRGB
sRGB profile: ICC name = sRGB IEC61966-2.1

To cut it short, the profile retrieved with CGColorSpaceCreateDeviceRGB() DOES NOT HAVE any associated ICC data, while the one retrieved with CGColorSpaceCreateWithName(kCGColorSpaceSRGB) corresponds to a standard sRGB ICC profile.

Also, quoting the Apple documentation:

Device color spaces are primarily used by iOS applications because other options 
are not available. In most cases, a Mac OS X application should use a generic 
color space instead of creating a device color space. 
However, some Quartz routines expect images with a device color space. 
For example, if you call CGImageCreateWithMask and specify an image as the mask, 
the image must be defined with the device gray color space.

Nowhere I see mentioned that CGColorSpaceCreateDeviceRGB() should return the display profile, and from what I observe it seems to be a very bad choice for handling color management stuff…

It’s not clear to me that Apple have broken their API so much as moved the ground under it. Rather than being able to set the native space using colorWithDeviceRGB etc., the null transform trick is now needed for non-primary displays. (See ArgyllCMS/spectro/dispwin.c variable nscs for how I’m handling it.)

Just to clarify, if the only processing steps done with darktable before exporting the image to disk are these:

  • demosaic
  • hot pixels
  • chromatic aberrations
  • white balance
  • raw black/white point

and if the input and output profile are the same (in my own case, a custom camera input profile) . . .

. . . then is there any forced conversion to Lab or to an RGB working space not selected by the user?

darktable passes my “very yellow flower” raw file test - there doesn’t appear to be any clipping in the blue channel from any forced color space conversions that might one way or another impose gamut clipping. Of course this doesn’t mean there aren’t forced non-gamut-clipping conversions to Lab or some internal RGB working space.

Maybe I should have said darktable doesn’t do any automatic non-user-selected conversions to internal working spaces? I’m guessing darktable doesn’t convert to Lab unless/until the user selects an operation that’s done in the Lab color space?

Yes, the Lab conversion is always done. That might change in the future but don’t count on it.

Sorry but I cannot stop thinking how funny is that I got into this forum as a new DT user with a “simple” doubt on profiles, and look at what this thread has generated.

Now, if at some point you could “translate” all the (undoubtedly necessary) technicalities would be nice, so that “simple” DT users can benefit of eventual upgrades in terms of color management and similar.

Thanks to you all.

3 Likes

Hi,

TL;DR, as I understand it (again, if I’m wrong I’m sure someone will correct me): use “system profile”, and you will see the “proper” colours, albeit clipped to the sRGB gamut. Importantly, this clipping only affects what you see when viewing the image from inside darktable. No clipping will occur on the output image. So, if you open the output image in another program that doesn’t suffer from the Cairo limitation on OSX (Photoshop should work, I suppose), you should see all the colours that your monitor is capable of displaying.

When the situation changes, you will definitely see this in the release notes of darktable and all other image processors that are affected by this.

2 Likes

Thanks.
I think I understood that. In fact as mentioned, at least in my case, “system profile” and “sRGB” are totally identical. Makes sense.
However, I dislike the idea of not getting the most out of my monitor, because it means falling back to sRGB. And I am sure many, many others of OSX with wide gamut display will feel exactly the same.

See you soon then.

Yes, it will be mentioned. And even then the system display profile setting should still be the right thing to use.

I’ve been using ArgyllCMS to “accomplish this” and “test that” for as long as I’ve been using Linux (since somewhere around 2007), and so far ArgyllCMS is the one and only color-management-related software that seems to me to be 100% reliable. And so I have a huge respect for anything @gwgill says about anything related to color management.

Color management on X works. I don’t mean that non-color-managed desktop widgets look good on a wide gamut display. I don’t mean that multi-monitor setups are easy to use. I don’t mean color management happens “automagically and correctly” without the user having some knowledge of what they are doing. I only mean two things:

  • When using ICC profile color-managed editing applications (GIMP/Krita/free-libre raw processors/etc), on X the images are shown with correct colors, at least on the main monitor, assuming of course the user knows how to calibrate and profile their monitor, how to load the vcgt tag if required, and how to tell the software to use the monitor profile.

  • When using a proper OCIO lut made from one’s properly made monitor profile and the profile of whatever working space one wants to use, on X the images shown by Blender and presumably other OCIO-color-managed applications have correct colors.

In case it might be useful to members of the pixls.us forum, here are links to various discussions of color management over on the Wayland development mailing list, more or less in order by time:

This page has a box for searching the wayland-devel archives:
https://lists.freedesktop.org/archives/wayland-devel/

A lot of the Wayland discussions go right over my head. But if the people developing Wayland don’t actually understand (or care about?) color management as it pertains to color-managed image editing, that’s a huge problem.

As the conversion to Lab doesn’t seem to clip colors, then of course that’s not a problem. From curiosity, does darktable use LCMS to make the conversion?

It depends. When the input/output profile is a matrix profile we use our own fast path, otherwise lcms2 is used. You can force darktable to always use lcms2 in the preferences though.

As promised, I will summarize here my findings after experimenting a bit with the color profiles in OSX.

First of all, here is some explanation of the logic being used by Cairo under OSX.
When painting pixels on the screen, Cairo uses the Quartz 2D backend. Internally, the pixel data is associated to a CGImage structure, and is transferred on screen by drawing the image into a CGContext associated with the application window.

The Quartz 2D graphics library is intrinsically color managed. The destination graphics context is associated with the display profile, and the input CGImage must also define the profile needed to interpret the pixel data. The library takes care automatically of the colorspace conversions when the image is drawn into the graphics context.

From the point of view of a photo editing application that wants to keep control of the colorspace conversions, this is of course not convenient, particularly because the Apple API does not provide any way to disable Quartz internal color management (probably they consider the software developers not smart enough to take care of color management :wink: ). The only way I could find to “bypass” the color management is to assign the current display to the CGImage structure, so that Quartz applies an identity transform (it seems that in this case Quartz does not apply any transform).

As far as I understand, the “identity transform trick” was the original goal of the Cairo developers as well, but was accomplished in a way that is not working anymore in recent OSX versions. For this, they associate to the CGImage a colorspace obtained with the CGColorSpaceCreateDeviceRGB() function. However, from my experimentation as well as from some Qt bug report it looks like DeviceRGB is treated internally as sRGB by Quartz, hence resulting is an unwanted color transform.

The best working solution I could find is to replace CGColorSpaceCreateDeviceRGB() by the following function in CairoQuartzCreateCGImage():

static CGColorSpaceRef CGColorSpaceCreateDisplayRGB()
{
  CGColorSpaceRef result = NULL;
  result = CGDisplayCopyColorSpace(CGMainDisplayID());
  if (!result) {
    result = CGColorSpaceCreateDeviceRGB();
  }
  return result;
}

The retrieved monitor profile corresponds to the “main display”. Hence, this works correctly in a single-display setup as well as if the application window is on the main display in a multi-display configuration.

In order to support displays other that the main one, one would need a mechanism for passing the display ID to the underlying Cairo library. Unfortunately, the Quartz API does not provide any interface for retrieving the color profile associated to the destination CGContext…

I am currently preparing a PhotoFlow package with this patch included, so that eventually other OSX users with wide-gamut displays can give it a try and see if they get the expected result.

2 Likes

Oh my, I actually understood this. No wonder my wife’s eyes just glaze over when I use the word ‘color’ in a sentence… :smile:

3 Likes

@Matteo_Bertolino @Morgan_Hardwood @houz - I have prepared a new version of my PhotoFlow editor that is linked with a Cairo library patched as I suggested above. You might use it to test the visualization of wide-gamut images on your wide-gamut display, without the limitations imposed by the standard Cairo code. For this, you have to set by hand your display profile in PhotoFlow’s preferences:

The updated photoflow package is here.

The reason why I am proposing to use PhF here is simply because I have full control over the build process, and I can easily do some testing. This exercise should serve as a proof-of-principle for a possible patch to be proposed to the Cairo developers, so that the sRGB limitation under OSX might be solved for all GTK-based image editors…

Let me know what do you think.

4 Likes

Hi.
Well, first of all, many thanks for your work. That’s good news indeed.
Now, I am not sure I will be the best person to run tests with…anyway, I’ll give it a try.
First I need to install Photoflow, which I come to know the existence of today. Once installed I’ll get back here, probably for further guidance.
I’m sure the others will be able to help out a lot more.

ok. installed.
Now, which profile shall I select from the .icc folder? My own, as produced by the calibration process? Actually in that folder I only have my own created .icc profiles. No generic AdobeRGB98. I wonder where all the others have gone.

@Matteo_Bertolino - Yes, you have to select the ICC profile you created for your display. Other profiles can be found in /Library/ColorSync/Profiles and the relative sub-folders, as well as /System/Library/ColorSync/Profiles.

Thanks for checking!