EXR profile from chromaticities error in darktable? in GIMP?

Elle · February 2, 2018, 10:12pm

This screenshot is from darktable and GIMP-2.9, both from git, both updated today:

It’s a shot of the sky, so maybe not the best example image. But notice the sky in darktable (“1.”) is more blue, and the sky after the image is opened in GIMP-2.9 (“2.”) is more purple. Duplicating the image and then assigning GIMP’s built-in linear sRGB profile (“3.”) makes the sky “more blue” again.

I used GIMP’s “Image/Color Management/Save color profile to file” to save a copy of the embedded profile in the image transferred from darktable to GIMP, and then used “iccToXml” to compare the “from the EXR file profile” XYZ values to the XYZ values found in GIMP’s own built-in sRGB profile (which in turn match the XYZ values in the ArgyllCMS “sRGB.icm” profile).

Here’s the “from the EXR file chromaticities” profile XYZ values:

<XYZType>
  <TagSignature>rXYZ</TagSignature>
  <XYZNumber X="0.38163757" Y="0.19277954" Z="0.00808716"/>
</XYZType>
<XYZType>
  <TagSignature>bXYZ</TagSignature>
  <XYZNumber X="0.15818787" Y="0.07135010" Z="0.73481750"/>
</XYZType>
<XYZType>
  <TagSignature>gXYZ</TagSignature>
  <XYZNumber X="0.42436218" Y="0.73587036" Z="0.08198547"/>
</XYZType>

Here’s the GIMP-2.9 (and ArgyllCMS) profile XYZ values:

<XYZType>
  <TagSignature>rXYZ</TagSignature>
  <XYZNumber X="0.43603516" Y="0.22248840" Z="0.01391602"/>
</XYZType>
<XYZType>
  <TagSignature>bXYZ</TagSignature>
  <XYZNumber X="0.14305115" Y="0.06060791" Z="0.71392822"/>
</XYZType>
<XYZType>
  <TagSignature>gXYZ</TagSignature>
  <XYZNumber X="0.38511658" Y="0.71690369" Z="0.09706116"/>
</XYZType>
<parametricCurveType>

“Linear Rec.709” and “linear sRGB” ICC profiles should be identical, apart from non-functional metadata tags. So something clearly went wrong somewhere either in the chromaticities embedded in the EXR file, or when making the embedded chromaticities into an ICC profile.

So is this a bug in the chromaticities that darktable embeds in the EXR file? Or maybe a bug in how GIMP or perhaps the darktable GIMP plug-in makes an ICC profile from the assigned linear Rec709 profile? Or a user error on my part (always a possibility )?

houz · February 3, 2018, 11:00am

It’s probably either a bug in dt or GIMP. Both sides do strange things, and maybe even both sides are buggy. I’ll have a look.

Elle · February 3, 2018, 12:22pm

@houz - thanks! for looking into this.

houz · February 3, 2018, 4:31pm

It seems darktable is using the correct primaries, however, they are D50 adapted. The problem is that we are still passing D65 to lcms2. Before I push the changes I’d like to ask you to have a look at the four profiles in the attached ZIP file. They now look good to me, but a 2nd set of eyes never hurts.

dt_profiles_d50.zip (10.1 KB)

Elle · February 3, 2018, 5:22pm

Hi @houz - the four profiles all look good.

houz · February 3, 2018, 8:22pm

Thanks. I pushed the change, now GIMP even recognizes the generated profile from the EXR file to be identical with the built in one and no longer asks about it.

Elle · February 3, 2018, 9:01pm

Thanks! I just pulled and compiled etc, and everything looks perfect.

houz · February 13, 2018, 12:10pm

@Elle: Let’s move the discussion about darktable’s color profiles here to not side track the other one even more.

First of all, the reason for darktable to create V2 profiles is so that other software like image viewers can use them. At least a while ago V4 wasn’t widely supported. When selecting an input profile in darktable a V4 profile is used internally.

The only thing I changed in the commit that fixed colors being moved from darktable to GIMP via EXR was the white point, the primaries were the same before. Now, EXR files don’t support ICC profiles. Instead darktable has to put the primaries and the white point into the file. darktable tries to get those numbers from the ICC profile.

github.com

darktable-org/darktable/blob/master/src/imageio/format/exr.cc#L134-L187

    
      
          cmsCIEXYZ *red_color = NULL,
                    *green_color = NULL,
                    *blue_color = NULL;
          cmsHPROFILE out_profile = dt_colorspaces_get_output_profile(imgid, over_type, over_filename)->profile;
          float r[2], g[2], b[2], w[2];
          float sum;
          Imf::Chromaticities chromaticities;
          
          
if(!cmsIsMatrixShaper(out_profile)) goto icc_error;
          
          
red_curve = (cmsToneCurve *)cmsReadTag(out_profile, cmsSigRedTRCTag);
          green_curve = (cmsToneCurve *)cmsReadTag(out_profile, cmsSigGreenTRCTag);
          blue_curve = (cmsToneCurve *)cmsReadTag(out_profile, cmsSigBlueTRCTag);
          
          
red_color = (cmsCIEXYZ *)cmsReadTag(out_profile, cmsSigRedColorantTag);
          green_color = (cmsCIEXYZ *)cmsReadTag(out_profile, cmsSigGreenColorantTag);
          blue_color = (cmsCIEXYZ *)cmsReadTag(out_profile, cmsSigBlueColorantTag);
          
          
if(!red_curve || !green_curve || !blue_curve || !red_color || !green_color || !blue_color)
            goto icc_error;

This file has been truncated. show original

Would you instead suggest to change the white point in the ICC profiles back to D65 and instead hardcode the white point in EXR files to be D50 since the primaries we get from the ICC should always be adapted to that?

Elle · February 13, 2018, 3:59pm

For most use cases including image viewers (do these image viewers allow soft proofing and/or making profile conversions?), users will be using relative colorimetric intent. So whether there is any source white point information in the profile, and where that information is provided in the profile, doesn’t matter from a practical point of view.

Personally for my own profiles, I decided to put the actual source white point in the white point tag for my V2 profiles, for reasons I already covered in the other thread. I think it makes more sense all things considered.

For the EXR files, given that the RGB and white point information is in the Chromaticities tag, then if the D50-adapted RGB values are used, then also the D50 white point information would need to be used. From this perspective the “sRGB adapted to D50” color space is a completely different color space than the “sRGB” color space.

It seems to me that the only other alternative is if everyone assumes that EXR files always contain the unadapted actual color space specs and white point, and then leave it up to GIMP or other ICC profile color-managed software to turn that information into a D50-adapted ICC profile. Does this perspective makes sense? I know a person who actually uses EXR files in a VFX workflow, that I can about this, unless maybe you already know the answer? The only context I’ve ever used EXR files in, is when transferring files between ICC profile color-managed editing applications.

For GIMP, would it simplify things color-management-wise if darktable sent a 32f linear gamma tiff instead of an EXR file?

As an aside, one change that might be worth making to the darktable ICC profiles is to change the copyright from “Public Doman” to “Public Domain/CC0” with perhaps a link to the appropriate CC version page. I was asked by a developer to please make this change to my own profiles because not all countries recognize Public Domain copyrights.

houz · February 14, 2018, 10:28am

True. I will revert that change then.

Well, I would love to put the D65 white point and primaries as stated in the standards (i.e., D65 adapted) into the EXR file, but in the general case of just being handed an ICC profile I don’t see a way to retrieve that information. The primaries I can get from the profile are D50 adapted, and I don’t see a reliable way of knowing the original white point to adapt the primaries for. wtpt can be D50 or D65, depending on the version of the V2 specs the profile follows, and other information might be there or not. Did I ever mention that this is a giant pile of mess?

That would be great, but how would I know what the values are? I don’t see a reliable way to infer that from an ICC profile.

Please do that! The OpenEXR specs are not that clear to me, and a lot seems to be left to the workflows implemented in the studios using the files.

In hindsight: Yes. I might actually do that. But I would still like to have proper EXR support.

Elle · February 15, 2018, 3:20pm

Hi @houz - I did ask about what chromaticities to put in the EXR files that darktable creates, and more generally, what to do when the EXR files are created by and also opened and processed by ICC profile color-managed editing applications.

The short answer was for files created by and for ICC profile color-managed editing applications (like darktable and GIMP), embed the D50 white point and D50-adapted chromaticities. This is the simplest, easiest, least error-prone thing to do.

I also asked about what to do in situations where EXR files are being shared between ICC profile color-managed software, and other software that doesn’t use ICC profile color management (Blender, I think Nuke, and etc). The answer was that different studios use different in-house approaches for handling such situations.

Sample “in house” solutions include (and hopefully I’m summarizing accurately):

Reading/writing the actual CIE chromaticities and white point to the EXR files (I’m assuming “CIE chromaticities and white point” means the actual color space specs values, not the D50-adapted values) , and making appropriate chromatic adaptations as required for different softwares in the processing pipeline, as per this set of BSD-licensed OpenEXR plug-ins: GitHub - fnordware/openexrAE: OpenEXR plug-ins for After Effects. These are the plug-ins that ship with AE.
Using ACES (which has a D60 white point) as the color space for all EXR files, which I assume implies making color space conversions as required for various editing software and input/output devices, because based on reading through forums devoted to ACES it looks like ACES is used for file storage/transport, and actual processing is often/usually/always done in a more appropriate color space.
(a very low-tech approach!) Using file-naming conventions to allow keeping track of what color space a file is in.

So clearly whatever standards there might be, aren’t sufficiently broad in scope to provide uniformly applicable guidelines for all conceivable use cases. And just as clearly, the people who use EXR files in their processing pipelines, feel completely free to design workflows that actually meet their workflow needs.

So for GIMP/darktable/other ICC profile color-managed software, the simplest and easiest thing to do is embed the D50 white point and D50-adapted RGB chromaticities.

If the BSD license is free-software compatible, the openexrAE plug-ins might be another option that might provide for better interoperability with non-ICC-color-managed softwares (“AE” means Adobe AfterEffects). Otherwise to use the original CIE values in the EXR chromaticities tag, someone would have to do a “clean room implementation” (no peeking at the plug-in code) of the plug-in functionality. The plug-in code does use LCMS, so hopefully “clean room implementation” wouldn’t be required. And then of course GIMP would need also to have and use the same plug-in code.

As an aside that isn’t relevant now, but might be eventually, all of the above assumes the ICC profile color-managed editing software is using V2 or V4 ICC profiles and color management software. Someday (probably not any time soon) GIMP/darktable/LCMS/etc might be able to use iccMAX (http://www.color.org/iccmax/index.xalter) color management and profiles, at which point everything will get more complicated.

houz · February 16, 2018, 3:22pm

Thank you an awful lot! I guess I will go the easiest route of hardcoding the white point to D50 and use the primaries as they are stored in the ICC profile – they should always be D50 adapted already.

Regarding the AE plugin, BSD 2 clause is GPL compatible, so we could borrow ideas without any problems. I will have a look at what they do.