Wayland color management

Just saw the following on the wayland developers mailinglist HDR support in Wayland/Weston and that reminded me that I always wanted to make a topic here to discuss color management on wayland, especially since the proposal from the wayland devs itself[1] is lacking in a few aspects, namely

  • It is missing profiling/calibrating support which in contrast to what the wayland dev thinks is not an orthogonal problem (IMHO)
  • Not all applications actually use ICC (anything that uses OCIO for example) so not everything can use devicelink profiles to tell the compositor what to do
  • The ICC spec is rather large and complex so libraries/programs that can read devicelink profiles suddenly have a larger attack surface, this is especially a problem for compositors since those are the root of trust in the wayland world

For this and other reasons I think it would be mandatory that people from the image editing/creation software are involved in this discussion. To frame this discussion I will try to put down what I think such a protocol should look like.

Proposal

There are 4 general cases here:

  1. legacy/sRGB
  2. Basic applications (browsers, basic viewers, video players, games, basically anything that a normal user would use to view images/videos)
  3. Advanced applications (image/video/3d editors, advanced viewers, so anything that needs more control over how the colors are actually rendered)
  4. Profiling/Calibration

In my proposal the compositor internally should composite in linear/scene referred color space with the rec.2020 primaries, legacy/sRGB would be converted to this color space, while basic applications would render directly to this color space. Then the compositor would use a shader to render this composite down to the screen color space and in this space would composite in the advanced applications. In this case profiling can just use the advanced application route and only for the calibration we need to add a way to change the calibration LUT[2] from the wayland protocol.

I hope this made sense since for me it is currently late in the evening so will see if I can add a diagram tomorrow.


[1] [RFC wayland-protocols v2 0/1] Color management protocol
[2] With is usually loaded into the graphics card LUT, but in advanced monitors could also be in the monitor. Theoretically this could also be a shader that would be applied at the end.

6 Likes

As much as I love to see any improvements of colour management and HDR display support in Wayland I think here is the wrong place. So please move all the discussion/communication to the Wayland mailinglist.

I disagree this is the wrong place since a) the only developer in the opensource graphics community that even follows the wayland dev list (that I am aware of) is @gwgill and b) since we first need the requirements a wider discussion among people aware what color management is, is important; after that the real protocol can be hashed out on the wayland dev list.

On top of that this post is (also) meant to raise awareness of the issue so that more people will join the discussion, if that ends up being here on the wayland devel mailing list is less of a concern.

6 Likes

As much as I love to see any improvements of colour management and HDR display support in Wayland I think here is the wrong place. So please move all the discussion/communication to the Wayland mailinglist.

Really ? Wow! Can you explain why you think it is the wrong place ?

Iā€™ll explain why I think it is critical that there be discussion about Wayland and Color Management on a forum exactly like Pixls.

As best I can tell (from the long and unsatisfactory thread I pursed on the Wayland Developers list some little time ago), the core issue with Wayland and Color is social, not technical:

  1. none of the key developers have the faintest idea what Color or Color Management is, and (worse):

  2. they have no idea why it would be at all important.

The takeaway from this is that, as and when the free software/Linux world migrates from X11 to Wayland, all the applications that get discussed on Pixls will be in trouble, because there are no tools to setup Color Management on Wayland, and no facilities for applications to employ Color Management on Wayland, and there seem to be very serous technical and social obstacles in the way of ever doing so. In order to get any traction on this, the key Wayland developers need to be convinced to revisit some of their most dearly held goals and past technical decisions. This is not going to be easy, but the very first step is to convince them that they were missing an absolutely must-have, mission critical feature from their list of goals for Wayland: support for modern Color Management. Whatā€™s needed to convince them, is all the developers and users of color critical applications telling them that Color Managemet is vitally important, and explaining why, and what the stakes are. Currently the key Wayland developers think that Iā€™m joking when I tell them that users will be forced to stick with X11 or (gasp) even switch to MSWindows or Apple if Color Management is not possible on Wayland. Either that or they dismiss such users as ā€œnot their target audienceā€.

So I think it is very much worthwhile having discussions here, to first raise awareness about the looming problems for graphical applications on Linux, and then to develop some ideas about how to get Wayland developers to change their priorities, and get them focused on facilitating Color Management as a vitally important goal for their project.

17 Likes

@gwgill for a user such as myself that can offer nothing in terms of technical contributions/code, but can offer my usecase, what do you think is the best path forward to convince wayland developers that color is important? e.g., what should I do to help?

5 Likes

I sustain. Linux intrinsically sucks at color management, because itā€™s made by geeks for geeks. Where are the artists ? On Mac. Is that going to change anytime soon ? ā€¦

3 Likes

You know, ideally there would be a co-hosted X.Org Developerā€™s Conference and Libre Graphics Meeting, and this would provide the opportunity for graphics application writers and the Open Source graphics system providers to talk to each other. Something like a ā€œColor - why it is so importantā€ keynote presentation would be a great way of getting in front of the Wayland developers. I could imagine someone like Chris Murphy presenting an introduction to color management, what it is and why it is important. Someone like (say) Florian Hoch giving an outline of how Color Management is setup on current (X11) systems, and one of the developers of an open source application that implements color management outlining how application use the CM setup and implement CM at an application level, and in particular giving an outline of how multi-monitor CM works.

Quite how something like this could be arranged and such a keynote sponsored, I have no idea.

7 Likes

I agree something like that would be the best thing but since I donā€™t know how to arrange that either I think the best option is to hash out a set of requirements that all of us can get behind and present that to the wayland devs. Such a list would roughly look like this I think

  1. Graphics applications need to know which monitor they are running on (or at least the color space of said monitor
  2. Need access to the video card LUT (or equivalent since we might decide to not touch that anymore and use shaders in stead to for example have a per HW plane LUT)
  3. If a graphics application pushes pixels those pixels shouldnā€™t be touched/be altered (color wise) by the compositor at all

Please tell me if I am missing something here?

1 Like

I think Wayland should do some sort of unified color management:

  • applications send pixels buffers with a colorspace signature embedded, but not color-corrected,
  • the compositor does on-demand color-corrections from buffer colorspace to display colorspace only for visible frames (saves a lot of perf), possibly on GPU (for now, AFAIK, graphic libs are not multi-threaded and super slow).

That way, your whole desktop is color-managed with the same predictable algo.

The problems of on-app color management are:

  • most devs are color-clueless
  • in a multi-screen setup, when you start an app on one screen and move it on another one after, what is the actual profile used ? (i.e. does the app know itā€™s not on the same display anymore ? are the pixel buffers refreshed ?)
  • are you even sure the original display profile was the right one in this multi-display setup ?
  • there are zillions of funky DE and counting. You canā€™t ask to app devs to maintain them all.
4 Likes

Although I agree that we will want this in the near future for basic stuff (large gamut monitors are becoming more mainstream) but it wonā€™t work for more advanced stuff (profiling among other things) you want to have full control over how the colors get mapped to the output, so for those cases and alternative must be provided.

Doubly so for compositor creators and the wayland devs (just see the mailinglist discussion @gwgill had with them)

In X11 we could just ask the X11 server about this and do the right thingā„¢, currently this is not possible under wayland so we need a protocol.

That why you need external profiling in calibration and protocols to communicate the right settings

Which is why we need a standardized protocol to talk to the DE/compositor so the DE just implements the protocol and we just talk it, no need to implement something different for each DE/compositor/window manager

On top of that most/all applications currently caring about color management already implement it in app (see for example this phoronix message from Boudewijn Rempt maintaner of Krita ). They do that most of the time by just linking to LCMS and/or OCIO and let all the colormanagement things be done by those libraries.

Maybe a petition would be useful in convincing them?

I personally do not like such approach, for at least one good reason: if one has to convert pixels from a linear wide-gamut working colorspace (like Rec.2020 or ACEScg) to the display, youā€™d want to do that on floating-point values, hence before sending pixels to the compositor. Unless Wayland supports floating-point buffersā€¦

Makes sense?

1 Like

Another is that in general you want to have control over rendering intent[1] (yes even in an OCIO based app although there it is often baked into the config so not user selectable)

EDIT: I do believe wayland can support FP buffers not sure of anything currently takes advantage of that tough


[1] In ICC terms, if you want Relative/Absolute Colirmeteric(sp?), Perceptual or Saturation. Also maybe black point compensation

I fear it would just come across as ā€œa bunch of people who donā€™t understand Wayland making crazy demandsā€.

Drew a daigram with how I think it should work (roughly), double arrows means the compositor needs to communicate information back to the application, single arrow is just image data. @gwgill does this make sense to you or am I missing something?

EDIT: Got one arrow wrong (fixed)

2 Likes

Iā€™d be happy to try and facilitate this, but LGM this year is only 4 months away. Would that be enough time to get the necessary people committed to coming?

I could at least get a ā€œBirds or a Featherā€ meeting set up. Itā€™s a few hours of dedicated space for discussion of topics separate from the main presentation track.

I donā€™t know about funding for attendance though.

1 Like

Based on the discussion here, I personally think only one of the two will make it happen:

  1. A ā€˜guerillaā€™ dev makes it into the Wayland cabal and produces patches that are accepted. Recent example: @anon41087856 and filmic, pushing forward the understanding and acceptance of scene-referredā€¦
  2. Some time passes, and it becomes evident that Wayland-based distros donā€™t gain ā€œmarket shareā€ (for immediate lack of a better term), and the reason is realized that image applications donā€™t work right. Might be an article somewhere interviewing a Pixar exec: ā€œWe wanted to use %$#@-buntu, but Wayland does not support our workflow toolsā€

Yes, a bit cynical, but Iā€™m growing old and irritatedā€¦ :smile:

4 Likes

Iā€™m afraid that kind of ā€œfull controlā€ is rather a chain of trust between all the CM layers in place now in Xorg. There are too many layers, no quality control. App devs wire their pixels buffers to some magic CM lib and expect to get magically cleaned crap at the end. The color corrections done by the CM or the video card are kind of opaque. I say put all that in one place, MacOS style, stop fragmenting and layering crap.

Sure, but that makes one team and one project to educate instead of thousands of them.

??? You can have calibration, but as long as you canā€™t assert what profile is in use for that particular app, and double-check the color pipe, you live in faith. Thatā€™s what I hate about ICC wizardry on Linux : as soon as people see ICCs, they are happy because their app is color-managed somehow. Nobody worries about what they actually see on screen. And when you want to assert it, you discover opaque stuff and undocumented features.

And most of the time, they suck (cc Firefox). Also LCMS performance is a disgrace: it does not do multithreading or SIMD, let alone GPU offloading. So much for FPS. Is that 2019 or what ? Besides, if I open the same picture in darktable, Gnome viewer and Firefox, I see 3 different results. Which one is the right ? I donā€™t knowā€¦ What quality insurance tools do I have to answer this question ? None ! Trust the devs ! They know what they are doing.

I think 2019 is not the time to be clever about color management. Figure out what Apple does, and do the same. These guys have invented GUIs, their OS is the de-facto standard for the graphic industry, they have been doing it for a long time and are good at it.

If apps want to be clever, they can always send pixel buffers with a ā€œdo-not-touchā€ colorspace tag.

Instead of bemoaning how much everything sucks, can we focus on what can be done going forward?

Sorry, but they did not invent the GUI.

2 Likes

Not rely on apps to perform the color management on single-threaded CPU tasks. Centralize/build-in the CM into Wayland, close to the GPU, so color transforms could be offloaded directly on it without fancy libs deprecated every once in a while.