Feature request: save as floating-point


(Graeme W. Gill) #182

I’ll make note of another source of inaccuracy in ICC profile conversions, this one being in absolute colorimetric conversions when the ‘chad’ tag is used rather than the older ICC V2/sRGB/AdobeRGB convention of using the white point tag in combination with the Bradford matrix. With the older convention the CMM can compute a float precision transform matrix between the absolute and relative WPs, the precision being limited by the ICC 1.15 bit white point tag precision. Using the ‘chad’ tag the abs<->rel matrix itself gets quantized, resulting in noticeably larger errors in the profiles ability to encode the devices measured color response.


(Clinton Ingram) #183

Hey Graeme, I’m really pleased you found your way into this thread.

Let me start with a few caveats before I get into your comments.

  1. I’ve been writing color management software for roughly 2% as long as you have
  2. I’ve never actually read the sRGB spec
  3. I unequivacally have zero knowledge of the inner workings of the ratification process for said spec
  4. I don’t necessarily expect you or anyone else to change their long-working implementations based on my math. After all, we disagree by a max value of 3/65536 in our primaries.

All that said, I’ll debate with you anyway :wink:

I don’t necessarily believe they’re more accurate, just that they’re more compliant with the spec.

That’s correct, and I believe that is the correct interpretation. The x,y values were the starting point for the sRGB primaries, but the spec deviated from those ever so slightly in order to make the math cleaner. That deviation was small enough that the value of the primaries is preserved to the level of precision given in both the sRGB and the Rec. 709 standards. In other words, using the XYZ values at the given precision maintains compatibility with the starting x,y values at their starting precision level. Using the x,y values does not maintain compatibility with the given XYZ values at their starting precision level. And that’s true no matter how much precision you keep in the intermediate processing.

This, I believe, is incorrect. The fact that the original matrices didn’t invert to each other would have justified publishing more decimal places. Instead of adding precision to the document/spec, they actually adjusted the numbers so they worked out at the lower precision. This is reinforced in the fact that not only were the matrix values adjusted, but the rounded XYZ value of D65 given is rounded in the wrong direction.

I can’t speak to lost accuracy within the CMS itself, but as soon as an ICC profile enters the mix, precision is lost anyway. The s15Fixed16Number format only has accuracy to log(2^16) =~ 4.8 decimal places, which really means 4 with the 5th being mostly right. Any precision you start with by trying to improve on the XYZ values given in the spec is lost when you save them to or read them from a profile.

Going back to the example I used in my post, this is the value of the D65 R’G’B’-XYZ matrix calculated with as much accuracy as I could preserve, starting from the x,y primaries. This was the matrix that produced the profile values that match the ones from ArgyllCMS (after adaptation, of course).

 0.4123907992659590 0.3575843393838780 0.1804807884018340
 0.2126390058715100 0.7151686787677560 0.0721923153607337
 0.0193308187155918 0.1191947797946260 0.9505321522496610

If you round-trip those though s15Fixed16Number format, they only maintain 4 dp of accuracy.

 0.4123840332031250  0.3575897216796880  0.1804809570312500
 0.2126464843750000  0.7151641845703130  0.0721893310546875
 0.0193328857421875  0.1192016601562500  0.9505310058593750

And taking that round-tripped matrix and inverting it, you get this.

 3.2411015823888700 -1.5374742408225600 -0.4986348425717410
-0.9693232691940230  1.8760231681184500  0.0415720618303295
 0.0556374237335379 -0.2039925695286720  1.0569719298571300

Which only matches the correct inverse matrix (below) to 3 dp.

 3.2409699419045200 -1.5373831775700900 -0.4986107602930030
-0.9692436362808800  1.8759675015077200  0.0415550574071756
 0.0556300796969936 -0.2039769588889760  1.0569715142428800

By contrast, the XYZ values used in the sRGB spec, with their intentional imprecision, maintain 4 dp of agreement on that round trip.


(Clinton Ingram) #184

I actually just started looking into this today, and that was my initial thought when reading the ICC spec. I can see there were some very odd decisions made between the V2 and V4 spec publication and in updating the V2 spec after.


(Graeme W. Gill) #185

I don’t see a practical way to take the sRGB spec. matricies as canonical - they are not perfect inverses of each other, so which one is the reference ?
(Note that an ICC matrix profile has just one matrix, the forward transform matrix. The CMM inverts this to create the reverse transform.)


(Clinton Ingram) #186

They are inverses of each other at 4dp precision. I actually did that math in my blog post. The only way I could get all the numbers in the spec (or rather, the Wikipedia summary of the spec :confused: ) to agree with one another was to use the published XYZ values, at their given precision. Any attempt I made to get extra accuracy, including starting from the x,y values, resulted in mismatches. That’s what ultimately convinced me that the XYZ values are the canonical ones. They all agree with each other and balance out perfectly (again, at their given precision).

The fact that the draft spec numbers aligned more closely with the values I calculated from the x,y starting point and that those values were deliberately modified between the draft and the final really drove it home for me.


(Graeme W. Gill) #187

Sorry, I’m completely unconvinced. The spec. starts with the x,y values and then does limited precision calculations to arrive at matrices that do not exactly correspond to their starting point, and are not exact inverses. Starting with the spec. primary x,y values and computing ICC rounded matrix values results a profile that (overall) more accurately models the original primaries, and therefore better represents the sRGB colorspace.


(Clinton Ingram) #188

I guess we’ll have to agree to disagree.

I do completely agree that the primaries and whitepoint you use are correct for Rec 709. And I do agree that sRGB could be described, in its original draft version, as ‘a colorspace with Rec 709 colors and whitepoint, and a gamma curve tuned for PC displays’.

Really, where we differ is that I believe the final version of sRGB could better be described as ‘a colorspace with colors and whitepoint based on Rec 709 but slightly modified for easier/cleaner PC software implementation, and a gamma curve tuned for PC displays and then slightly modified for easier/cleaner PC software implementation’

I believe the changes to the numbers between the draft and final spec back up that change in definition, and again, I’ll say the matrices given are exact inverses (at the precision given) and do still reverse to the x,y primaries/whitepoint (at the given precision). So yes, starting from x,y does more accurately model the original primaries, but I don’t believe it more accurately models the final specified primaries.

If only someone had a copy of the actual spec and could check it to see if it contains any clarification to explain the discrepancy…


(Glenn Butcher) #189

Language Lawyer here, just want to make sure communication is clear. The word “canonical” essentially means “of the canon”, which asserts the published specification as Correct. It doesn’t mean the contents of the specification are precise, accurate, or even appropriate, it means that articles developed to comply with the specification adhere to the prose as-written.

This distinction is important to the operation of complying mechanisms that need to interoperate, or operate with consistent results. They may all be “wrong”, but they’re consistently wrong.

It sounds to me like what’s being chased here is more along the lines of “colorimetrically consistent”, with respect to individual gamut transforms as well as their continuity across “adjacent” transforms. ??

The aerospace business has horribly maimed me… :smile:


(Graeme W. Gill) #190

You’re reading a lot more into the spec. than I am!
AFAIK sRGB isn’t meant to be aproximately the same primaries as Rec.709, it’s meant to be exactly the same primaries.

Well, no:

Fwd matrix:
0.4124, 0.3576, 0.1805
0.2126, 0.7152, 0.0722
0.0193, 0.1192, 0.9505

Inverse bwd matrix:
3.240625477320053 -1.537207972210319 -0.498628598698248
-0.968930714729319 1.875756060885241 0.041517523842954
0.055710120445511 -0.204021050598487 1.056995942254388

Bwd matrix:
3.2406, -1.5372, -0.4986
-0.9689, 1.8758, 0.0415
0.0557, -0.2040, 1.0570

Inverse bwd matrix:
3.240577163864186 -1.537114442037019 -0.498626528524369
-0.968926691985616 1.875748273242875 0.041517351473191
0.055812535603688 -0.204219316429837 + 1.056991553887184

Saying they match within the precision given, is the same as saying they match as long as you throw away the parts that don’t match!

[ Fixed fwd matrix number - they still don’t match ]


(Clinton Ingram) #191

Your math doesn’t match mine there, but I see you have a bad value in position 3,2 of the fwd matrix. That should be 0.1192

My math has them in perfect agreement up to 4dp

fwd

0.4124  0.3576  0.1805
0.2126  0.7152  0.0722
0.0193  0.1192  0.9505

inverse bwd

0.4124103360777000  0.3575962178119540  0.1804991017305060
0.2126157251481260  0.7151958779229800  0.0722134074030765
0.0193021307575116  0.1191881265507680  0.9504992763896300

bwd

 3.2406 -1.5372 -0.4986
-0.9689  1.8758  0.0415
 0.0557 -0.2040  1.0570

inverse fwd

 3.2406254773200500 -1.5372079722103200 -0.4986285986982480
-0.9689307147293190  1.8757560608852400  0.0415175238429540
 0.0557101204455106 -0.2040210505984870  1.0569959422543900

Basically, if a software developer uses the exact matrix given in the spec, their result will more closely match the results when using my inverted ICC profile matrix, if done correctly. And that, I think, is the goal of the profile: to be compatible with the spec and software built from the spec.


(Roel) #192

I am by no means an expert here, but the approach by @saucecontrol seems to introduce some selfconsistency, or am I mistaken?

Would there be a drift in values when you would repeatedly (let’s say 100x) apply conversions using the “imperfect” matrix?


(Clinton Ingram) #193

I’m inclined to agree with you. But, well, you know…

This is an interesting question because, as Graeme pointed out, the profile only stores the matrix in one direction, so it’s up to the CMM to invert it to go the other way. If all the precision is kept internally, you can go forward and reverse through the same profile infinite times with no difference, no matter which version of the matrix you use.

Really, it comes down to a matter of consistency between different implementations. My argument is that if you show the sRGB spec (or a Wikipedia description thereof) to 10 software developers and ask them to write code to convert XYZ to sRGB, somewhere between 9 and 10 of them will use the exact matrix published in the spec. Only a true masochist would decide to calculate their own more precise matrix based on the x,y coordinates given for the primaries. And I say that as a person who tried to do exactly that. The only thing that stopped me was that the numbers started to deviate from the spec with that approach.

I believe sRGB was designed to be simple to implement and that their imprecision is intentional to make that so. And I believe that being compatible with the 9+ out of 10 pieces of software written by developers that used the given matrices should be the goal of the One True sRGB Profile.

That doesn’t mean I expect anyone else to change things they’ve already built and had deployed for years, but I do think my approach is sound and is more in keeping with the spec as I understand it.


(Elle Stone) #194

I’m going to make one last post on this topic of “what’s the right way to make an sRGB profile”. My post focuses on two points:

  1. The major point of contention being raised by @saucecontrol on how to make an sRGB profile has to do with whether one should start with the xyY values given in the sRGB color space specs, or instead use rounded XYZ values given in various places by various people. My apologies, but I’m not persuaded by @saucecontrol’s argument in favor of using rounded XYZ values to make invertible matrices.

  2. It’s tempting to use dismissive language when critiquing someone else’s work, but doing so doesn’t further understanding or communication, in fact works against understanding and communication.

Quoting from part 3 of @saucecontrol’s posts on making an sRGB profile:

And here again is the XYZ->R’G’B’ matrix from the spec – as described to me by a little birdy.
. . .
And rounding that to 4 decimal places, we get the exact numbers listed in the sRGB spec (I’ve been informed by a Wikipedia author)

These are “appeals to expert authority”. However, appeals to little birdies and unnamed Wikipedia authors are not valid appeals to authority as the authorities in question aren’t actually named and no links are given to back up the appeal.

My statement that my ICC profiles have the same RGB XYZ values as Graham Gill’s ICC profiles is also an appeal to authority, as Graham Gill is a recognized authority on matters of ICC profiles. The difference in my appeal to authority is that anyone can go look at the ArgyllCMS website and download and examine the code and the profiles.

Quoting from part 3:

Elle rounds up every possible definition of every color referenced in the sRGB spec, does every bit of mathematical wrangling imaginable, and comes up with a final set of numbers that are very close to the ones that ArgyllCMS has in its reference sRGB profile. Score one for Argyll.

To rephrase in a less dismissive fashion, I set up a spreadsheet for calculating an sRGB ICC profile from the sRGB color space specs, using correct spreadsheet calculations and input values, all the way up to the point at the end, when comparing the calculated values to the values in the ArgyllCMS sRGB.icm profile, at which point I made a small error in my conversions from decimal to hexadecimal. That small error at the very end has no bearing on the correctness of the rest of my spreadsheet. And as it turns out, the ArgyllCMS sRGB.icm profile RGB XYZ values actually do match the calculations in my spreadsheet.

@saucecontrol wants to calculate the sRGB profile starting with four-digit XYZ values instead of using the xyY values given in the sRGB color space specs. So of course my spreadsheet calculations aren’t going to produce his version of the sRGB ICC profile.

Quoting from part 3:

Well, like Elle, I started with the assumption that more precision is better, and I didn’t like the look of those imprecise numbers from the spec. So, like Elle, I tried to calculate my own more precise matrix using the the published x,y values of the primaries. That effort was a complete failure.

To rephrase and reiterate what @gwgill already said, when converting from xyY to XYZ, and then converting back to xyY, one must use more precision for the XYZ values than for the xyY values, or else the reverse conversion doesn’t produce the starting xyY values to whatever degree of precision one has decided is required. And similarly when converting from XYZ to xyY and then converting back to XYZ.

I used 12 decimal point values in my spreadsheet calculations, which is sufficient to exceed the accuracy in actual ICC profiles and also as reported by iccToXML.

Quoting from part 3:

I drew two main conclusions from reading through those articles. The first was that well-behaved and correct sRGB profiles are difficult to create and are, consequently, rare. The second was that the reference sRGB profile shipped with ArgyllCMS happens to be that most mythical of profiles. The unicorn profile, if you will.

First, phrases like “unicorn profile” don’t further a discussion of how to make an sRGB profile. Second, creating a well-behaved (neutral gray axis) and correct sRGB profile is not difficult:

  1. Start with the xy red, green, blue, and (source) white point values given in the sRGB color space spec
  2. Use the ICC illuminant XYZ values for the D50 destination white point.
  3. Do a proper Bradford chromatic adaptation to make the sRGB ICC profile.
  4. Adjust the values in the resulting ICC profile to make the profile have a neutral gray axis.

The first major problem with many ICC profiles “in the wild” is that they weren’t made following the above procedure, and the main point of departure has nothing to do with “unicorn profiles” and everything to do with starting with incorrect white point values. The second major problem with many ICC profiles “in the wild” is that they aren’t well-behaved, which is a completely separate issue from the issue of starting with the wrong source and destination white points.

I’m sure nothing in this post will convince @saucecontrol that starting with four-digit XYZ values is the wrong way to produce a “by the specs” sRGB profile. I suppose my major motivation for bothering to make this post is that I’m rather tired of seeing dismissive terminology and arguments from unnamed authorities passing as valid ways to persuade someone that the person speaking knows exactly what they are talking about. Probably I’m a little overly sensitive regarding matters of terminology given certain other discussions we’ve had recently on this forum.


(Clinton Ingram) #195

Hey Elle,

First of all, please allow me to most humbly and sincerely apologize if the wording of my post (or any of my comments here) offended you.

Your point about the appeal to authority is well taken, although I’ll point out that in the context of a technical specification, there is very much an authority to appeal to. Unfortunately, it appears that none of the participants in this debate have actualy ever seen the specification in question, which makes this very difficult indeed.

I attempted to acknolwledge the irony of quoting values from a spec I’ve never seen by balancing that with some levity. Again, I apologize if you interpreted that as dismissive of the work you or Graeme or Pippin have done. My writing style is certainly more informal than yours, and I do tend toward hyperbole at times.

I did not, in fact, receive any of my numbers from a birdy or a muttering man on a bus, but I thought repeatledly referring back to the same wikipedia article as a proxy for the spec made for pretty boring reading. I hope everyone reading it understood that, but perhaps that’s not so.

I said earlier that I completely agree with both your and Graeme’s calculations of the proper profile values for the Rec 709 primaries. If we were to agree that those are the same as the sRGB primaries, we could stop the debate there.

I think I made a strong case for the alternate interpretation, which is that the final sRGB spec deviated from the Rec 709 primaries in order to enable simpler implementation and more consistency when implemented in software that does calculations with limited precision, as was most definitely the norm in the mid-90’s. It’s ok if you’re not convinced by that, and I do look forward to your follow-up article.

Again, I didn’t mean to suggest at all that your math was wrong at any point, only that it didn’t agree with what’s in the spec (that I still haven’t actually read). And again, my apologies if you read that as dismissive rather than the serious examination that I meant it to be. I have the utmost respect for the work you’ve done in researching all these specs and putting together the articles on your site. That’s why I have referenced them so many times.

I hope you’ll give my post another read and try to look past the language to the math and the argument I present that makes everything balance with the two seemingly conflicting versions of the colorant and whitepoint definitions. I found that they agree with each other if you take the XYZ values as truth and that they conflict if you take the xy values as truth. That was enough to convince me, but I’d certainly be open to an alternate explanation that explains things as neatly.


(Elle Stone) #196

None of your article or comments offended me personally. Perhaps my grad school teachers were too successful at instilling in me the idea that academic and technical discussion needs to refrain from dismissive language. Having dished such language out myself and also been the receiver of such language, and having been corrected and also watched the other person being corrected, I’ve concluded that discussions are more productive if people avoid such language.

Humor is always in the mind of the person who thinks something is funny, and in a technical article such as yours, your frame of mind - that which makes things funny to you - isn’t available to your readers. I think your article would be considerably improved if you revise it to exclude “humor along the way” and perhaps put a note at the end acknowledging the frustration and humorous aspects of working from such scraps of information as are available.

I think your alternative interpretation is fairly well argued for. I would add that diving in with the hex values without providing an introductory summary paragraph that explains the basis for all the hex values and especially the role you are giving to the four-digit XYZ values, makes understanding the article more difficult than it needs to be, at least for me.

“Well argued for” isn’t the same as “convincing”. If the ICC profile format stored floating point values to a sufficient degree of precision, we wouldn’t even be having this discussion. I think mostly what you are looking at (agreeing with one of your points) is a hold-over from the days of 8-bit computing. Also, what will happen to your argument in the context of iccMAX profiles, where the illuminant isn’t limited to the ICC’s XYZ values for D50? I don’t know the answer, so this isn’t a trick question in case you might wonder :slight_smile: .

My apologies, I’m not planning any follow-up articles - there are just far too many other things on my “to do” list and my ability to type is rather limited these days (I exceeded today’s limit about three paragraphs ago).

That’s a matter of interpretation and also humor, yes? :slight_smile:

One thing I appreciate greatly about this discussion is the reference to the “chromatic adaptation tag” being less accurate for recovering the source white point, than just using the V2 white point tag to store the source white point- I was wondering about that but wasn’t ever motivated enough to sit down and do the math (hint hint !)

On a related note, I’m very curious as to how the ICC comes up with their V4 sRGB matrix profile that can be downloaded from their website - if you have any insight into their actual math such that you could trace their profile-making from start to finish, I’d be very interested. I suspect their Bradford matrix might be relevant?


#197

@Elle I don’t think that @saucecontrol’s intention was to be dismissive, though I suppose it could be taken as such. (I didn’t read it that way either. Anyway, the good thing is that there was an apology.) I know less about profiles than everyone here. In terms of authority, has anyone consulted with ICC people or contributors? Or are they too busy to entertain this type of discussion.

I know that @snibgo doesn’t mind this diversion for the the main topic but I think that it would be beneficial for members and visitors to split it. What do you think @patdavid @paperdigits et al.?


(Clinton Ingram) #198

I really appreciate the feedback. And you’re almost certainly right. Unfortunately, I think leaving the humor (or attempt at humor) out makes writing less fun for me, and I already have trouble motivating myself to share the research I do. I can only hope that people that don’t get my jokes at least get my math, and vice-versa.

I think frustration is definitely the right word there. I find it absurd that the IEC has locked the self-proclaimed “Default Color Space for the Internet” behind a paywall. I’m frustrated at having to cite third-party sources for that information, and I express that frustration by making those third-party citations increasingly absurd. I thought that was funny, but that truly might just be me.

Let’s just agree amongst ourselves that when I say ‘the spec’ I mean ‘the Wikipedia article describing the spec’, and I’ll stop qualifying all my citations :wink:

That’s also good to know, since your level of understanding is already far above what I’d expect from the average reader. I figured the reference to your articles would serve as enough of a primer, but I’ll see if I can come up with an explanation that doesn’t take too much time.

Actually, I think we still would. The xy-as-truth argument presumes that the values could be more precise if only there were more decimal places used in the spec (or the profile). The XYZ-as-truth argument presumes that 4 decimal places are all that are required (or defined), because they have been adjusted to balance at that precision – and s15Fixed16Number gives us at least that.

I really should stop arguing the point, because I believe we’ve reached an impasse, but let me pose one simple question:
How do you explain the Z value of 1.0890 for D65? That’s not a rounded version of the correct number; it’s a different number. The key to balancing all the rest of the numbers is to use that different value. And while X=0.9505,Y=1,Z=1.0890 converts to x=0.3127,y=0.3290, the reverse is not true. Figure that one out, and you might get me back to your side of the argument.

This is on my list to look at. I’ll let you know if I learn anything interesting.

The ICC actually did publish a guide called ““How to interpret the sRGB color space (specified in IEC 61966-2-1) for ICC profiles”. That should be the authority on the matter, and I did cite that in my post because it agrees with my final numbers, although it presents them without much by way of explanation.

The problem with the ICC as authority is that they make some truly baffling decisions, and that they don’t use the values from their guide in their own profiles. If those of us debating the point can’t agree that the ICC has their %$@& together, they’re not much of an authority. But they did still take my side, for whatever that’s worth :stuck_out_tongue:


(Graeme W. Gill) #199

If you want the “One True sRGB Profile”, then continue using the Microsoft/HP one. That’s the one that was developed along with the spec., and very widely distributed. It even included a commentary on how it was constructed (possibly hard to locate now - I have a copy if anyone is really interested.)

To finish off in my response, let me summarize my position:

The sRGB spec is only self consistent to a limited precision. The primaries don’t exactly match to the matrices, the matrices don’t exactly match between themselves, and the fwd and bwd non-linear per channel encodings don’t exactly match within and between themselves.

So if one desires to make an sRGB conversion of higher precision and more self consistent (invertible), it’s necessary to take a subset of the specifications as canonical, and re-compute the derivative parts of it to the desired precision. Clearly the primary chromaticites are the starting point for computing the matrices, so that’s what I’ve chosen. I’d be surprised if the resulting difference between such an approach and using the lower precision matricies verbatim from the spec. is a whole lot larger than the inconsistency within the spec., and the self consistency is certainly better.

To put it another way, if a future standards committee was to take on the task of revising the sRGB spec. with an eye to improving its precision and self-consistency, I’d be a little surprised if they didn’t follow a similar path to the one I’ve followed. There aren’t a lot of alternatives that make any sense.


#200

This one https://www.w3.org/Graphics/Color/sRGB.html?

Have fun!
Colourfully Yours,
Claes in Lund, Sweden


(Elle Stone) #201

“This one” that your post points to is an old version of the sRGB spec. @gwgill was referring to an actual sRGB ICC profile distributed by HP. I think the profile in question is the same profile that libpng wants to eradicate from Linux by virtue of claiming that there is a problem wtih the profile:

Anyway, for personal and health reasons (my problem with typing isn’t getting any better), I’ve come to the conclusion that I should unsubscribe from as many forums and mailing lists as possible, to avoid temptations to type. Unfortunately I don’t know where the pixls.us “unsubscribe” button is. This is a great forum, and perhaps I’ll be able to rejoin the forum at some later date.