Color Calibration, icc, real colors

Software darktable
1

Hello everyone,

I’m new to DT and this is my first post.
I’ve read the book by “Michael Moltenbrey” and am trying to take my first steps in DT.
Unfortunately, my English isn’t very good, but I’m doing my best. So please try to use simple sentences when replying.

My basic idea is to get a solid base profile early in the Pixelpipe that displays colors correctly.
So, as described by “Moltenbrey,” I bought a color chart (SpyderChecker), photographed it, and calibrated it in the Color Calibration module using Color Checker.
I did this with three cameras—the Sony A58, Sony A77, and Sony A7—under exactly the same conditions.
But the more expensive the camera, the worse the Delta E was (1.6 for the A58, 2.1 for the A77, and 2.7 for the A7).
AI tells me that more expensive cameras often have more nonlinear sensors, which means that color calibration—which is purely linear (3x3 matrix)—is unsuitable.

So I came up with the idea of creating my own ICC profile. That worked so far.

Using:
scanin -v -dipn $FILE.tif $argyll/SpyderChecker24.cht $argyll/SpyderChecker24_Lk.cie diag_chart.tif
and
colprof -v -A “Sony” -M ‘A7RM5’ -D “SonyA7_LUT” -qh -ax -no -ua $FILE

But the result is worse than before (delta E is now about 6).
My guess is that the “default matrix” is causing interference.

The TIFF was created using the standard matrix, and I can’t just select “none” or something like that in the input profile.
But now the standard matrix is being replaced by the ICC, even though the ICC is actually a delta ICC (representing the difference between the standard matrix and the actual colors).

It also doesn’t seem possible to create a second instance of the input profile, with the standard matrix active in the first instance and the ICC file active in the second.

How can I use the color chart to generate correct colors right at the beginning of the Pixelpipe workflow?

2

Hi Thomas, welcome to the forum.

I wrote a general introduction to color management when processing raw files that might be of some help:

https://discuss.pixls.us/t/article-color-management-in-raw-processing/11521

One thing to clarify in your original post, the ‘standard matrix’ for your camera in darktable and the ICC file you created from the target shot should be quite close in performance, as both contain a 3x3 matrix of the camera’s color response, both (in the case of the standard matrix, “probably”) created from similar 24-patch ColorChecker targets. Your profile created wtih scanin and colprof contains just another 3x3 matrix that is used instead of the internal one.

Now, the deltaE (I’m assuming it’s the worst dE of the 24 patches) you got for your three cameras is actually quite good for a 24-patch matrix profile; I typically don’t get better than 3 dE from target shots. I use dcamprof to make camera profiles, it’s a command-line program written by one of the RawTherapee developers, and among other things it spews out tiff files showing patch dE comparisons in a real photo, and 3 dE is actually not that easy to notice.

I’ve done a lot with camera profiling, including measuring my cameras’ spectral sensitivity (another profiling rabbit hole), so between me and the others here who’ve done similar work we can answer your questions confidently.

6 Likes
3

Hi Glenn,

many many thanks for the quick answer. I will check the link tomorrow and hope I can come back in case I have questions. - But many thanks so far.

One thing:

My understanding according to AI (see below) was, that according to the -ax option the icc should be non-linear:

-ax: Creates a LUT-based profile (Look-Up Table) instead of a simple matrix profile. The “x” specifically stands for an XYZ cLUT profile (Profile Connection Space / PCS is XYZ)

4

Hey Thomas, welcome!

Diving in with colour calibration and a checker card certainly sounds like going into the deep end first.

May I ask: What is your use-case for dt?

1 Like
5

What was the delta E before you calibrated with CC module…just curious…

6

You have to be careful with LUT based profiles in DT esp with the scene referred workflow and the pipeline…its generally recommended not to use them I believe and to stick to matrix profiles…

7

Hi mino,
So far, with my old camera, I made only jpegs. Now I am retired bought a new camera and want to process raws to get full advantage of the new camera. Finally I need a hobby :slight_smile:

1 Like
8

I don’t remember exactly but I think arount 3.2

9

If I recall the calibration process using CC is essentially using an optimizer to add a matrix derived from the channel mixer. It will ignore any input profile and use only the standard matrix if I recall correctly… So when using it the best course of action is to use either your properly constructed profile from displaycal though as noted luts can cause clipping and other issues sometimes or use the standard matrix and take care and use the calibration in CC…to improve on that…I have found usually the neutral option is as good as any and it will also suggest an exposure correction so look for that…even then this is not a global solution…it will be decent for similar photos in similar lighting but its not one that will work with all the photo you take for that camera…

10

I am a little bit confused. What I understood is that the LUT 3D module is working in the display referred workflow and comes in the pixel pipe after filmic. Therefore so far I discarded to use the LUT 3D module.

But do you think that an icc which is based on LUT causes the same problems?

I mean I am not a product photographer but if I were, how would I get then accurate colors in DT?

My goal finally is to start RAW development with a solid foundation.

So the first step for me is to ask if I take photo of a color chart, knowing the Lab values of the color patches what I must do, to reproduce the colors (by checking the values with the pipette) exactly in DT.

I would be very happy if anyone had instructions for this.

11

So, if you don’t need “absolute color accuracy” I’d personally recommend starting with “creative editing” instead of diving into the color-accuracy end of the pool :wink: - but you do you if it brings you joy :slight_smile: . Exact color reproduction is something that goes far beyond using color calibration in dt and a rabbit hole that swallowed many a photographers ;-).

4 Likes
12

Ignore the 3dLUT module…At this stage we are talking about the input profile, in the simplest terms a lut based profile is going to give you a fixed table of values for a set of conditions. In DT, often if you are using the scene referred workflow you will be managing huge dynamic range manipulations…setting middle gray and then tone mapping to match your display dnr… under these conditons I think a matrix profile is faster and less likely to have artifacts from banding, clipping and color hue shifts than the LUT based profile might be…

I think you had delta’s that were not that bad except maybe you could do better with the A7 otherwise the standard conditions were quite respectable so I am not sure there is too much more to gain??

I’m sure others will perhaps give you better advice …

Did you try the different models when you were calibrating… I think in the past I found none or neutral to give the lowest values…

13

Thank you for this link, it helps to understand the process better.

First question: In your article you speak of “camera profile” and “working profil”. Is my understanding correct that the Input Profile in DT (where you can chose standard-matrix or the icc) corresponds to the “working profil”?

Second question: Am I right that in case I use an icc the standard-matrix is replaced by the icc, it is not a post-processing correction profile, as would be the case with color calibration?

3rd question: Can DT also use dcp profiles or only icc)

14

Yes I tried different ones especially “mean dE”. But lately I used “rich colors”. The differences were not so big.

Maybe I’m completely off the mark—but my intuition tells me there must be some way to match the values in my photo of the color chart (actual values) to the target values.
Every product photographer has to do that, after all.

15

First answer: they’re two different profiles.

One thing the article doesn’t do is describe the actual process of transforming an image from one color space to another, ICC-style. That operation depends on two profiles, and they’re used like this: input → XYZ, then XYZ → output, where the image is first transformed to XYZ using the input profile as the reference, then transformed from XYZ to the color space of the output profile. XYZ is the CIE 1931 tristimulus color space, in this case used as a intermediate destination so the input and output color profiles don’t need to know about each other. So, what happens in the raw processor is that the software first assigned the camera profile to the image (more of an administrative thing), which give the color transform the information needed to do the transform of the image from camera space to working space, usually ProPhoto or Rec2020, which are just two different high-gamut colorspace definitions.

Yeah, a lot of words, but really important to understanding how color management works. Working profiles are used by a lot of raw processing software to make the image colors ‘better-behaved’ for manipulation, and another transform is subsequently required from that to either the display or file export destinations.

Second answer: yes. The ‘replacement’ is just an administrative thing, it just assigns the matrix of either to the image data. Subsequently, that matrix is used to transform the image data, usually to the color space of the working profile. Me in my hack software, I don’t do that, I just keep the image data in the original camera space, do all my work, then do the color transform when I save the image to JPEG or whatever.

The key dynamic here is that the colors recorded by the camera are too ‘extensive’ (for lack of a better word) for direct display on media of the current technology, so the colors have to be ‘compressed’ (again,for lack of a better word) to the color gamut of the medium to be used for rendition, that is, viewing.

Third answer: No, to my understanding. I don’t use darktable, but I’ve heard that, someone more knowledgeable can confirm. But, what one can do is extract the relevant matrix from a DCP and use it to make an ICC camera profile, if one is willing to spend time mucking around with esoteric softwares… :crazy_face:

3 Likes
16

I don’t have a sony camera…

But I found this shot on pixls…

I was able to get delta E of < than 1.2… and from the picker values in rgb and looking at the rgb numbers… they were pretty close … they will never be identical, lab values were pretty good as well…

1 Like
17

I use ART instead of darktable, so this is an “out of sample” “fer instance” viewpoint. I’m using a Sony A7 III and a ColorChecker Passport, which has 50 color patches including a set having higher saturation than the original 24.

The camera profile I use most often was made using these argyll options:

scanin -dipn $filename.tif ColorCheckerPassport.cht ColorCheckerPassport.cie diag.tif
colprof -v -qm -am -u $filename

where $filename.tif is the reference file output by the raw processor. (Substitute the actual filename for $filename.)

This makes a linear matrix profile with no shapers, and it can be extrapolated for values higher than reference white.

Checking the accuracy with profcheck -v1 $filename.ti3 $filename.icc results in:

No of test patches = 50
Profile check complete, errors: max. = 7.674471, avg. = 1.698007, RMS = 2.371464

This profile was made from a reference photo that was shot with an inexpensive triplet lens. The max error corresponds to one of the more highly saturated colors, beyond the original 24 patches. (All of the CC Passport patches fall within, or slightly beyond, the sRGB space.)

A similar profile I made using a lens that would seem to be well-suited for making profiles, a Zeiss 100/2 Makro Planar, resulted in:

No of test patches = 50
Profile check complete, errors: max. = 12.105585, avg. = 3.388899, RMS = 3.957033

Visually, there isn’t much difference in the two profiles, although the Zeiss profile has higher contrast, slightly more saturated colors, and is more accurate in the yellow/orange region where people and sunsets fall.

The LUT profiles I’ve made have all been disappointing. The profcheck numbers are better, but they tend to have dull, lifeless tones; sometimes very noticeable banding in the sky; and clipped, off-color highlights.

A comment, though: If darktable’s default matrix is in the processing chain when the reference TIFF file is made, that would seem to me to be an error unless it is always in the chain (which would be another error that would prevent the proper use of third-party camera profiles). Maybe someone who knows how darktable handles this could clarify how it really works.

2 Likes
18

For the ICC I think in DT you set all the colorspaces to linear rec2020 and so you get a pass through…for the calibration in the CC module its basically an iterative channel mixer adjustment adding to wb and the input profile…all other modules other than necessary ones are usually off or should be…

This is my understanding …

19

Thanks a lot :slight_smile: This is exactly what I wanted to say with: “My guess is that the “default matrix” is causing interference. … The TIFF was created using the standard matrix, and I can’t just select “none” or something like that in the input profile.”

So this is exactly my question, but you said it with better words - thanks. I excuse again because my english is not so good.

So how can I make the reference tiff file without the impact of the standard-matrix? Or how could I put the standard-matrix always in the chain e.g. with a second instance of the input profile - which seems not to be allowed.

@reffort1: You use ART. Do you think ART is better than DT especially with respect to a proper mathematical approach? - and does ART also use the scene referred workflow?

20

You set input , working and output profiles to linear rec2020…making them all the same creates a passthrough in DT… That is the conditions for the Tiff export…

1 Like