The Quest for Good Color - 3. How Close Can IT8 Come to SSF?"

Note: This post is a follow-up to The Quest for Good Color - 2. Spectral Profiles "On The Cheap" and The Quest for Good Color - 1. Spectral Sensitivity Functions (SSFs) and Camera Profiles. You might want to read forward from the first post to understand what I’m doing here…

As I review the results of my Rube Goldberg approach to characterizing a camera’s spectral sensitivity, I’m also becoming sensitive to how far an individual would want to go in pursuing better camera profiles than the good ole’ 3x3 matrix profiles to which most raw processing software defaults. I’m having some fun playing with power measurement, although I can’t bring myself (yet…) to pry off the IR filter in the monochrome camera I bought to make a Raspberry Pi spectrometer capable of measuring power of a light source across the spectrum. That might be the topic of the subsequent post, but I’m thinking that, if one can’t use generic power data for making camera SSF profiles, building a contraption that includes power measurement might just be a bridge too far for most folk.

So, the obvious question in that regard is, can higher-resolution color patch targets be a viable alternative. By “higher-resolution”, I mean more color patches, a quantity sufficient to inform the construction of a LUT camera profile. I believe in the first post I illustrated the shortcoming of a 24-patch ColorChecker target in making LUT profiles. So, the essential question becomes, can a target provide enough color patches to approach or equal the granularity of a 10-nanometer resolution SSF dataset? Let’s find out…

There are a few commercially-available color targets that provide larger-than-Colorchecker patch sets. You can find a lot of discussion of them on the internets, although some is dated with respect to availability. Here’s one such: IT8 Target Review. That particular discussion revolves around the IT8.7 standard, which proscribes a 264-patch set. If one bought a ColorChecker and was taken aback by the cost, more aback will be taken from you when you go shopping for this class of target. A couple of hundred dollars is where most commercial targets reside on the shopping scale, makes that lab-grade diffraction grating look attractive… :stuck_out_tongue: There’s one exception to that, in the offerings of a gentleman named Wolf Faust at http://www.targets.coloraid.de. As of today, his price for the C1 variant, meant for cameras instead of scanners, is $30US, not including shipping. Quite reasonable, considering the effort that goes into making these things. Coloraid IT8 targets come with a CD containing the reference measurements needed to take shots of the target and turn their information into conformant camera profiles. This data is measured by production lot, or “charge” (there are two such data sets for ColorCheckers, one for “before 2014”, the other for “after 2014”), and the production of the chart needs special attention to the fidelity of the patch specification. IMHO one of the better bargains in photography…

(Note: as of 6/10/2020, he’s not shipping to most places due to the COVID-19 foo…)

I would have bought my own a while ago, but the only way to pay is through PayPal and we don’t go near that. Anyway, @paperdigits graciously offered lend me his.

Now, there’s a lot of prior writing on making camera profiles from target shots. @Elle Stone has one of the better articles, and that is what I’m using for this endeavor: Make a better custom camera profile. Also worth reading is Anders Torger’s dcamprof-oriented missive: Making a camera profile with DCamProf. So, what follows is a description of specifically what I did…

It’s really a pretty simple workflow: 1) use the camera in question to shoot an image of the target, and 2) use some software to extract the relevant information from the target shot to make a camera profile. There’s actually a lot of software out there to do step 2; I’m going to focus on the two free alternatives of which I know: ArgyllCMS and dcamprof. Argyll’s author, @gwgill, posts here occasionally; I don’t think Anders does, but I understand at one time he was a RawTherapee contributor. Of note is that dcamprof doesn’t stand by itself in this; it relies on scanin, an Argyll tool, to extract the target shot data. But, let’s talk briefly about the target shot step before we consider the software…

Unlike SSF profiles, target shot profiles need to respect the color temperature of the light in which they are captured. Indeed, Adobe DCPs are organized on the use of data for two illuminants, daylight and a tungsten-level light, and an interpolation scheme to describe the camera color at an intervening temperature. Sorry, just getting a single profile to work is almost more than bear-of-little-brain here can handle, so I’ll leave that to someone smarter. That said, there is a rather simple way to use the camera profile to both convert color and to correct white balance in the same transform; I’ve played with doing that with matrix profiles and it can be made to work quite well. Something for a different article.

Anyway, I captured my target shots this past weekend during a family outing to a cabin in the mountains. 9900ft above sea level seemed to be something to use in terms of atmospheric attenuation, and there was a nice-sized stump nearby on which to prop the target. I mounted the camera on a tripod, oriented the whole thing so the target faced south at noon, and shot about five exposures starting with 0EV and working my way up to +2EV. I used matrix metering for the exposure. 10 minutes of time, including the 15 steps to the stump and tripod setup/breakdown. You can read elsewhere about black cloth, boxes, and all sorts of other things to control stray light and glare, glare being IMHO the most challenging thing to control. I found the following to be important:

  1. Find a good angle for the target relative to the sun and the camera. The objective is to eliminate any glare on the target.
  2. The optical path from the camera to the target should be centered on the target, and perpendicular to the target face. This is so the software isn’t confounded trying to line up the image to skive off the patch measurements.
  3. The exposure should be ETTR, with absolutely no blown values in the patches. Indeed, it’s probably preferable to leave some headroom from saturation than to saturate anything.

Process the best exposure to a linear 16-bit TIFF. In rawproc, I applied the following toolchain:

  1. White balance. My default is to use the camera multipliers, but instead I chose a neutral patch on the target shot.
  2. Demosaic. i just used the half algorithm; I prefer it for such things as it doesn’t change values as it builds the half-sized RGB image. Well, it averages the G1 and G2 values to get a single G (oh, that should be B1 and B2, to get B…), but that operation isn’t significant to the spectral characterization.
  3. Black/white point scaling to the extents of the data. I don’t think it’s important to do this, but it helps to see the image for the crop.
  4. Lens correction. If the lens and focal length used introduces significant distortion, correcting it helps the profiling software to find the patches.
  5. Crop. Cropping down to the registration marks also helps in finding the patches.

Processing the image can be a bit daunting, due to the plethora of options available. The two software alternatives I used, dcamprof and ArgyllCMS, are command line tools, which can also challenge folk not used to running software that way. There are GUI tools such as Lumariver (incidentally, written by the dcamprof author), but all those I know are not free. To simplify the work, as well as to make it reliably repeatable, I wrote a bash script to run the tools with the intended parameters:

#!/bin/bash

set -e

#files, formatting:
ARGYLLDIR=/home/glenn/ImageStuff/Argyll_V2.0.1/
DCAMPROFDIR=
CHARTFILE=${ARGYLLDIR}ref/it8.cht
CIEFILE=R151101.cie
OUTPUTSPEC="Nikon_D7000_IT8"

#profile creation parameters:
COLPROFPARAMS="-ax" #-as for matrix/shaper
DCAMPROFPARAMS="-p xyzlut" #-p matrix for matrix

filebase=$(basename "$1" .tif)

#collect the measured values from the target shot, produce the .ti3 file:
${ARGYLLDIR}bin/scanin -v -p -dipn -G1.0 $1 $CHARTFILE $CIEFILE

#use the .ti3 file to make a dcamprof LUT (-p xyzlut) profile:
dcamprof make-profile -i D50 -C $filebase.ti3 $filebase.json
dcamprof make-icc -W $DCAMPROFPARAMS $filebase.json ${OUTPUTSPEC}-dcamprof.icc

#use the .ti3 fikle to make a Argyll LUT (-ax) profile:
${ARGYLLDIR}bin/colprof -v $COLPROFPARAMS -u -C"No copyright, use freely" -D$filebase-colprof.icc -O ${OUTPUTSPEC}-colprof.icc $filebase

Basically, the sequence of tools is:

  1. Argyll scanin is used to read the TIFF, use the .cht file to find the patches, then use the .cie file to assign the reference patch values, and save the required data in a .ti3 file
  2. dcamprof is used to read the .ti3 file and produce a XYZLUT ICC profile
  3. Argyll colprof is is used to read the .ti3 file and produce a XYZLUT ICC profile.

At the top of the script are variables to set to one’s particular configuration. Of note are the CHARTFILE and CIEFILE variables; the first specifies the chart format file, and the second specifies the reference spectral data for the chart, in my case R151101.cie, from the data included with the Coloraid IT8 target.

The script file, cam.sh, is run like this:

$ ./cam.sh DSG_xxxx.tif

It produces two ICC files, one each from dcamprof and Argyll tools. Now, my Argyll LUT profile didn’t work so well, so I’ll only be showing results from the dcamprof profile. With these ICCs, I processed my problem-child theater picture. For reference, here are the extreme blues rendered with a matrix profile:

Here’s the same image and processing, using the dcamprof IT8 profile:

Definitely better than the matrix profile. And finally, the same image and processing with the SSF profile of the previous post:

It may be hard to see scrolling back and forth between the screenshots, but the matrix->IT8 difference is marked, and the IT8->SSF difference is subtle but definitive; the SSF profile provides better color gradation than the IT8 profile.

So, the pertinent question is, “better enough to use IT8 profiles instead of SSF profiles?” One processing example doesn’t tell the whole story; in another image I processed the same way, foliage took on a “greener” (less yellow) green with both IT8 and SSF profiles, but I didn’t poke at them enough to see a significant IT8-SSF difference. Based on the extreme blue performance I’d say that an IT8 profile is worth the effort. Of note is that the SSF profile works well, even the dodgy one I made with a school-grade diffraction filter and generic power data from a similar light. Since I’ve already built the Rube Goldberg SSF box, I’m going to stick with the SSF profile, but for anyone not wanting to take the trouble to build one, an IT8 profile should still provide better extreme color handling.

I’m going to pick at the Argyll profile to figure out what I’m doing wrong and report back here. I’m also going to continue poking at my SSF profile because, well, just because. It’s all interesting stuff, related to how to get the most out of my cameras…

14 Likes

It is indeed! This series is a good read, even though it’s barely relevant for anything I do myself :slight_smile:
There’s something really great about watching experiments where you build something.

1 Like

Fascinating stuff @ggbutcher! Thank you for continually sharing about your progress. Two questions I have at the moment:

Have you experimented with the -y limit parameter of Dcamprof to control the handling of deep blues like the ones you showed in your example?

Have you noticed any meaningful differences between the results you got with the IT8 target versus the standard Colorchecker 24? I am asking since I’ve not got an IT8 target to try for myself, and Ander’s commentary of suggests that the IT8 mostly does not produce as good a profile as the standard CC24.

http://rawtherapee.com/mirror/dcamprof/camera-profiling.html#target_eval

There is also the possibility of combining two targets e.g. the CC24 and the IT8 or an inkjet printed target (if you have the ability to make your own) which may yield quite a good result? Anders also did say the CC24 appears to be rather weaker in the handling of deep violet colours, something I’ve certainly noticed in a couple of recent pictures of deep blue-violet flowers.

A bit, and my observation is that you can tame the blues with doing that, at the expense of the other colors in the image.

Hmmm, no, that’s a rather obvious comparison that I’ve missed. I’ve been so focused on the difference between LUT and matrix that I didn’t fully consider the source of measurement, although I think most dcraw matrices were probably sourced from ColorChecker measurements. To be specific, it would be LUT profiles created with both IT8 and ColorChecker? I have a D7000 set of ColorChecker target shots, so this should be easy…

WRT Ander’s target eval, I’d surmise his reference spectra do not include situations like the blue theatrical lighting I’ve been using. I think that’s the thing about targets - they capture a representative set of spectral reflectances, and one hopes the reflectances adequately represent the colors, and their gradations, of interest. SSFs have no such bias; they consistently describe the camera response over the entire spectral segment of interest, with no focus on particular colors or their gradations.

That’s interesting, do you mind going into detail what you mean by “at the expense of the other colors”? For my Sony a7R II which is overly sensitive to blue, I’ve played around with the -Y parameter too but I’ve not noticed this issue of other colours getting worse. If there is, I hope to be turned on to it.

Yes that would be LUT profiles with both targets (best not to mix too many variables). Good to know you have the captures on hand already, hopefully photographed under the same illuminant.

It’s pretty clear Anders is not specifically commenting on only this issue. He states he is observing these spectra:

The following real spectral data has been used for evaluating matching performance:

  • nordic-nature: leaves, flowers, etc natural colors from nature in Sweden/Finland
  • lippmann2000: skin, hair, lips

Perhaps modifying the look of the profile’s LUTs might go a longer way towards getting the look you desire with extreme non-white lights. It may be necessary to build different profiles that work optimally for extreme situations because they won’t work as well for regular colours/situations. I appreciate the theoretical advantages of SSFs (if only they were easy to have, like the camera companies publishing the data) but it seems to me that more significant gains could be had with well-designed (for one’s needs) LUTs than focusing on targets vs SSFs for the profile build.

With the same theatrical image, I experimented with Y-variance, and it didn’t make the other colors “worse”, but rather “different”. All the colors changed in the image, both the blues to benefit, and the others to something else. I want to maintain original colorimetric intent as much as possible, so I abandoned the endeavor upon witnessing that change. I’ll have to dig a bit to see if I still have the renders, or even the excursion profile…

I think those trades will depend on what one shoots, and how frequently. If I were a portrait photographer, I’d definitely consider a target profile with a good representation of flesh hues. On the other hand, a photojournalist would probably prefer a single profile than handles the vast majority of the situations they encountered, and I think in that case the time and effort of a SSF profile would be well-spent, even to the extent of paying someone to do it with a lab monochromator setup. Of note is that I believe all of Adobe’s Camera Standard profiles are based on spectral measurements, so just using LR would be the easiest avenue.

The use case that I think might give traction to a single-spectrum SSF approach would be a session at, say, LGM where the box and light are set up and folk line up to capture spectra and calibration with their cameras. With 2-4 minutes of time per-camera, they’d walk away with the measurements needed to make their own general-use SSF profile at their leisure…

1 Like

This is super interesting! Thank you so much for your effort.

Is there a way to compensate for that? I am not sure how big the spectral shifts are, but it is not hard to imagine an impact. It also begs the question at which altitude the CC24 target was shot, whether it was compensated for altitude, or how consistent in general an off-the-mill-homejob-CC24 calibration is.

I’m a bit surprised how close the IT8 target came to the SSF in your comparison. I still see differences, but I agree that this is a very noticeable improvement over the CC24 target. It is an indoor shot, so quite some Kelvins away from the sunlight illuminant it was created from.

Maybe you ‘just’ found an expensive way to show that the initial CC24 created matrix profile had some systematic flaws? Or the dcamprof part introduces some extreme saturation handling that was not used for the creation of the matrix profile.

I really like that idea. As a community effort this could be used to create consistent new standard-profiles for quite a bunch of cameras.

Thanks for clarifying. I noticed only that the other colours were very subtly shifted to be slightly lighter overall, not different in the sense of changing to another hue or colourfulness perceptually speaking. Anders says this makes for a more robust general-use profile with cameras that are overly blue-sensitive and it only sacrifices colourimetric accuracy very slightly in more common, less saturated colours for significantly better handling of the extremely highly saturated colours. He cautions to expect that a profile which handles extreme colours well (depends on what each of us mean by ‘well’) might not be so accurate for less saturated colours and contrariwise. I have a set of profiles for my Sony with the Y limit at a less negative value and another where the Y is allowed to be as negative as the optimizer thinks it should be for a more colourimetrically accurate result.

That’s an interesting idea. I think that a company like Lensrentals, actually I can really only think of them in this context, is ideally positioned to offer SSF data. They can measure multiple copies of the same camera, average the variation if they find any (I’m under the impression that it is very minimal), provide pure camera or camera+lens SSF combinations of all sorts to us because of the amount of inventory they have. I would be happier to pay for that data rather than have to travel somewhere in person with my camera or ship it to someone to have it measured and Lensrentals has established themselves as some of the best and most trustworthy in the world for optical measurements.

Good idea. @ggbutcher, maybe approach them with the idea?!

I was doing a bit of Googling regarding IT8 targets (thinking along the same lines as @ggbutcher …) and came here.

One thing is: It appears that you’re comparing IT8 “full” profile vs. CC24 matrix - what about dcamprof vs dcamprof with the two targets?

I’m probably going to buy an IT8 target anyway… And I’m wondering if there might be some good use cases for the transmissive IT8 charts (put a diffuser behind a transmissive chart and shoot through it to eliminate the glare concern?)

All three images in the OP are rendered with dcamprof profiles. Look in the colorspace parameters, bottom-left, and the profile filenames confirm it (I had to go look, couldn’t remember… :crazy_face: )

Glen going back in the post…why the xyzlut vs one of the other types??

If I understand the question…

xyzlut was the only LUT option available in dcamprof for ICC profiles. (??)

DCamProf can make a pure matrix profile (with shaper curves if a transfer function is provided via the -t parameter), and a LUT profile with either camera RGB to XYZ conversion or camera RGB to Lab.

Quoted…

Also quoted…

DCamProf can make an RGB to XYZ or an RGB to Lab LUT, the latter is the default. Currently the XYZ LUT uses the forward matrix directly which means that extreme value handling is not as good as the Lab LUT, so I recommend using the default Lab LUT.

Oh, that’s right, forgot…

So, in the ICC transform, there are two steps: camera → PCS, then PCS ->destination. If the camera profile converts to Lab, then the destination profile needs a matrix/LUT that converts from Lab. I don’t have any; I use Elle Stone’s ‘well-behaved’ profiles for most of my destinations, and those are all XYZ anchored.

Unless I’m missing something about the process, which is likely…

1 Like

Not likely , I really just wondered if there there was a technical reason and I had remembered a tid bit that lab was preferred for lut creation…I went to check on the color charts…listed as sold out…

What do you think of these… CMP charts from France… No Title

The Mini - matte finish.

When you stick 2 profiles together to make a colour transform, the CMM* will perform conversions between XYZ and Lab PCS encodings as necessary. So (unless you are writing your own colour management software) you don’t need to worry about it.

* Colour Management Module - e.g. LittleCMS

Sooo… if I create a camera profile with dcamprof and the lablut parameter, if I use this profile in a transform to, say, Elle Stone’s v4 g22 sRGB profile (which has XYZ round trip matrices), LittleCMS will do camera → Lab, Lab → XYZ, and finally XYZ → sRGB? Never got that from the docs, although my reading was mainly "how does cmsTransform() work?, to do rawproc coding…

I think we do too much slinging of the pixels raw processing, so adding more transforms, especially one that departs, then returns to RGB coordinates, is not appealing…