From what I can see you can process your raw file to a TIff and use Argyll or you can use Darktable-chart to create a profile, here the files are saved to pfm set to the Lab colorspace. There are a few documents out there about the processing of the raw file and I was hoping to clarify something…Harry Durgins videos are older so he doesn’t mention the input color profile settings…I have seen it recommended to use the embedded profile and I have seen Rec709. These are older videos so I am expecting those suggesting 709 would now recommend rec2020 or linear prophoto. I think Shane Milton’s video recommends setting all the color profiles including display to rec709. I would welcome comments on what setting people would use with the current version of darktable. Is there a relation to the colorspace used to process the raw image of the colorchecker card or should you just use the widest space you can. Second I have seen a couple that would indicate that highlight recovery should be off and others leave it on. Finally I have a Spyder Colorchecker24 card …They recommend white RGB to be ~240 and Black ~10 and to white balance on the second grey patch for their software which doesn’t work with DT but should I follow that guideline…Most of the DT material talks about the LAB value of the white patch and picking an image around 92 or 96 depending on your card but there is no mention of correcting the black point and often it is suggested to use the grey patch that is L>50 and has the smallest sum of A^2 + B^2 Lab values…The cards have both rgb and lab values provided so either could be used I guess and it is a bit of nuance but I was wondering if anyone had suggestions particularly around the input colorspace setting and what they should be set to…Thanks in advance
Actually, you shouldn’t use working or output colorspaces in making a linear TIFF of a color target for a camera profile. Argyll colprof needs the image data in the original camera space in order to make the corresponding profile.
Really, the only adjustment I’d consider appropriate for a target shot would be exposure compensation, but you really should strive to get a ETTR exposure instead. That’s what’s behind most recommendations to get a bracketed set of exposures.
What you want is a really neutral patch. I’ve used the various gray patches on the bottom row of the ColorChecker, but I’ve read elsewhere that those aren’t as neutral as one would want. I didn’t notice any detrimental color cast in using one of the CC patches, however. Maybe I’m just insensitive… 
I’ve also not white-balanced my target shot and then used it to make a profile that both described the camera colorspace and corrected the white balance. It actually works well, avoiding sling-shotting the data around in a separate application of white balance, but it requires you to make a target shot and camera profile for each lighting you encounter.
I learned the most about camera profiling reading Elle Stone’s writings on the topic:
@ggbutcher Thanks Glenn, it seems more clear in RT as you can set an initial neutral style which applies no edits and then your can go to color management and select no camera profile and finally they have an option to save a reference file which is just the demosaic’d data to a linear TIFF. I was trying to establish a similar set of settings for Darktable but it is not as clear how to do this with their settings of input and output profiles…The closest and most updated reference seems to come from here…GitHub - pmjdebruijn/colormatch: ColorMatch under the processing the raw section…so setting input and output to rec2020 and intent to absolute and gamma to 1 should be similar??
Rec2020 is a smaller gamut than your camera’s spectral sensitivity, so output to that will change the data.
This is a bit dated, but Elle says at the time of writing that “Rawtherapee provides a button for outputting raw color.” Raw Processor Review, Part 1 (scroll down to F3).
You might consider just using dcraw to do this: dcraw -o 0 -g 1 1
Ya I was thinking about that as the output from DT and RT using its save reference image are very different. The RT image is much darker more like what I would get with DT if I set the input to sRGB?? It may be better to just try some command line approach although it would be nice to see what the correct settings for DT might be. Some of the difference is that DT uses a pfm file from the chart and or a JPG to create a style with a tone curve and a lut…a few people have done some work with argyll but again there are a couple of recommended approaches and you can’t seem to set no input profile in DT…interesting on your comments I always thought rec2020 and prophoto were wide enough for cameras
The unprocessed “image” will be darker. Don’t forget, for this purpose you’re not producing an image, you’re producing data…
I plotted my matrix camera profile against ProPhoto some time ago, surprised me how much larger was the camera response.
When I went to do this work with rawproc, I had to make software changes to save with no transform/embed of an output profile. Hadn’t occurred to me that I’d really want to do that sometime…
"You might consider just using dcraw to do this: dcraw -o 0 -g 1 1"
I think maybe I need to add -W…there seems to be an automatic lightening if not…also will likely try to do a WB although you mention that you have experimented with not using it…-a vs -w was quite a bit different…I guess it can do coordinates and do a WB so I could retrieve those for the neutral patch… I tried ufraw as well and went through to produce what I thought would be linear and it was different as well…dcraw default outputs a PPM that looks different PPM is lighter than if I output to Tiff…I think I will do a systematic image creation and attempt to see what appears to be the most likely pure raw output…
Looks like this is the string as given in dcamprof…dcraw -v -r 1 1 1 1 -o 0 -H 0 -T -6 -W -g 1 1
actually, if you set the input profile to Linear Rec2020 RGB, then darktable is going to assume the raw values from the camera are already in that colourspace, and if the working profile and output profile are also set to Linear Rec2020 RGB, then the values will just be passed through transparently, isn’t it?
There is only a remapping where you are moving from one colour space to another (eg. if input profile is Linear Rec2020 RGB, and output profile is sRGB, then a correction matrix will need to be applied to account for the different primary colours, and rendering intent needs to be applied to handle any negative or too large values that occur as a result of that transform (since in this new vector space such negative or overflow values are considered “out of gamut”).
Thanks Matt,
I still can get confused by DT but you are correct I believe. As long as all the color spaces stay the same in DT then the raw values are intact. There is a style in DT in noise tools called linear raw that looks like it does this …its uses rec709 but it also set the WB values all to 1….maybe this is the true raw values however many of the color management videos for ICC suggest that you should spot wb on the chart for ICC. In the end if it is just the raw values you want and they are sensor data I guess it should not matter what colorspace. The problem I was running in to was what to set the histogram and display profiles too as in one case things are quite dark and others they are very light…sometimes depending on the combination the picker values don’t seem to change and sometimes they do…this was the tricky part as I was trying to assess the white and black patches for suitability to use them. Finally DT chart takes that modified approach and creates a style and it seems that you turn off most modules but you use the standard or enhanced matrix for the raw image that it pair with the jpg to create that style so again a bit different than creating an icc with Argyll ….
In this regard the colormanagement module of RT is so much clearer and it supports icc and dcp files so you can grab them from adobe and modify them and its quite clear at each step what is going on…they also allow no input profile and a direct save of the image before the input profile in the pipe as a reference image so its much clearer to me how to proceed in that setup…just wanted to go end to end in DT and I think it should be okay but it could be clearer for sure in my opinion…
Hmmm, preservation of the original values depends on the inverse-ability of matrix mulitiplication. That is pretty straightforward, but the numeric precision in which it is done may have an impact on how ‘identical’ is the output. It may be close enough for gov’t work…
Hmmm, preservation of the original values depends on the inverse-ability of matrix mulitiplication. That is pretty straightforward, but the numeric precision in which it is done may have an impact on how ‘identical’ is the output. It may be close enough for gov’t
But we are talking about the identity matrix here, are we not? By setting the input profile to “Linear Rec2020 RGB”, we are saying that the <a, b, c> values received from the raw file should be interpreted as being vectors in Linear Rec2020 RGB space. The working profile is also Linear Rec2020 RGB, and so the base vectors are the same in that vectors space, meaning you are multiplying by the identity matrix. Same deal at the output profile. The identity matrix is of course invertable, and its inverse is simply itself. Have I understood this correctly?
There is a style in DT in noise tools called linear raw that looks like it does this …its uses rec709 but it also set the WB values all to 1….maybe this is the true raw values however many of the color management videos for ICC suggest that you should spot wb on the chart for ICC.
This WB point confused me too. If we do a WB adjustment on the raw values, it means the values in the “R” and “B” channels will be scaled to equal the “G” channel for a neutral colored pixel. So, when you calculate the coefficients to map from the raw space to a true Linear Rec2020 RGB space, the assumption is that the white balance has already been done. However, in practice, the values coming from the camera will not have been balanced when they hit the input profile matrix, but this doesn’t matter, because the white balance module later in the pixel pipe will do this balancing. Because these transforms are linear, we can do the WB first and then apply the input matrix, or we can apply the input matrix followed by WB scaling, and the order of operations doesn’t matter if they are linear (in maths we say the composition of linear functions is commutative).
So, in short, the reason we want the raw values doing into the profiling matrix to be scaled for white-balance is because when we use the input profile matrix, it will always be used together with white balance module, and therefore we don’t want white balancing to be baked into our input profile matrix, otherwise the white balance scaling will be applied twice, once by the input profile matrix, and again by the white balance module, and this will then mess things up.
The ICC way of doing these transforms is input → linear XYZ → output, two transforms. That way, an input profile doesn’t need to have a matrix for every destination. The profile contains one matrix, input → XYZ, and if the profile is used as output the inverse of that matrix is calculated, I think, one of the big heads can weigh in here. I just use LittleCMS to construct and invoke the transform and it does something internally.
rawproc will let me do this exercise; let me construct a few screenshots of a toolchain…
Eeeeenteresting… I opened a colorful raw in rawproc, and did the following:
-
changed the display profile to Rec2020-elle-V4-g10.icc. Not interested in how good it looks, interested in the difference. Need to remember to change it back…
-
Appllied my standard toolchain:
- colorspace:camera,assign
- subtract:camera
- whitebalance:camera
- demosaic:half
- blackwhitepoint:data
- In the tool chain, changed the colorspace tool from ‘camera,assign’ to ‘file,assign’, and selected the Rec2020-elle-V4-g10.icc.
At this point, I had one color transform going in the toolchain, the display transform at the end, Rec2020 → XYZ ->Rec2020. Then, I turned off the colorspace:file,assign tool, which disabled the display transform, rawproc in that case just blats the image on the screen untransformed. Noted was no change in the displayed colors or in the histogram of the last tool. Wanting to see some numbers to corroborate, i stuck a colorspace:file,convert tool in after the demosaic, selected the Rec2020 profile, and that let me collect image stats with the colorspace:file,assign tool turned on, then off. Here are the numbers:
On:
channels:
rmin: 0.000000 rmax: 0.202877
gmin: 0.000000 gmax: 0.194107
bmin: 0.000000 bmax: 0.126031
channel means:
rmean: 0.008395
gmean: 0.007008
bmean: 0.003750
Off:
channels:
rmin: 0.000000 rmax: 0.202877
gmin: 0.000000 gmax: 0.194107
bmin: 0.000000 bmax: 0.126031
channel means:
rmean: 0.008395
gmean: 0.007008
bmean: 0.003750
Whatever’s going on in LittleCMS, it’s not changing anything I can discern in this scenario. Not a scientific test by any means, but it indicates the behavior you’re expecting is valid…
Thanks @patdavid I have read this and I am going to try this to compare…it is interesting using this approach it does appear that it is suggested to set the input profile in DT to the camera matrix and use the camera WB …and the working profile to rec2020 or rec709 and the output is in LAB… so there are conversions here for sure and this is then referenced against either a Jpg.pfm or cie file to create the style but ICC creation from what I can tell usually wants WBalanced linear raw input so that is where I was trying to sort it out…this is straightforward in RT you just save the reference image and RT will let you save an image before the input profile with or without WB but it just got confusing in DT as there are so many profiles…preview, display, histogram, input output working…input output and working I get but I was trying to be sure that he values I saw on the pickers likely coming from the histogram for patch evaluation were what I wanted them to be…some writeups in DT have set all these profiles to either rec709 or rec2020…maybe this is the safest route. Thanks everyone for you comments
Todd, I never claim that my tutorials are 100% correct. If I wrote one, then because there was no detailed documentation available or there was and I didn’t find them good enough. So I encourage everyone to also question them and suggest improvements. For most of them I got a lot of good feedback and updated them accordingly.
About white balace: It is set to camera WB because I measured and set WB using a grey card! I waited till I got the right sunlight. Read the whole tutorial please 
As I wanted to match the output of the JPG I choose “standard color matrix” because I when I open a RAW file that’s what will be applied by default. I understand the camera profile as a diff between the RAW and the JPG. So when I edit a raw I start with the standard color matrix. If I create the profile with linear RAW, I would have to edit a raw always using a linear input profile!
Note that darktable chart create a color LUT and a curve to match the JPG. I was just interested in the color LUT which obviously only works on daylight pictures
By the way, here are my files for the Sony A7 III I used for my profile:
https://xor.cryptomilk.org/darktable/ILCE-7M3/ColorCheckerSG/
Andreas, thanks for the background and please note my comments reflect my ignorance and were in no way criticisms of your approach in fact I was sure there were certainly reasons to try this method vs creating an ICC profile and I was just getting lost in the details or if not lost just not sure I was creating the proper reference image for input…thanks for taking the time to elaborate….
It took me a long time to get usable shots of the color checker. That’s why I used the black box, used a grey card and checked onsite for glare to fix my setup. Glare is the killer, if you have it your shots are useless.
Also you have only 10 minutes with good sunlight. Time is short so prepare well. I created a preset in camera with all the settings I needed to not forget anything.