Warning: auto-selected neutral patch (D02) [...]

Processing
1

I’ve been experimenting with and diving into dcamprof to create dcp files and although there are some ups and downs I’m getting there and all starts working nicely.

There is one thing I’m not certain about, this message when creating the json profile step, which always shows up:

Warning: auto-selected neutral patch (D02) is not the lightest, as the
  lightest patch is considerably off-white. That is if you later use the target
  for white balancing you should use the indicated patch instead of the
  lightest.

Is this an actual warning or is it actually an informational message?

D02 is, of-course, not the lightest patch. It is the 18% grey patch on the X-Rite colour checker and not the white patch (D01). The message also mentions using the indicated patch for white balancing. D02 is meant here, right?

2

That is what I would understand it to be from that wording. I think maybe its there because you don’t usually wb the reference images for DCP if I recall and so likely just a warning if you do use the image later for wb to use that patch???

3

https://torger.se/anders/dcamprof.html#white_balance

Explanation in the third paragraph I think should clarify…and there is a bit more info below that…

4

Yeah, I’m not entirely sure yet how to read/interpret this. Especially if this is purely mentioned for when you would want to use D02 (or another?) patch to use as white balance target when doing your edits later on.

I found this in the docs:

White balance

You can control white balance settings with the -b and -B parameters. Per default DCamProf will make a profile which expects the white balance to be set by color picking the most neutral light patch. In some cases the target “white” is actually considerably less neutral than a darker neutral gray patch, if so that will be used instead. If you are going to use the target as white balance setter for a scene it’s safest to specify a specific patch as reference, you do that with -b.

The most accurate correction is however had if you let DCamProf optimize towards a virtual 100% neutral patch, this will typically place the ideal white balance a little bit off the target’s white. As it’s only about 1-2 DE it’s really only of mathematical interest though, it shouldn’t make a visible difference in any normal circumstance. If you want to do this you enable the -B flag.

Note that this only affects the forward matrix (which is used for the color corrections), the values used for color matrix calculation will not be re-balanced as it doesn’t make sense; it’s not used for color correction but only for estimating the light’s temperature and tint and thus re-balancing its data would only reduce its precision.

There’s much info about white balance in both dcamprof docs. Getting them in the correct context (at the scene. while creating a profile, when editing using the profile) is a bit hard at the moment…

1 Like
5

I think by default it would use D1 if that was the most neutral. I think it uses the lightest patch that is the most neutral patch which will I think most often will be D2 but could be D1 so as such they warn you that this is what they have used and so if you down the road need to use a picker in some raw software then choose the suggested patch?? That would be my take

6

LOL and here I am thinking it is D02 based on this:

[...] for a CC24 the second patch in the grayscale row is actually more neutral that the lightest.

I do need to do some more testing and let the messages shown be for what they are for now. I’m getting reasonable end-results at the moment using the defaults mentioned in the docs, although not consistent enough yet.

One of the things I do need to try is using -B (disable using an actual patch), optimal white balance is then automatically calculated. I can do this, I think, because I use a grey card to set the white balance and immediately after that I shoot the colour checker, which seems a good way to do things. So white balance should be correct for that scene at that moment.

Man, scanin is finicky about alignment :smile:

7

I have cycled through many of these. I even tried the xrite software which I hacked to run against my spyderchecker card. I found sometimes orientation was an issue. You had to check the *.cht and cie files you were using to see if there was the expectation for portrait or landscape ie the white patch top left or bottom left…

I was mostly doing icc as I use DT most often.

For DCP I have just poached them from Adobe…

I’m on windows so I can also use the Adobe profile editor which is a nice GUI to edit curves and colors in a dcp profile…it can match a color checker as well…

8

I would look at the output diag file to see if it actually properly chose the correct patches.

I’ve found that the patch autoselection doesn’t work right most of the time, but there’s a solution to that:

You are right that scanin is REALLY finicky about alignment. Good luck having it autodetect from a reference image that was captured with a fisheye lens (fixed-lens camera, no other choice) no matter how carefully you defish in Hugin.

Argyll-scanin has a -F function that lets you input the coordinates of the fiducials. They are clockwise from the nearest point to the brown patch. Finding these is easiest with gimp, although you want to add an ICC profile to the image that at least tells gimp to linearize it. (argyll-scanin will ignore whatever is in the profile, but you want it to be flagged as linear data in gimp or things will be too dark.) Relevant issue that I need to revisit: Reference TIFFs are not tagged with an ICC profile that indicates linear data · Issue #5575 · Beep6581/RawTherapee · GitHub (Specifically I still haven’t had a chance to look at what @Thanatomanic posted on Drive…)

Also, for whatever reason, the “ColorChecker Passport” chart file included with Argyll doesn’t actually match Passports - the normal “ColorChecker” chart does.

9

The finicky part I run into is mainly about cropping and perspective. If either or both aren’t exactly right it just aborts and doesn’t create any files (so no diag.tif either). scanin being able to do its job doesn’t guarantee anything though, that where diag.tif comes in (otherwise dcamprof make-dcp… will crash/abort or create a non-useable dcp profile).

Well, scanin was basically created for scanning targets that are placed on a scanner…

That is nice to know, thanks!

I’ve been using ColorCheckerPassport.cht (from Argyll_V2.1.2) and cc24_ref-new.cie (created by Anders Torger). Haven’t noticed any issues. But I’ll keep an eye out for possible problems.

EDIT: @Entropy512 Just had a look at your github issue and I do believe that I ran into that. I’m not able to create usable 16 bit linear tiffs using RawTherapee. I put that aside and am using dcraw instead atm, but it is on my list to check why that doesn’t work. You might have saved me some work.

10

I’m pretty sure the reference TIFFs are linear, they’re just not tagged as such. You need to manually tag them as linear with exiftool. (I use RT’s ICC profile generator to make a linear profile, and tag it using:
exiftool '-icc_profile<=foo.icc' file.tif

Alternatively you might be able to use the following:
Set camera profile to “none”
Set working space to sRGB
Use the ICC profile creator tool to create a linear sRGB output profile

@Thanatomanic found some sort of differences, I had no issues myself, and haven’t yet had time to track down the issues he saw.

Weird that ColorCheckerPassport.cht worked from you, every time I tried to use it, I got really bogus results in my .diag file

11

@Entropy512 or @Jade_NL could you try processing these two shots with dcamprof to see if there are noticeable differences in output?
https://drive.google.com/open?id=1IwoNkv6gkuWDBgJe0BKzxu3YTH5afJH3
https://drive.google.com/open?id=1aMLJOnbH2yplDGno-Bds08hOPl4BsucX

This has not so much to do with the original issue from Jacques, and more with the linked GitHub issue about embedding a linear icc profile. But if you could check that would be great. The ‘reference’ image is created by the ‘Save reference TIFF’ option in RT, the ‘manual’ image is created by disabling WB, setting input profile to ‘None’, working profile to ‘sRGB’, the abstract profile to ‘Linear’ and export profile to a custom linear sRGB icc profile that gets embedded.
When opened in GIMP these files look identical, but the color spaces are interpreted differently.

13

Used the wrong cht file in my first attempt, results are slightly different:

Having a look at these with XnView, immediately after downloading: Roughly half of the patches of the manual tif are noticeably darker compared to the reference tif. Not sure if the other patches are influenced as well, but I’m guessing they are.

Running scanin on colorcheckr-sample-manual.tif:

$ /opt/Argyll_V2.1.2/bin/scanin -v -p -dipn colorcheckr-sample-manual.tif   /opt/Argyll_V2.1.2/ref/ColorChecker.cht /home/jade/.local/git/dcamprof/data-examples/cc24_ref-new.cie
Input file 'colorcheckr-sample-manual.tif': w=1090, h=719, d = 3, bpp = 16
Data input file '/home/jade/.local/git/dcamprof/data-examples/cc24_ref-new.cie'
Data output file 'colorcheckr-sample-manual.ti3'
Chart reference file '/opt/Argyll_V2.1.2/ref/ColorChecker.cht'
Creating diagnostic tiff file 'diag.tif'
About to allocate scanrd_ object
Verbosity = 2, flags = 0x62a01
About to read input tiff file and discover groups
adivval = 1.000000
About to calculate edge lines
94 useful edges out of 107
About to calculate perspective correction
Perspective correction factors = -0.000010 0.000000 545.000000 359.500000
About to calculate rotation
Mean angle = 0.108095
Standard deviation = 0.653510
Robust mean angle = 0.091734 from 81 lines
About to calculate feature information
About to read reference feature information
Read of chart reference file succeeded
About to match features
Checking xx
Checking yy
Checking xy
Checking yx
Checking xix
Checking yiy
Checking xiy
Checking yix
Axis matches for each possible orientation:
  0: xx  = 0.168754, yy  = 0.177742, xx.sc  = 0.293563, yy.sc  = 0.293267
 90: xiy = 0.084548, yx  = 0.191020, xiy.sc = 0.573383, yx.sc  = 0.372608
180: xix = 0.168315, yiy = 0.132635, xix.sc = 0.293563, yiy.sc = 0.293267
270: xy  = 0.084583, yix = 0.187454, xy.sc  = 0.573383, yix.sc = 0.234639
r0 = 0.244845, r90 = 0.135748, r180 = 0.214079, r270 = 0.084157
There are 0 candidate rotations:
About to write diag file
/opt/Argyll_V2.1.2/bin/scanin: Error - Scanin failed with code 0x3, Pattern match wasn't good enough

Error occured and I cannot continue. The diag.tif

Running scanin on colorcheckr-sample-reference.tif:

$ /opt/Argyll_V2.1.2/bin/scanin -v -p -dipn colorcheckr-sample-reference.tif   /opt/Argyll_V2.1.2/ref/ColorChecker.cht /home/jade/.local/git/dcamprof/data-examples/cc24_ref-new.cie
Input file 'colorcheckr-sample-reference.tif': w=1090, h=719, d = 3, bpp = 16
Data input file '/home/jade/.local/git/dcamprof/data-examples/cc24_ref-new.cie'
Data output file 'colorcheckr-sample-reference.ti3'
Chart reference file '/opt/Argyll_V2.1.2/ref/ColorChecker.cht'
Creating diagnostic tiff file 'diag.tif'
About to allocate scanrd_ object
Verbosity = 2, flags = 0x62a01
About to read input tiff file and discover groups
adivval = 1.000000
About to calculate edge lines
153 useful edges out of 199
About to calculate perspective correction
Perspective correction factors = -0.000002 -0.000001 545.000000 359.500000
About to calculate rotation
Mean angle = 0.104256
Standard deviation = 0.610275
Robust mean angle = 0.088960 from 124 lines
About to calculate feature information
About to read reference feature information
Read of chart reference file succeeded
About to match features
Checking xx
Checking yy
Checking xy
Checking yx
Checking xix
Checking yiy
Checking xiy
Checking yix
Axis matches for each possible orientation:
  0: xx  = 0.308687, yy  = 0.358595, xx.sc  = 0.287892, yy.sc  = 0.291609
 90: xiy = 0.156132, yx  = 0.264904, xiy.sc = 0.570563, yx.sc  = 0.372744
180: xix = 0.337920, yiy = 0.337946, xix.sc = 0.287630, yiy.sc = 0.291609
270: xy  = 0.157982, yix = 0.292337, xy.sc  = 0.570563, yix.sc = 0.372705
r0 = 0.467127, r90 = 0.200882, r180 = 0.471389, r270 = 0.217062
There are 2 candidate rotations:
cc = 0.467127, irot = 0.088960, xoff = -43.273007, yoff = -19.679275, xscale = 3.473525, yscale = 3.429254
cc = 0.471389, irot = 180.088960, xoff = -1111.115879, yoff = -757.046693, xscale = 3.476690, yscale = 3.429254
About to compute match transform for rotation 0.088960 deg.
About to setup value scanrdg boxes
About to read raster values
About to compute expected value correlation
About to compute match transform for rotation 180.088960 deg.
About to setup value scanrdg boxes
About to read raster values
About to compute expected value correlation
Expected value distance values are:
0, rot 0.088960: 2728.691291
1, rot 180.088960: 3972.455379
Chosen rotation 0.088960 deg. as best
About to compute final match transform
Improve match
About to setup value scanrdg boxes
About to read raster values
About to write diag file
Writing output values to file 'colorcheckr-sample-reference.ti3'

The diag.tif

Not sure if still relevant but: No errors in the following 2 steps, here are the files created (tie, json and dcp):

colorcheckr-sample-reference.zip (802.1 KB)

14

Hmm, those results are different from what I got - I wonder if the problem is that new “abstract profile” functionality. When I last did the manual workflow, I got nearly identical data to the reference TIFF, and “abstract profile” wasn’t a thing yet. Your data is vastly different, almost like an attempt was made to convert sRGB->linear on data that was already linear.

15

At its core, the ‘Abstract profile’ is exactly the same as the options your previously had to set the slope and gamma for the working profile.

16

Hmm, I don’t remember those options, only for the colorspace.

What happens if you use “None” for the abstract profile?

(I unfortunately didn’t get around to looking at this last night and didn’t see your post until this morning just before leaving for work. I’ve been tending to get nerd-sniped in the evenings by reverse engineering an oddball 360 camera…)

17

I’m just going to put my findings up to now here. It is not a how-to but it might be of use to others.

Goal: Using RawTherapee to create a useable 16 bit linear tiff file from a X-Rite ColourCheckerPassport (2019 edition) that can be used to create a dcp profile. And I’ll add the commands I used to create the actual dcp profile.

After some inconsistent results and outright failures I’ve been experimenting most of today to get together steps that ensure a high success rate. I’ve been able to repeat what I’ve found and determined that failure using these steps are most likely due to not cropping or applying perspective correction correctly.

I’ve been able to create these profiles using dcraw to create the 16b linear tif, loading that tif into rawtherapee to do the perspective correction and cropping. But that’s just one step too many. These profiles I want to create are scene specific and not meant as a global, very precise, dual-illuminant input profile for a specific camera. I want to be able to spent as little time creating them and still be assured they are correctly created and usable.

Up till now I either got a result that did not match the dcp profile that was created using dcraw (and didn’t look right when applied) or I wasn’t able to complete the process due to an array of errors thrown by either the scanin command or the dcamprof make-dcp command. @Thanatomanic mentioned in post #11 using the ‘Save reference TIFF’ option which seems, at the moment, the only way to create usable 16 bit linear tifs using rawtherapee.

The resulting dcp profile created with the below steps matches the dcraw based profile (well, it does when meticulously looking at it on a good calibrated monitor…).

Needed software: RawTherapee, Argyll and dcamprof.

These steps work for me:

Creating the 16 bit linear tif

  • Open RAW with colourcheckerpassport in rawtherapee,
  • Set profile to neutral,
  • Set correct perspective,
    • Don’t use the automatic options but draw your own control lines (2 horizontally and 2 vertically) to straighten out the image. Use the 4 check-marks on the right pane to do this.
  • Crop the image,
    • Crop on the border of the matt black and the plastic cover. Ignoring the vertical centre, do try not to include any of the plastic.
  • Save this using Save Reference image.

This is what it looks like after saving and still in rawtherapee:

rt.16b.lin.tiff
rt.16b.lin.tiff1920×1080 1.27 MB

Read patches
scanin -v -p -dipn ccp.tif ColorCheckerPassport.cht cc24_ref-new.cie

ColorCheckerPassport.cht can be found in Argyll’s ref directory.
cc24_ref-new.cie is part of the dcamprof files.

Check the created diag.tif file. It should look something like this (I converted it to jpg for showing here):

rt.asref.diag
rt.asref.diag1920×1370 443 KB

Make json profile
dcamprof make-profile -i D50 -C ccp.ti3 ccp.json

Convert to dcp profile
dcamprof make-dcp -n "Camera name" -d "Profile name" -t acr -g adobergb-strong ccp.json ccp.dcp

Please, do not just copy and paste the above commands. I put them up here to show what I have done and to give people a bit of a reference. Depending on the circumstances and colour checker used some options need to be changed.

I strongly suggest to have a look at, or better still, actually read the following:

Encountered errors/warnings are:

  • glare warning is being mentioned when running the scanin command.
    • Try a better crop. The surrounding plastic might be the cause,
    • This could be caused at shooting time and especially semi-glossy targets seem to suffer from this (no personal experience),
    • This specific warning will not end with an abort, but the end result will be far from preferable!
  • Failed to generate TPS! Aborting (by dcamprof)
    • I’m not sure yet what actually causes this to happen but I only see this if I try to create a 16 bit linear tiff using RawTherapee and do not follow the above mentioned steps (not using the Save Reference image step in particular).
  • Scanin failed with code 0x3, Pattern match wasn’t good enough
    • Crop and/or perspective are set incorrectly. scanin is very, very finicky when it comes to those two!

I still don’t have a clue if the neutral patch warning is to be ignored or not. I’m guessing it is.

1 Like
18

Great write-up! Though, you are aware this exists? How to create DCP color profiles - RawPedia

2 Likes
19

No I was not. I am now though :laughing:

20

Just read the RawPedia article and it is more elaborate then mine and less technical then the dcamprof articles I linked to. Nice.

There’s one thing that I need to retry though:

I started today only using the 24 patch panel, rotated +90 and cropped to be in line with ColorChecker.cht instead of ColorCheckerPassport.cht. But whatever I tried the scanin part did not work correctly. No errors where shown but the allignment was off (as if the bottom part was pushed up 1/4 of the way).

This was not RawTherapee specific, also happened when doing the conversion to tiff step with dcraw instead of rt.

Another item on my list. Aren’t lists supposed to get shorter :thinking:

21

So I finally figured out why using the “ColorChecker Passport” chart has always failed for me - I only used the panel that is a 1:1 map with a CC24 - the “chart” assumes that you opened up the Passport flat and shot the whole thing - which I didn’t!

Not sure if that improves or hurts the profile.

I can tell you that if you just shoot the main panel with fiducials, using scanin’s -F argument works wonders. I’m planning on doing some edits to the rawpedia article to point people to that argument if they have issues with autodetection.