Mathematically right values in decomposing to LAB ?

I’m not sure what you mean by “binning”.

Suppose we have four values (0, 1/3, 2/3 and 1), and we downsize by “binning” to two values. What are those values?

IM won’t do any gamma correction unless you tell it to. There are a very small number of exceptions to this rule, and “-set colorspace sRGB” takes care of those. This operation doesn’t change pixel data, it only changes metadata, so IM won’t internally convert the image to sRGB.

Beware that some file formats can’t store gamma metadata, and for those formats that can, some software ignores it.

I guess that @MarionGaff1’s “no color correction” means the image is recorded as linear RGB, rather than non-linear sRGB. Testing with and without should answer that question.

I didn’t make measurements, but those scans look very flat - nice!

If you upload a similar low resolution scan of the other side (just the scan without any profile), if it would help I would try to make an ICC profile for the scanner using ArgyllCMS, just to see if results look believable without any surprises in the procedure. I’ve never profiled a scanner, but it can’t be that different from profiling a camera.

If you’d rather experiment first before sharing (totally understandable!), here’s the relevant ArgyllCMS “how to” page:
http://argyllcms.com/doc/Scenarios.html#PS1

And here’s where to download ArgyllCMS if you don’t already have it installed (or don’t have the latest version): http://argyllcms.com/ - just scroll down to where it says “Downloads” - there are separate downloads for various operating systems.

Well, downsizing is a standard way to reduce noise. For a camera raw file it can be done by asking for the type of interpolation that just combines the RGBG pixels into one “RGB” pixel, producing a smaller but cleaner image. I haven’t ever used that type of interpolation. But in GIMP there is “linear” interpolation, that seems basically to do something like taking the median of the pixels surrounding each pixel (including the actual pixel). Asking for 50% reduction in width and length does produce a much cleaner image, at least when starting from a non-demosaiced image (it was an experiment, and worked really well as the starting file really was very noisy).

This “binning” - and maybe I’m using totally the wrong word, but hopefully I’ve explaing what I’m trying to point to by using the word - is different from using a scaling algorithm that tries to preserve detail. The assumption is that noise and detail look an awful lot alike to a scaling algorithm, so if size reduction while also decreasing noise is the goal, don’t use algorithms that try to preserve detail.

Here is what I got scanning the other side (200 dpi, no color correction, 48 bits colors, automatic level of exposure is intermediate, Tif format). The same as for the “gray side” : 2 files : one with ICC integrated, one with NO ICC profileTarget%20chart%20color%20-%20no%20correction%20-%20200%20dpi%20-%20ICC%20profile006|nullxnull Target%20chart%20color%20-%20no%20correction%20-%20200%20dpi%20-%20no%20ICC%20profile005|nullxnull

I will use ArgyllCMS later on, as soon as I can find the time to do so, because I have several experiments in parallel :smiley:

To me, “binning” means categorizing a large number of pixels into a smaller number of “bins”, and then processing all pixels in the same bin in the same manner.

But never mind the terminology, I’m looking for a precise definition of what you want, so I can suggest the IM operation(s).

IM can do median, if that’s what you want, eg “-median 3x3” changes every pixel to be the median of the 8 neighbours and itself. It doesn’t change the image size.

If your answer to my question about (0, 1/3, 2/3 and 1) was (1/6, 5/6) then the operation would be “-scale 50%”.

What I’m looking for is an imagemagick down-sizing command that is something like the code in this file: https://gitlab.gnome.org/GNOME/gegl/blob/master/gegl/buffer/gegl-sampler-linear.c, which according to the online documentation (https://docs.gimp.org/2.10/en/gimp-tools-transform.html#gimp-tool-transform) takes the average of the four nearest pixels, not including the actual pixel. It does work well to downsize when the goal is noise reduction, when starting from a raw file that hasn’t been demosaiced, but instead simply output to disk without first doing an interpolation.

But I guess I shouldn’t have used the word “binning” - sorry! And given that the scanner actually produces RGB, no interpolation needed, maybe some other down-scaling algorithm would work better.

But again, the goal of the down-scaling, down-sampling? is noise reduction rather than preservation of detail.

Wouldn’t a gaussian blur do the trick?

With work, I expect that any Gimp process can be replicated in ImageMagick, though it might need C code.

If the Gimp process gives a good result, I would simply use Gimp. Gimp can edit images non-interactively. (At least, it used to be able to. I haven’t tried this recently.) So it could batch-process the images.

IM’s common downsampler is “-resize”, which has infinite varieties, using “-filter”. Most people want to add sharpness, but I expect some varieties will do the opposite.

If noise is a problem, I would treat that as a separate issue to downsampling. Denoise first, then downsample. Different types of noise need different treatment. Maybe a simple blur to remove or reduce high-frequency data. This will also blur edges, which I guess doesn’t matter here.

I’m currently playing with techniques that limit outliers. For example, calculate the mean and standard deviation in a small area around every pixel. If the pixel is outside the range (mean +/- k* std_dev), cap it to be within that range.

But will noise be a problem when scanning orange peel? I don’t know.

Sorry about leading you all down a rabbit trail with binning and down sampling. I admit I haven’t been exactly concise with my posts but glad the hints lead to places. I blame insomnia. :blush:

The first question we need to ask is
– What are the native input properties of the scanner? That would be its optimal setting.

The next question we ask is
– What data resolution (spacial, temporal, etc.) and precision would best represent the thing being observed? There is usually a sweet spot: more or less information than necessary usually introduces more problems and therefore requires more considerations (time and energy).

This is why reading dozens if not hundreds of papers of previous research is vital and why I keep on pushing on my other points. The design of the experiment and research guide what we do.

The point of down-sizing in this discussion is that the scanner produces files that are very large. A first step towards making the file size manageable is to downsize. Some down-sizing operations actually add sharpening. That would be a mistake for the current use case, would just add artifacts.

I think GIMP’s linear scaling is a really good choice for the current use case. Nohalo/lohalo are very CPU-intensive, “cubic” adds sharpening, and “none” just takes every other pixel when you ask for a 50% size reduction, so isn’t really any different than selecting a lower-than-native scanning resolition.

I don’t know how to use GIMP from the command line, and I bet trying to open the full-size scan in the GIMP UI would be difficult, depending on one’s computer and amount of RAM.

Scaling in GIMP is actually done using GEGL, which also can be used directly at the command line. I don’t know the command. People on GIMP IRC might know, in particular Pippin would know for sure. Another place to ask Pippin about using GEGL a the command line would be on the GIMP-dev mailing list or the babl/GEGL-dev mailing list.

@ggbutcher - A small gaussian blue can be useful as a preliminary step before downsizing. But gaussian blue is not a down-sizing operation, which I’m sure you know :slight_smile: so clearly there’s been a miscommunication somewhere. I’ll take the blame!

Gaussian blur technically speaking actually samples the entire image for every pixel in the image, though in practice various short-cuts are taken. The point of down-sizing in the current use case isn’t to blur the image but to down-size the image to a more manageable file size.

I don’t think it’s a rabbit hole. Rather it’s a very interesting and useful/practical topic, especially today when cameras and scanners can produce huge files, much huger than perhaps the user in any way wants or needs. I’m thinking about starting a new thread on the topic unless someone else does so sooner than I get around to it :slight_smile: .

@MarionGaff1 - I did make several different types of ICC profiles for your scanner using the reduced-size target chart you uploaded. I’ll try to post some images and the relevant commands later today or more likely not until tomorrow morning (US east coast time). In the meantime I would suggest a couple things when scanning the chart:

It would be good to put a piece of black paper behind the chart, large enough to cover the scanner bed. It might not matter for a scanner - I just don’t know. But for a camera, surrounding the target with white just causes camera “veiling flare” - I think that’s the right term. So to be on the safe side it might be better to just eliminate the possibility by using a black background behind the target chart.

Also, speaking of the white background, I measured a small amount of difference at one end of the background vs the other end, around L=92-93 at one end, down to L=89-90 at the other end. I’m not sure how much this might affect results, but maybe flat-fielding - again, something I’ve never done - before making the ICC profile might be a good idea. The fall-off in intensity seems uniform and consistent for all the various scans.

On the other hand, maybe flat-fielding would be more trouble than it’s worth. It might be interesting to put the target in the center of the scanner and scan it twice, spinning it 180 degrees for the second scan, and comparing resulting ICC profiles from each scan.

Does anyone here have experience with using flat fields? I know RT makes the process easy, but the online documentation makes it seem as if RT only works with raw files for flat-fielding.

I was thinking more of taking a patch from the image and blurring it, which would ‘average’ out the pixels to allow taking a more consistent LAB value.

Thank you so much!

Here are the scans that I obtained without ICC profile integrated.Target%20chart%20color%20black%20backG%20-%20200dpi%20-%20no%20correction%20-%20exposure%20interm%20-%20no%20ICC|nullxnull Target%20chart%20color%20black%20backG%20-%20200dpi%20-%20no%20correction%20-%20exposure%20interm%20-%20no%20ICC004|nullxnull
I still did not figure out if I should register my scans with ICC profile integrated or not.
If I register all my scans with ICC integrated, then I will not be able to “replace” the ICC profile integrated by the scanner by my own “customized” ICC made with the target chart OR are the ICC profile integrated by the scanner and the image “glued forever”?
If I understand well what ICC profiles are, they are made to give information about the frame of reference to use to interpret properly the coordinates that define each color.
So, if I don’t integrate any ICC profile, how are the softwares able to open the scans with the right colors (right colors being information given by the scanner).

And with ICC profile (black background 200 dpi)


Hi @MarionGaff1 - whatever your scanner software or user manual might be telling you, there’s no embedded ICC profile in any of the various scan files labelled with and without profiles, at least not in the ones that I’ve downloaded and checked, that had file names indicating there was a profile embedded.

You might want to open one supposedly with, and one without a profile embedded by the scanner - both scanned without repositioning the target or opening the window, etc - and calculate the “difference” which should be zero for all pixels. Visually there doesn’t seem to be any difference. Imagemagick does allow this “difference” operation from the command line. Example imagemagick commands can be found here: https://ninedegreesbelow.com/photography/lcms2-unbounded-mode.html - but that’s an old article, perhaps syntax has changed in the meantime. Don’t bother reading the text unless you just happen to want to, just locate the sample imagemagick commands. Or ask @afre or @snibgo for the syntax.

When using command line tools such as exiftool or ArgyllCMS to examine metadata for example to see if there’s any embedded ICC profile), life is hugely simpler if file names don’t have any spaces - use underscores or hyphens instead of spaces.

Doing a quick check, the darkest patches of a scan of the target chart with the white background are consistently a bit lighter compared to a scan with the dark background. But the Lab Lightness difference is less than one. Of course comparing two scanned files in such a slapdash fashion isn’t worth much from a statistical perspective so I’ll leave it to you to do further experimenting, or else just use the black background. You might also want to experiment to see what difference using a blue vs a black background for the orange peels might make in terms of possibly scattered light.

You are absolutely right. Software needs an embedded ICC profile to properly interpret the colors:

  1. First scan the target chart.
  2. Then use ArgyllCMS to make the scanner input profile. ArgyllCMS profile-making utilities don’t need an input profile.
  3. Then use Imagemagick or GIMP or other command line utility or image editing software to embed the scanner input profile into the scan of the target chart and also embed the input profile into all the scans of the orange peels and anything else you might need to scan.

Here’s a sample command to use exiftool to embed an ICC profile:

exiftool "-icc_profile<=/path/to/your/scanner-profile.icc" /path/to/your/scanned-target.tif

So after you make the input profile, if the scanner profile is named “scanner-profile.icc” and the scan of the target chart is named “scanned-target.tif”, and both are in the same folder, then cd to the folder and type this:

exiftool "-icc_profile<=scanner-profile.icc" scanned-target.tif

And the usual warning: Any time someone hands you a command to type into a terminal, test, test, test and test again to verify that the command is working as expected. Even with trustworthy sources of such commands, typos happen, people forget the right syntax, and syntax does vary from one OS to the next. I don’t know Windows or Mac syntax. I only know Linux syntax.

Here’s the exiftool web page: https://sno.phy.queensu.ca/~phil/exiftool/
There’s an exiftool forum for asking about syntax and such.

ImageMagick, LCMS, ArgyllCMS, and no doubt quite a few other softwares all have utilities for embedding ICC profiles at the command line. Exiftool isn’t the only option.

Well, that’s all the typing I want to do for now. My apologies for running out of steam before uploading the scanner profile and sample commands :slight_smile:

Those ImageMagick commands should work fine for IM v6, or IM v7 if installed “with legacy commands”.

I recommend not using “composite” for anything. That is a very old tool, long deprecated. Instead, use “convert … -composite”, with the two input images in the opposite order. That way, a sequence of IM commands can be readily squished into a single command.

@snibgo - thanks! I doubt I’ll ever redo those commands to update them to IM v7. But I’ll add a note to my article with the information you provided in case anyone is ever actually tempted to try to follow along the steps in the article.

I don’t want to derail this thread, so I’ll just mention that the v7 command is “magick”, rather than “convert” or “composite”.

The IM forum https://www.imagemagick.org/discourse-server/index.php is a good place to ask questions specific to IM.

@MarionGaff1 - here are example commands for making a profile for your Epson scanner, along with some comments about the scanner and the target chart:

When making a scanner input profile, the first step is to create a “ti3” file (http://argyllcms.com/doc/File_Formats.html#.ti3). The ti3 file contains information read from the target chart. Creating the ti3 file requires three different files:

  1. A scan of the target chart. In the example commands below the file name for the target chart scan is “epson-gt2500-spyderchecker24.tif”.

  2. A “cht” file that specifies the locations of the various patches of the target chart. This file is supplied by ArgyllCMS. It’s in the “ref” folder. The “cht” file for the SpyderCheckr24 is “SpyderChecker24.cht”.

  3. A reference file for the target chart that supplies the CIEXYZ values for each patch. There is a reference file for the SpyderChecker24 in the ArgyllCMS “ref” folder, with the file name “SpyderChecker24.cie”.

    You can open the “.cie” reference file with a text editor and see that the create date is Dec 19, 2011. The CIEXYZ values might or may not be accurate enough for your specific SpyderChecker24 target chart. You could measure the values yourself, or download similar information from the datacolor website, or perhaps you were already given such a file along with your SpyderCheckr24 target chart. Hopefully your target chart isn’t too old and has been stored in such a way as to minimize color changes, which do happen over time.

    There is a bit of information in the “cie” reference file that says it’s for D65 light. I don’t know what difference this bit of information might make. I’ve only ever made and used an IT8 camera target chart, for which I used a reference file for D50 light, and did my best to make the target chart shot under roughly D50 lighting.

To create the ti3 file - that will then be used to create the actual scanner profile - put all three files in the same folder. Then at the terminal cd to the folder and type the following commands, modified of course to reflect whatever file name you’ve given the target chart shot:

scanin -v -dipn epson-gt2500-spyderchecker24.tif SpyderChecker24.cht SpyderChecker24.cie

The “-v” switch just says “spew out a lot of information about what’s being done”. The “-dipn” switch says “also output a little diagnostic image that shows where scanin located the various patches”. The resulting file “diag.tif” is useful to make sure scanin actually did correctly locate all the color patches on the target chart, and also shows which color patch goes with which patch ID in the reference file.

Here’s what the “diag.tif” looks like (this is a reduced-size jpeg):
diag

The next step is to use the “colprof” utility along with the ti3 file to make the actual ICC profile. ArgyllCMS can be used to make several different types of ICC profiles, including XYZ LUT, LAB LUT, and shaper matrix profiles, with many possible variations on each type of profile controlled by using various switches including the “-q”, “-a”, “-r”, “-u”, and “-ua” switches. The colprof documentation explains each type of profile and what all the various switches do: http://argyllcms.com/doc/colprof.html

If I were making a “general purpose, use the scanner like a camera” profile for the Epson scanner using the SpyderCheckr24 target chart, this is the command with the specific parameters that I’d use - notice you don’t type “.ti3” after the base file name for the previously generated ti3 file:

colprof -v -qm -al -r0.5 -u -D"epson-gt2500-spyderchecker24-qm-al-r050-u" -O"epson-gt2500-spyderchecker24-qm-al-r050-u.icc" epson-gt2500-spyderchecker24

Which writes out an ICC profile with the file name “epson-gt2500-spyderchecker24-qm-al-r050-u.icc” and also prints out some interesting information to the terminal, including a summary error report.

  • The -qm, -al, and -r0.5 options are actually what the colprof command does by default, and ask for a medium quality LAB LUT profile, using 0.5% as the average deviation for the “device+instrument readings”.

  • The “-D” option allows you to put any description you want into the description field in the ICC profile and the “-O” option let’s you specify the ICC profile file name. When making a series of ICC profiles it’s nice to use these fields to keep track of things like “which device, which target chart, shot/scanned under which conditions (eg using or not using a blue or black background), using which ti3 file, and which switches, etc”. I always make the “-D” and “-O” strings match because a lot of software doesn’t show you the file name for ICC profiles, instead only shows you the description field, when you are trying to see what profile is actually embedded in an image.

  • The “-v” switch asks colprof to print extra information including a summary error report.

According to the terminal output, for the resulting ICC profile, “peak err = 1.953983, avg err = 0.557773”. Whether this peak and average error is sufficient is not something I can really comment on. But it’s worth using a couple of other ArgyllCMS tools to investigate a bit further, starting with “profcheck” (http://argyllcms.com/doc/profcheck.html) which can output a complete report on the error for each patch, using several different ways to calculate the error. Here’s how to get the default error calculation for each patch:

profcheck -v2 epson-gt2500-spyderchecker24.ti3 epson-gt2500-spyderchecker24-qm-al-r050-u.icc

The patch with the highest error is B1, which is a blue patch. For C1 - the “orange color” patch, the error is 0.606490. The other ways of calculating the error provided by profcheck are worth investigating. Also, I didn’t check, but for other types of profiles, I’m guessing that which patch has the highest error will vary depending on the profile type that you choose to make.

You can get arbitrarily small errors for the colors in the target chart by asking for a higher quality ICC profile (-qh or even -qu) with increasingly smaller “-r” values. Unless you actually have an independent measure of the errors in the target chart, all that using a small value for “-r” accomplishes is curve-fitting to the target chart patches. If you assume a smooth variation in device response then the ArgyllCMS “xicclu” utility can provide a visual guestimate of “curve fitting” (xicclu):

xicclu -ir -fif -g epson-gt2500-spyderchecker24-qm-al-r050-u.icc

which produces this, which to me looks believably smooth for an RGB device:

xicclu-graph-epson-gt2500-spyderchecker24-qm-al-r050-u_dot_icc

Here’s what the scanned target chart looks like when first opened in GIMP, before assigning the newly made scanner profile - as the image doesn’t have an embedded ICC profile GIMP assigns GIMP’s built-in sRGB profile:

epson-gt2500-spyderchecker24-assume-srgb

Here’s what the scanned target chart looks like after assigning the newly-made scanner profile:

epson-gt2500-spyderchecker24-scanner-profile-assigned

Assigning the profile makes the colors a little brighter. But it seems to me that the scanned target chart has already had some sort of TRC modification done before the “raw” RGB values were saved to disk during the scanning process. In other words, this doesn’t look like an actual raw scan, some sort of preprocessing seems to have been done by the scanner.

Switching topics, there really aren’t enough sample points in the target chart to justify making a LUT profile. But when I made a shaper matrix profile the max and average errors were higher:

colprof -v -qm -as -r0.5 -u -D"epson-gt2500-spyderchecker24-qm-as-r050-u" -O"epson-gt2500-spyderchecker24-qm-as-r050-u.icc" epson-gt2500-spyderchecker24
peak err = 11.608028, avg err = 3.449169

and the resulting “xicclu” graph was very “squiggly” in a way that one wouldn’t expect a linear device to be - I’m assuming scanners are inherently linear devices but maybe I’m wrong:

xicclu-graph-epson-gt2500-spyderchecker24-qm-as-r050-u_dot_icc

FWIW, a truly linear device such as many cameras when shooting raw has a much more deeply downward-curved xicclu graph. A straight line xicclu graph would mean the device itself has a perceptually uniform responses to light.

The ArgyllCMS documentation (Argyll Usage Scenarios) says this:

If the purpose of the input profile is to use it as a substitute for a colorimeter, then the -ua flag should be used to force Absolute Colorimetric intent, and avoid clipping colors above the test chart white point. Unless the shaper/matrix type profile is a very good fit, it is probably advisable to use a LUT type profile in this situation. . . .
For the purposes of a poor mans colorimeter, the following would generally be used: colprof -v -D"Scanner A" -qm -ax -ua scanner

I don’t have a clue what’s involved with using a scanner as a colorimeter, but here’s the command rewritten to include descriptions and such:

colprof -v -qm -ax -r0.5 -ua -D"epson-gt2500-spyderchecker24-qm-ax-r050-ua" -O"epson-gt2500-spyderchecker24-qm-ax-r050-ua.icc" epson-gt2500-spyderchecker24

which reports peak err = 7.720558, avg err = 2.309703.

Here’s the xicclu graph, which to me looks very odd:

xicclu-graph-epson-gt2500-spyderchecker24-qm-ax-r050-ua_dot_icc

And here’s what the target chart looks like when this “colorimeter” profile is applied:

epson-gt2500-spyderchecker24-colorimeter

2 Likes

Disregard the advice about creating a cLUT profile for using a scanner as a “poor man’s” colorimeter - it is given on the assumption that a reasonable sized test chart was being used, i.e. a ColorChecker DC or a custom chart with a lot more patches. If you have a 24 patch ColorChecker, then stick to shaper/matrix.

Hi all!
Thanks a lot to you all for your precious help! I am struggling experiments and reports. Sorry, I did not answer before.
I had hoped I would have time to go deeply into Image Magic earlier, but it appears that my “free-time” is now preciously kept to have a bit of sleep!
@Elle You did an impressive work and very detailed and precise instruction manual. I am deeply thankful.
Fingers crossed, I should have time to go back to colorimetry next week, and will get back to you! :slight_smile:

1 Like