Color profile from Tiff file

Software RawTherapee
21

Yup. Sometimes doing this requires having an “intermediary/interim” profile designed to make some software not make undesirable assumptions. That’s part of the rationale for the pull request I linked (and the issue referenced in it) - RT’s old “profiling” system saved out a linear TIFF with no ICC profile. This would cause some software (such as Hugin) to assume it had an sRGB transfer function. I needed to fix this to handle a camera (Mi Sphere 360) with a fixed fisheye lens, as I needed to defish in Hugin prior to feeding to dcamprof.

sRGB primaries with linear transfer function (gamma = 1.0) seems to be the “best” tradeoff as far as minimizing the chances of an undesired colorspace or gamma transform.

I admit, I use GIMP so rarely that I’m not sure what will happen if you try to roundtrip your data through it. You may find it necessary to generate your “complete” color patch image with ImageMagick or some Python scripts.

One thing I just noticed: Your gamma-encoded example file provided in your OP is 8 bits/sample. While gamma encoding compensates for this to some degree, 8 bits + sRGB is going to have quantization errors here. You really need a 16-bit linear workflow.

Something is a little strange here, although maybe you’ve got a better feeling for what is “real” and “not real”:

image
image1920×1080 368 KB

Left is your “linear” TIFF interpreted as sRGB, right is with it tagged as being linearly encoded - it looks really washed out in a manner consistent with sRGB data being misintepreted as linear to me.

22

Off means None.

The abstract profile reinterprets the colors with respect to sRGB. If you use the sRGB values, you get the identity transformation, i.e. no change in the colors.

23

@Entropy512
I don’t have the software right now with me but I believe that saving with 16 bits is only available with linear encoding. That means that the picture quality is more difficult to preview. In addition, some useful options are not available, like adding a scale or comments. But this can be overcome in post-processing. Moreover, it’s not true 16 bits as the sensor is only using 12 bits.

Concerning the result, how have you managed to convert to sRGB with Gimp? The right picture is closer to reality. I didn’t mention it to not add confusion, but there is a saturation slider during the picture shot. This specific one was taken with no increased saturation, giving a desaturated look. I should have sent you another one.

24

Well all of the things you shot are greyscale, so saturation wouldn’t make a difference, but if it is indeed that it should be brighter/more “washed out” looking, then it was as follows:

exiftool "-icc_profile<=RTv4_sRGB_Linear_g=1.0.icc" 20230929_01_ORG_icc.tif

(I copied your TIFF to a second file so I could do a comparison)

The ICC profile file above is part of that pull request I linked, or you can go into RawTherapee and use the ICC profile generator - sRGB primaries and linear transfer function (gamma=1.0), and save that wherever.

One thing I don’t know is whether dcamprof will assume all input is linear, or pay attention to the transfer function of any attached color profile. It will, obviously, ignore any color profile information (primaries, etc) since its whole purpose is to determine the color behavior. :slight_smile: I’ve only fed it linear data, with and without an attached ICC profile.

Interesting that there’s a saturation slider - so I’m guessing this is not raw sensor data, it’s already been “cooked” color-wise to some degree?

25

Guillaume

Just for info.
This “ICC profile Creator” allows you to do pretty much what you want (or almost) in terms of output ICC profile.

image

Jacques

26

@Entropy512 @jdc
Wonderful, these are the ICC profile and tools I was looking for :star_struck: !

The Macbeth chart was assembled using gimp starting from the individual patches. Don’t pay attention to the gray background, which is a mask coming from another picture.

First test
Applying RTv4_sRGB_linear using exiftool; calculating profile using dcamprof
→ before calibration deltaE avg 9.05, max 14.36
→ after calibration deltaE avg 6.15, max 9.89

Second test
Adjust exposure using RT by -0.40 EV, input profile = output profile = RTv4_sRGB_Linear_g=1.0.icc;
calculating profile using dcamprof
→ before calibration deltaE avg 8.23, max 12.05
→ after calibration deltaE avg 4.64, max 8.52
The average is below 5 which is not so bad

Here is the obtained profile, and the assembled chart before and after calibration. The result is not perfect but it is very promising. This dataset suffers from small light gradients visible on the white patch.
gv_Zeiss_508_sat0

Zeis_508_sat0_RT_lin

Zeis_508_sat0_RT_final

It seems. The raw data file is CZI, which is unkown to RT. The specifications can be found at CZI Image File Format . I need to reacquire a dataset with full saturation.

Additional question
The microscope is equipped with a light polarizer which has a strong impact on glare reflection. All the pictures above have been acquired with parallel polarization, corresponding to maximum glare. Do I need to use perpendicular polarization, which will avoid glare ? What will be the impact of glare on profiling ?

dcamprof make-profile -g 
Reading target...
Glare test before glare matching...
Warning: large dynamic range difference detected. Likely glare issue.
Camera G on darkest patch(es) is 136.9% lighter compared to observer Y.
  Y dynamic range is 4.82 stops, G dynamic range is 3.57 stops, difference
  1.24 stops. A small difference is normal, while a large indicates that there
  is glare.
Glare-matching target...
  Minimum Y changed from 0.034068 to 0.085042. Glare was modeled into the spectra.
Testing glare after adjusting reference values (camera G and observer Y should
  be close).
Camera G on darkest patch(es) is -5.1% lighter compared to observer Y.
  Y dynamic range is 3.50 stops, G dynamic range is 3.57 stops, difference
  -0.08 stops. A small difference is normal, while a large indicates that there
  is glare.

Regards,

27

@Guillaume

Too bad the Raw file is not documented and recognized by RT. This would (maybe) improve the result ?? :sweat_smile:

The problem of reflections and differences in brightness is one of the major problems when we want to calibrate…
But, by putting a polarization - which will of course reduce or cancel the reflections, what is the transformation that the illuminant undergoes?

If we want to obtain good results, but here I speak like a book, the illuminant should have a CRI of 100, like Daylight, or Tungsten stdA.

Look at these diagrams which represent illuminant versus wavelength in nm (blue to red), the ones with a CRI of 100, and the others like the LEDs at the bottom right.
As soon as the CRI is less than 100, “holes” are created which will lead to poorer reproduction – if at all – of these colors. Unfortunately there is no solution.
https://rawpedia.rawtherapee.com/Color_Management_addon/fr#Diagrammes_des_illuminants_et_Color_Rendering_Index_(CRI)

What is very comforting is that we sense the optimism and progress in your words. :+1:

Jacques

28

Yeah, definitely not currently supported, and probably something that is so niche that it would likely not be worth the effort to directly support it.

Niche applications like this are why I wrote a few conversion tools: GitHub - Entropy512/pyimageconvert: Various Python image conversion scripts - I’m a firm believer in the “Unix way” of doing things, instead of having huge monolithic tools that do everything, have an interoperable ecosystem of tools focused on a specific function. Right now I’m guessing neither libraw nor imagecodecs support CZI as it’s THAT niche, but with a few CZI samples I could probably whip up a CZI-to-DNG converter as a derivative of my existing tools. I’ve gotta give props to Zeiss - at least in theory they have a documented file format and open source libraries to support it. pylibCZIrw · PyPI should make this easy? Need to wake up to futz with it. I’ll check RPU ( https://raw.pixls.us/ ), if RPU doesn’t have them I’ll need a few examples.

As to glare - usually glare is known to cause issues. There are methods for attempting to detect/compensate for glare, but these only apply to “whole-card” shots.

29

Hi @Entropy512,
I agree, it’s really a niche application. I wouldn’t you to waste your time on it for a single person. But if you want some Zeiss CZI files for the challenge, I can provide you a few ones. However, it will probably not be representative of all the possibilities of the format. Just tell me what kind of photographs you need (chart, powder, object…). There is also another available library: czifile · PyPI. I will have a look with python.

More about saturation: I don’t know if it’s hardware or software related. My interpretation is that’s a sort of electric gain, which is then saved to RGB values without any calibration. When applying saturation, L*a*b* values move from 73.6/2.0/45.6 to 82.2/1.9/90.4 for yellow patch (D4), and from 64.9/-19.9/-2.3 to 72.4/-49.0/-4.7 for bluish green patch (F1). These 2 patches are along the a* or b* axes. One can observe roughly a doubling of a* or b* values and an increase of luminance.

I acquired a full set of ColorChecker patches on the stereo microscope with saturation. The individual gamma-2.2-encoded TIFF files were processed as follows:

  • RawTherapee preprocessing using input profile = output profile = RTv4_medium and shrinking to 500 pixels;
  • Gimp assembling and global shrinking to 1200 pixels;
  • RawTherapee conversion with input profile = RTv4_medium, output profile = RTv4_medium_linear, exposure -0.25 EV adjusted on neutral 5 patch (D4);
  • DcamProf profile creation;
  • RawTherapee input profile attribution and export to RTv4_sRGB;
  • Darktable color check using color calibration module.

Here are the results:
→ before calibration deltaE avg 5.05, max 11.52;
→ after calibration deltaE avg 3.81, max 7.44.
This is an improvement over the previous trial, but it’s still a bit high. This dataset has a homogeneous illumination but suffers from blue RGB values at 0 for yellowish colors, highlighting oversaturation. The next improvement could be reached using lacam16n.

Concerning the abstract profile, I just realized that I was improperly using it to compensate a mismatch between a linear profile and a gamma-encoded image, or the other way around.
EDIT: Is there a way to convert a linear profile to a gamma-encoded one?

Have a great weekend
Guillaume

30

Hello,
I tried to use a custom.json file but the result in RT is identical to the linear tone curve. Is there an expert on dcamprof here? @ggbutcher maybe?

"dcamprof.exe" make-icc -t custom.json -W -n "target_dcam_g2.2" "target.json" "target_dcam_g2.2.icc"

custom.json

{
    "CurveType": "Spline", // must always be spline
    "CurveHandles": [
      [ 0,0 ],
      [ 255, 255 ]
    ],
    "CurveMax": 255,
    "CurveGamma": 2.2 // use the string "sRGB" to specify the special sRGB gamma.
}
31

Hello Guillaume

For me, deciphering a Raw is Chinese :wink:

I think (I don’t know this software) that the saturation slider does something else… otherwise the luminance would not move.

Overall happy that you are progressing, I admire your pugnacity. :grinning:

Aberrations in colorimetry are probably due to “holes” in the illuminant of the LED lamp (no solutions).

It is more than normal that there is a software learning phase.

To convert a profile (ICC or DCP) I don’t know, moreover I don’t know Dcamprof.

Good day

Jacques

32

Just tried to mess with your custom.json, I think I need the particular target.json you’re working with…

That said, I’m not sure dcamprof will, at least easily, allow one to “building-block” construct an ICC profile. I know I could do it with a C++ program using the LittleCMS library, but that’s probably a bridge too far.

And all that said, if you want working profiles with various TRCs, you might take a look at Elle Stone’s “well behaved” profile collection:

https://github.com/ellelstone/elles_icc_profiles

The profile names don’t always align with common definitions due to licensing foo, but you could start with:

LargeRGB = ProPhoto
ClayRGB = AdobeRGB

33

So, with a little time on my hands, wrote this:

/*

Requires LittleCMS2:
$ sudo apt install liblcms2-dev

Compile with:
$ gcc -o trcswap trcswap.cpp -llcms2

Then, run with:
$ ./trcswap linearprofile.icc gammaprofile.icc

*/


#include <stdio.h>
#include <lcms2.h>

int main(int argc, char ** argv)
{
	cmsHPROFILE inprofile;
	cmsToneCurve *g;

	if (argc < 3) {
		printf("Usage: trcswap [infile] [outfile]\n");
		return 1;
	}

	inprofile = cmsOpenProfileFromFile(argv[1], "r");

	g = cmsBuildGamma(NULL, 2.2);   //makes a gamma 2.2 tone curve

	cmsWriteTag(inprofile, cmsSigRedTRCTag, g);
	cmsWriteTag(inprofile, cmsSigGreenTRCTag, g);
	cmsWriteTag(inprofile, cmsSigBlueTRCTag, g); 

	cmsSaveProfileToFile(inprofile, argv[2]);

	cmsCloseProfile(inprofile);
}
34

@ggbutcher That’s amazing, thank you so much :+1: :star_struck:. I will give it a try tomorrow.

@jdc Thank you also for your encouraging words and your help, publicly and privately :slightly_smiling_face:. I tried to use lacam16n but it’s rather difficult to adjust the primary colors. Do you think you could try if I provide you with the picture?

Both of you, do you know if RT profiles are well behaved, ie color balanced and normalized?

Good night, it’s time to sleep in France.

35

@Guillaume

No problems in principle. You send me the corresponding TIF file(s) with the shooting conditions (if you know them) cf. Illuminant… And if possible if you have them the test pattern definition files (Colorchecker24) text files where there is the patch number, the Lab values and the XYZ values

I can’t guarantee anything, but I’ll try.

As for the quality of the ICC profiles, they are built in the same way as those of Elle Stone (I even copied part of the code).
The difference is that instead of choosing from a list, you can create specific profiles with a dedicated tool.
A particularity (as Elle Stone also did) is to favor the TRC “sRGB gamma=2.4 Slope=12.92” - the one we usually use for our screens and TIF output to Photoshop or Gimp.

On the other hand, if you modify the basic primaries and the illuminant linked to it (for example Prohoto primaries, and D50), you will generate something unconventional. That doesn’t mean it doesn’t work.
But be careful, it’s obvious, not to confuse, “Working profile” and “Output ICC Profile”. “Working profile” which I think is a (very) bad choice of vocabulary, is not an ICC profile, but a 3x3 RGB=>XYZ conversion matrix.

In the code of “ICC profile Creator” as in "Abstract profile"there is a common part to develop the 3x3 matrix XYZ. We start from the primaries (standard or personal), we assign an Illuminant (D50, D65…), the system will calculate the matrix taking into account a chromatic adaptation (by default Bradford), to bring the whole thing back to D50.
This code of this matrix is “hard code” in the C++ code to make the Working Profile. ICC Profile Creator or Abstract Profile (or the same in ‘lacam16n’), adds a TRC to make it either an ICC profile or a virtual profile.

Jacques

1 Like
36

Ah, another @cgohlke module. Those are the basis of most of my image conversion scripts, great starting point here. Right now for the purposes of poking at the format, one shot of nearly anything in CZI format along with an sRGB TIFF of the same subject to serve as a sanity check will work.

Making a camera profile with DCamProf wasn’t intended for this use case, but PROBABLY could be used here?

Either way, it’s a very niche use case, but I’m sure some other microscope user is looking for something similar. :slight_smile:

1 Like
37

Thinking about how to organize a profile creation program using LittleCMS. It’s been done before, but I might try incorporating input formats like dcamprof JSON and exiftool -S output. No traction, just cogitation right now…

38

Yeah that’s another approach, but probably some easier options with the building blocks we already have.

One thing I’m curious about (and won’t know for sure until we get an example file) is whether CZI “raw” images are demosaiced or not. czifile/czifile/czifile.py at master · cgohlke/czifile · GitHub doesn’t do any demosaicing, so if a CZI file is Bayer-mosaiced, then I’ll probably need to add CZI support to libraw2dng.py so that it will create a mostly-properly-tagged DNG. “Mostly” being “throw in an sRGB color matrix which is clearly wrong but better than nothing, since this is intended for profiling runs and/or use with a DCP to be generated later”.

39

Hi @jdc, wonderful to know that ! :+1:
I just realized that the white and black points of my picture are not at all accurate. I need to work on TRC before sending you the pictures.

Hi @Entropy512 , here is the link to download a CZI file. The TIF file and the metadata files are included. The photograph is not a calibrated target but just paper printed with color patches. An encoding gamma of 0.45 has been applied. RGB colors are not calibrated.

Hi @ggbutcher, that would be very useful in RT, and also a checking module to compare colors to a known target.

Thank you all for your helpful insights!

40

Thanks! Looks like the CZI files are demosaiced. czi2tif writes BigTIFF which exiftool hates. Since it’s demosaiced, just converting to a properly tagged TIFF with a minor variation on pyimageconvert/imagecodec2tif.py at master · Entropy512/pyimageconvert · GitHub is probably the best approach for now to determine the next step (e.g. whether this is camera native color space or has already had white balance of some sort applied).

Edit: So far, it looks like the CZI data isn’t really raw. It’s demosaiced, AND it has clearly had a colorspace conversion applied. There’s no good way to do a side-by-side conversion in GIMP these days, but converting the 8-bit rendition to 16-bit linear using an sRGB assumption is nearly identical to the CZI data being treated as linear data with sRGB primaries with a white point of 4096. Any minor differences might be the difference between a pure gamma transfer function in your 8-bit TIFF and an actual sRGB TRC. I’m trying to find a way to get the white point from the XML metadata, right now I’m hardcoding a scale factor of 16.

This is interesting in the metadata XML:

            <PanoramaSetup IsActivated="false">
              <SubDimensionSetups></SubDimensionSetups>
            </PanoramaSetup>

Does this microscope have some sort of automated panorama/stitching mode? That might make a full ColorChecker capture easier.

Edit: Rough initial CZI to linear TIFF converter at GitHub - Entropy512/pyimageconvert at czifile

Edit 2: @ggbutcher check out page 3 of https://microscopy.wfu.edu/wp-content/uploads/2020/11/a9986ff9-9f2e-45c4-ba7e-03bcfdd26a8a.pdf ! - too bad there isn’t a raw table of that SSF data. Not like it matters right now since the “raw” in the CZI file is not only demosaiced but white balanced/colorspace-converted unless @Guillaume finds a setting that fixes that. :slight_smile: