@ggbutcher After rereading your post, I am beginning to understand your quandary. Gamma is a part of the profile but, when the working and output gammas are different from the display gamma, it can be a source of confusion for the user.
A long time ago, I spent a week or so helping this lady restore a large set of old photos only to have her be extremely disappointed with and condescending to me because I did so with a different gamma than her printer. Fortunately, I never met this terrible person again.
Maybe you could do it the way some image editors do it, where you can assign the profile and have the display emulate the gamma, such that the histogram would follow the input/output profile but the visuals would be adapted. I might not be describing it right: just launch GIMP and observe what the Image > Color Management > Assign / Convert to Color Profile options do. Try with profiles with different gamma.
Yes! That’s it. I’ve been loading raws with prophoto/1.8 gamma because that’s what all the ‘cool kids’ (those that understand color) say, and producing display and output with 2.2 gamma appropriate for sRGB. I think LittleCMS has been protecting me here, insuring the gamma is properly considered in the transforms, but the histograms confuse me.
Maybe I should be using the same gamma throughout?
Thanks @afre, I’ve already spent quality time reviewing @Elle 's articles. I use her profiles to make internal profiles for rawproc, but I’m finding I probably need to extract the actual dcraw profiles instead. They all appear to be linear gamma, which doesn’t then reflect the application of gamma with -g. Still investigating, will write a more definitive (and correct) account later.
Edit: Just finished extracting dcraw icc profiles for selected colorspace/gamma combinations, and dcraw is indeed encoding the gamma used in processing. Also, the primaries are pretty close to @elle 's LargeRGB profiles, so apparently no images were harmed in the production of rawproc 0.5… Still need to understand the histogram differences, more to come.
I now know my problem. It is I don’t know enough about ICC profiles to program them…
I’d implemented a set of internal profiles, based on @Elle’s profile set. Turns out the primaries that get stored are just not the same ones I put in, so I’m missing something. So, I spent the afternoon removing them from the program, and now all works fine just using @Elle’s files.
Nothing since my return to photography has hurt my head like color management. The ICC profiles are an extremely useful abstraction, but even then you can mess up with nary a clue to guide your recovery. My younger twin granddaughter is very into color; at the age of 4 she actually chose a set of socks because they made her dress “pop”. Geesh… she’ll probably get all this way before I do…
@ggbutcher - Hi Glenn. I read through your github rawproc page and rawproc sounds like a very nice raw processor. My name seems to have been invoked several times in this thread, so I thought I would chime in. These points might be relevant to the general questions that I think you might be asking:
I saw a reference to “dark” in the github rawproc write-up and I think also above. If the image looks darker than it should, this almost certainly indicates a color management issue that should be dealt with by assigning the correct ICC profile (if that’s the problem), or else by finding and fixing whatever “calibrate/profile/color-manage” issues there might be in the workflow and display chain.
If the monitor isn’t calibrated and profiled (both require a color measuring instrument), then the monitor isn’t showing correct colors (unless the monitor has a decent sRGB preset and hasn’t drifted over time since it left the factory, in which case you can get by with using sRGB as the monitor profile).
I’m assuming rawproc uses LCMS color management to send the image to the screen. That is, LCMS converts the image RGB values from the image ICC profile to the screen ICC profile, and then the converted RGB values are sent to the screen. So there is absolutely no reason to worry about matching ICC profile “gamma” values. LCMS takes care of this. As relevant notes:
If the monitor’s color gamut is significantly smaller than sRGB, the monitor can’t show correct colors even for sRGB images.
For images in other color spaces, the monitor can’t show correct colors for any image colors that fall outside the monitor’s color gamut (as defined by the monitor profile, which underscores the need to have a good monitor profile). “Perceptual intent” doesn’t mitigate this problem, even if your monitor profile actually does have a perceptual intent table. Matrix monitor profiles (most common case) don’t have perceptual intent tables, in which case asking for “perceptual intent” actually gets you relative colorimetric intent, which simply clips all out of gamut channel values.
For the chain from camera raw file to interpolated image file, I would recommend the following (but take my recommendations with a grain of salt, or perhaps the whole salt shaker ):
Use the equivalent of “dcraw -4 -T -o 0” to output a linear gamma, “raw” color file.
Use LCMS to assign an appropriate camera input profile. Where to get camera input matrix profiles:
Make your own camera input profile.
Or use PhotoFlow (requires the linear gamma branch?) plus ArgyllCMS extracticc to extract the camera matrix profile for your specific camera and save it to disk.
Or use the dcraw adobe_coeff table to construct the actual ICC profiles.
Or use the augmented matrix profiles available from various free/libre raw processors.
Use LCMS to convert from the assigned camera input profile to the chosen RGB working space ICC profile (over the years I’ve encountered several issues with dcraw color management). My recommendation is to use Rec.2020 or ACEScg as the large-gamut RGB working space:
Both have fewer imaginary colors than ProPhotoRGB.
Both seem to produce “nicer than ProPhotoRGB” results for chromaticity-dependent editing operations (anything involving multiplication or division by a non-gray color; anything that involves using individual channel data, including non-luminance-based conversions to black and white).
At this point you have the digital equivalent of a “flat print”. Except that it’s actually more flat than a “wet” darkroom black and white “straight print” because film and paper involve shoulder and toe rolloff, and digital does not (unless your camera settings involved shadow lifting/highlight compression/similar algorithms).
For actually editing the “flat print” to suit one’s artistic intent for the image, use two versions of the chosen RGB working space profile, with the same chromaticities but different TRCs - one version with a linear TRC and one version with a perceptually uniform TRC. Depending on the editing algorithm and the artistic intent, use LCMS to convert the layer or entire layer stack between the two profiles as desired, or else program in the two TRCs with some way to flip back and forth between the two TRCs. In my opinion, these are the only two TRCs that are technically relevant to high bit depth photographic editing:
linear gamma (gamma=1.0), for proper color mixing (most layer blend modes including Normal, blurring, down-scaling, Levels, Curves, etc).
the LAB companding curve (perceptually uniform). This second TRC is for things like most “find edges” algorithms, soft light blend mode, adding uniform RGB noise, and such.
Regarding ProPhotoRGB:
The gamma=1.8 TRC was chosen by Kodak because it’s closer to linear than gamma=2.2, thus mitigating somewhat the gamma artifacts from processing non-linearly encoded RGB images. Also ProPhotoRGB is a very large color space, which also helps mitigate gamma artifacts. It’s better to just use a linear gamma RGB working space and avoid the gamma artifacts altogether.
The ProPhotoRGB primaries are better primaries for general editing than the sRGB primaries, for various reasons covered in the Kodak documentation (http://photo-lovers.org/pdf/color/romm.pdf). But they aren’t as good as some other editing spaces that have been promulgated since ProPhotoRGB was released. http://colour-science.org/posts/about-rendering-engines-colourspaces-agnosticism/ “ranks” various color spaces. But I’m also speaking from my own testing, and I encourage other people to make their own tests.
@Elle, thank you so very much for the detailed response. First, I’d like to thank you for posting your profile set, they’re the reference that has restored my confidence in my elementary color management logic in rawproc.
Essentially, for raw files delivered by libraw, I’m assigning a profile I create with LCMS using an exif dump of each dcraw -o X’s ICC. I use another LCMS tool I wrote that finds the primaries, black/white points and TRCs, and I use those to produce a (hopefully) equivalent profile. That is what rawproc’s workflow uses as the working profile for the rest of the edit.
rawproc is what i call a ‘chain’ editor, you add tools to the processing chain in whatever order you choose, and what you eventually save to a TIFF or JPEG is the result image at the end of the chain, Each tool has a checkbox that specifies that tool to be the display image, and it is there that the working space → display transform is conducted. At output, there is also a working space → output transform. Right now, I’ve deleted all my internal profile generation code except for the dcraw profile creation, and I’m using your profiles for display and output. BTW, that straightened out my ‘dark’ thing; I was using my generated prophoto profile for TIFF saves, and they apparently are wrongly specified, all’s well using LargeRGB-elle-V2-g18.icc.
So, I still need to get the dcraw assignment straight. Yes, the more I consider it, it is best to abandon the dcraw concoctions and work with the linear raw and assigned camera profile. I’m probably going to release rawproc 0.6 with the dcraw-assigned working spaces because I need to add code to manage a separate working space.
I’m going to take some time to digest all you’ve written, both in this thread and at your website. I’ve worked a long time with various geodetic reference frames; unless my dementia prevails, i should be able to eventually grok profiles…
Oops! I wrote the wrong thing: I already extracted the ICCs . However, that format isn’t human readable.
What I wanted to ask: how did you get to the point where you could compare the profiles with @Elle’s. I know I could probably examine dcraw.c and make-elles-profiles.c, but I am not a math or code wiz. It would be great if you, @Elle or someone more capable than I could guide me through the steps. Thanks.
@afre There are quite a few ways to extract information from an ICC profile in human-readable form (even including using a hex editor). Somewhere on this forum there is a thread that lists a lot of profile-related software. I think Rawpedia has a write-up.
The simplest way to turn a profile into something readable with a text editor is to use iccToXml:
SampleIcc Version 1.6.8, ICCXML Version 0.9.8
Usage: IccToXml src_icc_profile dest_xml_file
Thanks for the refresher. Lately I have been burnt out physically and mentally – experiencing lots of musculoskeletal pain, so I am not the sharpest knife in the cupboard. I will try IccToXml later and report back.
So you don’t have one universal working color space, but you edit in camera RGB? That makes it hard to do the same edits to images taken with different cameras, and might even give other problems, depending on the specific color space. I’d reconsider that and move from camera RGB to somethign like Rec2020 as Elle suggested.
I think @ggbutcher meant that he currently uses dcraw to convert from the dcraw matrix camera input profile to a dcraw output space (the “-o #” options), and then makes an ICC profile from the dcraw output space, for actually editing the interpolated image.
As far as editing in “camera RGB” I find this to be very useful for certain purposes, including setting different white balances for different portions of mixed lighting images, using layers and masks to choose which white balance is applied to which portion of the image.
I think this “multi-layer/multi-white-point/mixed-lighting” approach probably works for any camera input profile. But it’s easy to make a custom camera input profile (http://ninedegreesbelow.com/photography/well-behaved-camera-profile.html) that’s also a well-behaved (eg neutral gray axis, with white=Lab 100,0,0 and black=Lab 0,0,0), that can be used for other types of edits.
However, as @houz notes, using “camera RGB” can cause issues:
For editing operations that are chromaticity-independent, as long as camera RGB is also a well-behaved RGB working space, then these operations produce the same results regardless, assuming no precision losses from editing in such large and oddly shaped color spaces.
For other operations, well, it depends. If the goal is to retrieve individual channel information as captured by the camera, use “camera RGB”. If the goal is to split-tone using Curves, this operation is likely to produce unpleasant results in “camera RGB”, as will making single-channel gamma modifications.
What I wanted i my initial try at CMS was to use the dcraw output colorspace (-o n) as my working space. Thing is, libraw doesn’t make the dcraw ICC until time to save as TIFF, or so it appears, as I can’t find it in any of the data structures available after dcraw_process. So, my first attempt was to use LCMS to build internal profiles for each of the -o options, and assign them to the libraw image for subsequent processing. Actually, that seemed to work quite well, although I can’t really know for sure until I acquire the resources to calibrate my displays. The only thing vexing was when I went to save a TIFF with the working space profile, when I opened it the histogram was different than when output, but the image seemed identical. That’s when I started this thread…
I got back to goodness by abandoning my internal profiles and just using @Elle’s profiles. But, simply assigning her LargeRGB g1.8 to the dcraw -o 4 output just didn’t seem proper, so I’m going to make the following modifications before releasing 0.6:
input.raw.cms.rawprofile - name of a .icc to assign to the input image in place of any previously delivered profile This would be intended to be a camera profile, but could be a .icc equivalent for any dcraw-produced profile.
Implement a Convert Colorspace tool. As I described above, rawproc is a ‘chain’ editor, the GUI approximation of gmic or imagemagick. I’ll just implement a tool that can be added at any point to convert from the assigned colorspace to one in a supplied file. That’ll be the first tool applied in a workflow that starts with a camera profile-assigned image, to convert to a working colorspace for the subsequent tools.
I debated doing the working space conversion as part of opening the file, but doing it as a tool provides flexibility to apply color space transformations anywhere in the tool chain. Not sure for what, just seems useful, to bear-of-little-brain here…
I want to thank all who’ve responded here; your words have been extremely helpful in schooling me through another part of color management. I’m starting to get the mechanics; will need another couple hundred cups of coffee for the theory.//
What was it that felt wrong? The fact that you are not 100% sure that the profile dcraw uses is identical to the one you got from Elle? In that case you could use -o 5 to get XYZ from dcraw which is universally defined and then use lcms2 to convert that to the desired profile.
):
.
. However, that format isn’t human readable.