The feeling is nearly universal, I’m sure. 
1 Like
@ggbutcher - Hi Glenn. I read through your github rawproc page and rawproc sounds like a very nice raw processor. My name seems to have been invoked several times in this thread, so I thought I would chime in. These points might be relevant to the general questions that I think you might be asking:
I saw a reference to “dark” in the github rawproc write-up and I think also above. If the image looks darker than it should, this almost certainly indicates a color management issue that should be dealt with by assigning the correct ICC profile (if that’s the problem), or else by finding and fixing whatever “calibrate/profile/color-manage” issues there might be in the workflow and display chain.
If the monitor isn’t calibrated and profiled (both require a color measuring instrument), then the monitor isn’t showing correct colors (unless the monitor has a decent sRGB preset and hasn’t drifted over time since it left the factory, in which case you can get by with using sRGB as the monitor profile).
I’m assuming rawproc uses LCMS color management to send the image to the screen. That is, LCMS converts the image RGB values from the image ICC profile to the screen ICC profile, and then the converted RGB values are sent to the screen. So there is absolutely no reason to worry about matching ICC profile “gamma” values. LCMS takes care of this. As relevant notes:
-
If the monitor’s color gamut is significantly smaller than sRGB, the monitor can’t show correct colors even for sRGB images.
-
For images in other color spaces, the monitor can’t show correct colors for any image colors that fall outside the monitor’s color gamut (as defined by the monitor profile, which underscores the need to have a good monitor profile). “Perceptual intent” doesn’t mitigate this problem, even if your monitor profile actually does have a perceptual intent table. Matrix monitor profiles (most common case) don’t have perceptual intent tables, in which case asking for “perceptual intent” actually gets you relative colorimetric intent, which simply clips all out of gamut channel values.
For the chain from camera raw file to interpolated image file, I would recommend the following (but take my recommendations with a grain of salt, or perhaps the whole salt shaker 
):
-
Use the equivalent of “dcraw -4 -T -o 0” to output a linear gamma, “raw” color file.
-
Use LCMS to assign an appropriate camera input profile. Where to get camera input matrix profiles:
- Make your own camera input profile.
- Or use PhotoFlow (requires the linear gamma branch?) plus ArgyllCMS extracticc to extract the camera matrix profile for your specific camera and save it to disk.
- Or use the dcraw adobe_coeff table to construct the actual ICC profiles.
- Or use the augmented matrix profiles available from various free/libre raw processors.
-
Use LCMS to convert from the assigned camera input profile to the chosen RGB working space ICC profile (over the years I’ve encountered several issues with dcraw color management). My recommendation is to use Rec.2020 or ACEScg as the large-gamut RGB working space:
- Both have fewer imaginary colors than ProPhotoRGB.
- Both seem to produce “nicer than ProPhotoRGB” results for chromaticity-dependent editing operations (anything involving multiplication or division by a non-gray color; anything that involves using individual channel data, including non-luminance-based conversions to black and white).
-
At this point you have the digital equivalent of a “flat print”. Except that it’s actually more flat than a “wet” darkroom black and white “straight print” because film and paper involve shoulder and toe rolloff, and digital does not (unless your camera settings involved shadow lifting/highlight compression/similar algorithms).
-
For actually editing the “flat print” to suit one’s artistic intent for the image, use two versions of the chosen RGB working space profile, with the same chromaticities but different TRCs - one version with a linear TRC and one version with a perceptually uniform TRC. Depending on the editing algorithm and the artistic intent, use LCMS to convert the layer or entire layer stack between the two profiles as desired, or else program in the two TRCs with some way to flip back and forth between the two TRCs. In my opinion, these are the only two TRCs that are technically relevant to high bit depth photographic editing:
- linear gamma (gamma=1.0), for proper color mixing (most layer blend modes including Normal, blurring, down-scaling, Levels, Curves, etc).
- the LAB companding curve (perceptually uniform). This second TRC is for things like most “find edges” algorithms, soft light blend mode, adding uniform RGB noise, and such.
Regarding ProPhotoRGB:
- The gamma=1.8 TRC was chosen by Kodak because it’s closer to linear than gamma=2.2, thus mitigating somewhat the gamma artifacts from processing non-linearly encoded RGB images. Also ProPhotoRGB is a very large color space, which also helps mitigate gamma artifacts. It’s better to just use a linear gamma RGB working space and avoid the gamma artifacts altogether.
- The ProPhotoRGB primaries are better primaries for general editing than the sRGB primaries, for various reasons covered in the Kodak documentation (http://photo-lovers.org/pdf/color/romm.pdf). But they aren’t as good as some other editing spaces that have been promulgated since ProPhotoRGB was released. http://colour-science.org/posts/about-rendering-engines-colourspaces-agnosticism/ “ranks” various color spaces. But I’m also speaking from my own testing, and I encourage other people to make their own tests.
4 Likes
. Hopefully my ramblings got somewhere near the actual questions!
@Elle, thank you so very much for the detailed response. First, I’d like to thank you for posting your profile set, they’re the reference that has restored my confidence in my elementary color management logic in rawproc.
Essentially, for raw files delivered by libraw, I’m assigning a profile I create with LCMS using an exif dump of each dcraw -o X’s ICC. I use another LCMS tool I wrote that finds the primaries, black/white points and TRCs, and I use those to produce a (hopefully) equivalent profile. That is what rawproc’s workflow uses as the working profile for the rest of the edit.
rawproc is what i call a ‘chain’ editor, you add tools to the processing chain in whatever order you choose, and what you eventually save to a TIFF or JPEG is the result image at the end of the chain, Each tool has a checkbox that specifies that tool to be the display image, and it is there that the working space → display transform is conducted. At output, there is also a working space → output transform. Right now, I’ve deleted all my internal profile generation code except for the dcraw profile creation, and I’m using your profiles for display and output. BTW, that straightened out my ‘dark’ thing; I was using my generated prophoto profile for TIFF saves, and they apparently are wrongly specified, all’s well using LargeRGB-elle-V2-g18.icc.
So, I still need to get the dcraw assignment straight. Yes, the more I consider it, it is best to abandon the dcraw concoctions and work with the linear raw and assigned camera profile. I’m probably going to release rawproc 0.6 with the dcraw-assigned working spaces because I need to add code to manage a separate working space.
I’m going to take some time to digest all you’ve written, both in this thread and at your website. I’ve worked a long time with various geodetic reference frames; unless my dementia prevails, i should be able to eventually grok profiles… 
1 Like
@ggbutcher I don’t know how to generate ICCs from dcraw's output color spaces. If you could share them, that would be great.
-o [0-6] Output colorspace (raw,sRGB,Adobe,Wide,ProPhoto,XYZ,ACES)It’s simple: generate an image with each color space/gamma combination, then extract the profiles with exiftool.
That is probably the approach I’ll use in the interim.
Oops! I wrote the wrong thing: I already extracted the ICCs 
. However, that format isn’t human readable.
What I wanted to ask: how did you get to the point where you could compare the profiles with @Elle’s. I know I could probably examine dcraw.c and make-elles-profiles.c, but I am not a math or code wiz. It would be great if you, @Elle or someone more capable than I could guide me through the steps. Thanks.
@afre There are quite a few ways to extract information from an ICC profile in human-readable form (even including using a hex editor). Somewhere on this forum there is a thread that lists a lot of profile-related software. I think Rawpedia has a write-up.
The simplest way to turn a profile into something readable with a text editor is to use iccToXml:
SampleIcc Version 1.6.8, ICCXML Version 0.9.8
Usage: IccToXml src_icc_profile dest_xml_file
Thanks for the refresher. Lately I have been burnt out physically and mentally – experiencing lots of musculoskeletal pain, so I am not the sharpest knife in the cupboard. I will try IccToXml later and report back.
So you don’t have one universal working color space, but you edit in camera RGB? That makes it hard to do the same edits to images taken with different cameras, and might even give other problems, depending on the specific color space. I’d reconsider that and move from camera RGB to somethign like Rec2020 as Elle suggested.
I think @ggbutcher meant that he currently uses dcraw to convert from the dcraw matrix camera input profile to a dcraw output space (the “-o #” options), and then makes an ICC profile from the dcraw output space, for actually editing the interpolated image.
As far as editing in “camera RGB” I find this to be very useful for certain purposes, including setting different white balances for different portions of mixed lighting images, using layers and masks to choose which white balance is applied to which portion of the image.
I think this “multi-layer/multi-white-point/mixed-lighting” approach probably works for any camera input profile. But it’s easy to make a custom camera input profile (http://ninedegreesbelow.com/photography/well-behaved-camera-profile.html) that’s also a well-behaved (eg neutral gray axis, with white=Lab 100,0,0 and black=Lab 0,0,0), that can be used for other types of edits.
However, as @houz notes, using “camera RGB” can cause issues:
-
For editing operations that are chromaticity-independent, as long as camera RGB is also a well-behaved RGB working space, then these operations produce the same results regardless, assuming no precision losses from editing in such large and oddly shaped color spaces.
-
For other operations, well, it depends. If the goal is to retrieve individual channel information as captured by the camera, use “camera RGB”. If the goal is to split-tone using Curves, this operation is likely to produce unpleasant results in “camera RGB”, as will making single-channel gamma modifications.
What I wanted i my initial try at CMS was to use the dcraw output colorspace (-o n) as my working space. Thing is, libraw doesn’t make the dcraw ICC until time to save as TIFF, or so it appears, as I can’t find it in any of the data structures available after dcraw_process. So, my first attempt was to use LCMS to build internal profiles for each of the -o options, and assign them to the libraw image for subsequent processing. Actually, that seemed to work quite well, although I can’t really know for sure until I acquire the resources to calibrate my displays. The only thing vexing was when I went to save a TIFF with the working space profile, when I opened it the histogram was different than when output, but the image seemed identical. That’s when I started this thread…
I got back to goodness by abandoning my internal profiles and just using @Elle’s profiles. But, simply assigning her LargeRGB g1.8 to the dcraw -o 4 output just didn’t seem proper, so I’m going to make the following modifications before releasing 0.6:
-
input.raw.cms.rawprofile - name of a .icc to assign to the input image in place of any previously delivered profile This would be intended to be a camera profile, but could be a .icc equivalent for any dcraw-produced profile.
-
Implement a Convert Colorspace tool. As I described above, rawproc is a ‘chain’ editor, the GUI approximation of gmic or imagemagick. I’ll just implement a tool that can be added at any point to convert from the assigned colorspace to one in a supplied file. That’ll be the first tool applied in a workflow that starts with a camera profile-assigned image, to convert to a working colorspace for the subsequent tools.
I debated doing the working space conversion as part of opening the file, but doing it as a tool provides flexibility to apply color space transformations anywhere in the tool chain. Not sure for what, just seems useful, to bear-of-little-brain here… 
I want to thank all who’ve responded here; your words have been extremely helpful in schooling me through another part of color management. I’m starting to get the mechanics; will need another couple hundred cups of coffee for the theory.//
What was it that felt wrong? The fact that you are not 100% sure that the profile dcraw uses is identical to the one you got from Elle? In that case you could use -o 5 to get XYZ from dcraw which is universally defined and then use lcms2 to convert that to the desired profile.
@houz, thanks, that’s a great idea I hadn’t considered. And yes,
comparing the primaries, they’re similar but not identical, and I’m not
yet familiar enough with this stuff to know how significant is the
difference.
After I conclude the on-going family project, I’m going to profile my
camera and calibrate my displays. What I want to have in rawproc is
enough CMS to accommodate those profiles, and later implement
soft-proofing and other things I come across.
I want to thank all who have helped me in this thread, particularly
those who author/work on other raw processors. What this had been for
me is 1) a significant learning experience, and 2) implementing
something that supports my own peculiar notions about workflow on
small-ish computers. I’m very much not out to compete with the more
capable offerings, rather, I want a learning-oriented platform where
basic digital image manipulation can be experimented with, for all its
power and pitfalls. G’MIC was especially influential; the whole idea of
stringing together image operations in any order prompted the tool that
eventually became rawproc.
In the learning vein, I’m going to clean up the gimage code so that the
algorithms are easier to follow. If you go to the internets for
literature and reference for a lot of these, it can be quite daunting to
assemble a working implementation of a particular operation. I hope
that code can help others grasp the fundamentals enough to write their
own rawprocs…
1 Like
@ggbutcher @Elle @Morgan_Hardwood
Following up with my post, I generated XML files from the dcraw ICCs in which I am interested; i.e. XYZ and ACES. I have included the raw and commands in case they are affected by them. I am still not quite sure on how to compare them with @Elle’s ICCs. Please excuse my Windows (gasp) syntax.
dcraw -v -w -H 2 -4 -T -o 5 -O XYZ.tif _MG_0419.CR2 && ^
exiftool -icc_profile -b -w icc XYZ.tif
dcraw -v -w -H 2 -4 -T -o 6 -O ACES.tif _MG_0419.CR2 && ^
exiftool -icc_profile -b -w icc ACES.tif
for %i in (*.icc) do IccToXml %i %~ni.xml.txt
ACES.xml.txt (2.0 KB)
XYZ.xml.txt (2.0 KB)
ACEScg-elle-V4-g10.xml.txt (2.9 KB)
ACES-elle-V4-g10.xml.txt (2.9 KB)
ACEScg-elle-V2-g10.xml.txt (2.6 KB)
ACES-elle-V2-g10.xml.txt (2.6 KB)
XYZ-D50-Identity-elle-V4.xml.txt (2.1 KB)
Context
I guess my goal is to figure out how to get the most out of dcrawa, without going the camera profiling route, generating solid RGB input data for G’MIC processinga, which apparently doesn’t concern itself with input and output color space. aProbably applicable in other contexts as well.
Hi @afre , and my apologies for taking so long to reply.
The relevant ACES profiles from my profile pack are ACES-elle-V4-g10.icc, and the V2 version of the same profile. The ACEScg profile has a smaller color gamut, close to the Rec.2020 color gamut.
If you compare the matrices for the XYZ profiles:
- My XYZ profile has the identity matrix for the Red/Green/Blue colorants.
- The dcraw XYZ profile also has the identity matrix for the colorants, but it’s been Bradford-adapted from D50 to D65.
I don’t know why the dcraw XYZ profile is the way it is. I had Marti Maria look at my own XYZ profile and he said it seemed fine (https://sourceforge.net/p/lcms/mailman/message/35693996/). I also had some people with access to image editing software that can process XYZ profiles check my XYZ profile, and again, nobody found any problems. I did these checks several months ago, precisely because a person using dcraw sent me an email saying that my XYZ profile isn’t the same as the dcraw XYZ profile. And obviously it’s not!
The dcraw profile-making code uses the D65 white point in a somewhat odd way, producing somewhat non-standard profiles with varying amounts of deviation from normal D50-adapted ICC profiles and color space conversions.
Given what you want to do, I would recommend the following, which allows you to bypass using dcraw’s ICC profiles:
-
Use PhotoFlow to export to disk a sample interpolated raw file from your camera, with the embedded color space profile still being the default dcraw input profile for your camera. @Carmelo_DrRaw can explain how to save a sample image to disk with the default camera input profile still embedded.
-
Using the image produced by step 1, use whatever program you like to extract the embedded default camera input profile and save it to disk. Perhaps the easiest to use is GIMP, which has an option to save the image’s ICC profile to disk.
-
Use these command line options to process images using dcraw:
dcraw -o 0 -4 -T <whatever white balancing and other options you want> file-name.raw-extension
-
Assign the extracted camera input profile from step 2 to the dcraw output.
-
Convert the dcraw output to your chosen RGB working space.
The first two steps only need to be done once. For steps 4 and 5, various programs can assign/convert from the command line, and of course just about all GUI editing software can assign an ICC profile and then convert to another ICC profile.
Regarding which RGB working space to choose, G’MIC doesn’t concern itself with ICC profiles, but many G’MIC operations do use hard-coded sRGB matrix values and/or TRC. So depending on the image colors and the precise G’MIC command you want to use, you might want to convert from the camera input profile to the regular sRGB color space.
Just be aware that image colors outside the sRGB color gamut will be clipped upon conversion to sRGB, if your destination sRGB profile is a matrix sRGB profile. This is true even if you use perceptual intent, because matrix ICC profiles don’t have perceptual intent tables, so if you ask for perceptual intent, what you get is relative colorimetric, which clips.
My sRGB profiles are matrix profiles, as are sRGB profiles from GIMP, RT, dt, PF, ArgyllCMS, LCMS, dcraw, and so on. The only LUT sRGB profiles I know of are from color.org, and are designed for use in a very specific print-oriented workflow.
1 Like
In case anyone might wonder, if you don’t convert from the camera input profile to an actual RGB working space such as sRGB or Rec.2020 before starting to edit the image, and instead you try to process the image while still in the camera input profile color space, various operations are likely to produce variously odd results, depending on the particular operation and the particular camera input profile.
But some operations (for example white balancing) do work quite well while the image is still in the camera input profile, though this depends on the type of input profile. My camera input profiles are well-behaved custom linear gamma simple matrix input profiles made using ArgyllCMS. I haven’t ever used other types of camera input profiles for white balancing and such, but I imagine results would be not so good.
@Elle Thanks for piecing it together for me. I read things over and over again but sometimes it doesn’t click (because of the stressful life I have ; this forum has been a good distraction). That last point about sRGB is disappointing but once I figure this out I guess it is only the last step that clips.
I sometimes use PhotoFlow but to me — maybe out of habit — it is convenient to use dcraw and G’MIC together. Maybe if @Carmelo_DrRaw can guide me through the raw development steps (or if there is a tutorial handy), I would feel more confident using PF more often; but currently I feel that I don’t know what I am doing.