Thanks for chiming in. It was my intention to invoke the @Elle command 
.
Raw Histogram, or Display Histogram
Hopefully my ramblings got somewhere near the actual questions!
@Elle, thank you so very much for the detailed response. First, I’d like to thank you for posting your profile set, they’re the reference that has restored my confidence in my elementary color management logic in rawproc.
Essentially, for raw files delivered by libraw, I’m assigning a profile I create with LCMS using an exif dump of each dcraw -o X’s ICC. I use another LCMS tool I wrote that finds the primaries, black/white points and TRCs, and I use those to produce a (hopefully) equivalent profile. That is what rawproc’s workflow uses as the working profile for the rest of the edit.
rawproc is what i call a ‘chain’ editor, you add tools to the processing chain in whatever order you choose, and what you eventually save to a TIFF or JPEG is the result image at the end of the chain, Each tool has a checkbox that specifies that tool to be the display image, and it is there that the working space -> display transform is conducted. At output, there is also a working space -> output transform. Right now, I’ve deleted all my internal profile generation code except for the dcraw profile creation, and I’m using your profiles for display and output. BTW, that straightened out my ‘dark’ thing; I was using my generated prophoto profile for TIFF saves, and they apparently are wrongly specified, all’s well using LargeRGB-elle-V2-g18.icc.
So, I still need to get the dcraw assignment straight. Yes, the more I consider it, it is best to abandon the dcraw concoctions and work with the linear raw and assigned camera profile. I’m probably going to release rawproc 0.6 with the dcraw-assigned working spaces because I need to add code to manage a separate working space.
I’m going to take some time to digest all you’ve written, both in this thread and at your website. I’ve worked a long time with various geodetic reference frames; unless my dementia prevails, i should be able to eventually grok profiles… 
1 Like
@ggbutcher I don’t know how to generate ICCs from dcraw's output color spaces. If you could share them, that would be great.
-o [0-6] Output colorspace (raw,sRGB,Adobe,Wide,ProPhoto,XYZ,ACES)It’s simple: generate an image with each color space/gamma combination, then extract the profiles with exiftool.
That is probably the approach I’ll use in the interim.
Oops! I wrote the wrong thing: I already extracted the ICCs 
. However, that format isn’t human readable.
What I wanted to ask: how did you get to the point where you could compare the profiles with @Elle’s. I know I could probably examine dcraw.c and make-elles-profiles.c, but I am not a math or code wiz. It would be great if you, @Elle or someone more capable than I could guide me through the steps. Thanks.
@afre There are quite a few ways to extract information from an ICC profile in human-readable form (even including using a hex editor). Somewhere on this forum there is a thread that lists a lot of profile-related software. I think Rawpedia has a write-up.
The simplest way to turn a profile into something readable with a text editor is to use iccToXml:
SampleIcc Version 1.6.8, ICCXML Version 0.9.8
Usage: IccToXml src_icc_profile dest_xml_file
Thanks for the refresher. Lately I have been burnt out physically and mentally – experiencing lots of musculoskeletal pain, so I am not the sharpest knife in the cupboard. I will try IccToXml later and report back.
So you don’t have one universal working color space, but you edit in camera RGB? That makes it hard to do the same edits to images taken with different cameras, and might even give other problems, depending on the specific color space. I’d reconsider that and move from camera RGB to somethign like Rec2020 as Elle suggested.
I think @ggbutcher meant that he currently uses dcraw to convert from the dcraw matrix camera input profile to a dcraw output space (the “-o #” options), and then makes an ICC profile from the dcraw output space, for actually editing the interpolated image.
As far as editing in “camera RGB” I find this to be very useful for certain purposes, including setting different white balances for different portions of mixed lighting images, using layers and masks to choose which white balance is applied to which portion of the image.
I think this “multi-layer/multi-white-point/mixed-lighting” approach probably works for any camera input profile. But it’s easy to make a custom camera input profile (http://ninedegreesbelow.com/photography/well-behaved-camera-profile.html) that’s also a well-behaved (eg neutral gray axis, with white=Lab 100,0,0 and black=Lab 0,0,0), that can be used for other types of edits.
However, as @houz notes, using “camera RGB” can cause issues:
-
For editing operations that are chromaticity-independent, as long as camera RGB is also a well-behaved RGB working space, then these operations produce the same results regardless, assuming no precision losses from editing in such large and oddly shaped color spaces.
-
For other operations, well, it depends. If the goal is to retrieve individual channel information as captured by the camera, use “camera RGB”. If the goal is to split-tone using Curves, this operation is likely to produce unpleasant results in “camera RGB”, as will making single-channel gamma modifications.
What I wanted i my initial try at CMS was to use the dcraw output colorspace (-o n) as my working space. Thing is, libraw doesn’t make the dcraw ICC until time to save as TIFF, or so it appears, as I can’t find it in any of the data structures available after dcraw_process. So, my first attempt was to use LCMS to build internal profiles for each of the -o options, and assign them to the libraw image for subsequent processing. Actually, that seemed to work quite well, although I can’t really know for sure until I acquire the resources to calibrate my displays. The only thing vexing was when I went to save a TIFF with the working space profile, when I opened it the histogram was different than when output, but the image seemed identical. That’s when I started this thread…
I got back to goodness by abandoning my internal profiles and just using @Elle’s profiles. But, simply assigning her LargeRGB g1.8 to the dcraw -o 4 output just didn’t seem proper, so I’m going to make the following modifications before releasing 0.6:
-
input.raw.cms.rawprofile - name of a .icc to assign to the input image in place of any previously delivered profile This would be intended to be a camera profile, but could be a .icc equivalent for any dcraw-produced profile.
-
Implement a Convert Colorspace tool. As I described above, rawproc is a ‘chain’ editor, the GUI approximation of gmic or imagemagick. I’ll just implement a tool that can be added at any point to convert from the assigned colorspace to one in a supplied file. That’ll be the first tool applied in a workflow that starts with a camera profile-assigned image, to convert to a working colorspace for the subsequent tools.
I debated doing the working space conversion as part of opening the file, but doing it as a tool provides flexibility to apply color space transformations anywhere in the tool chain. Not sure for what, just seems useful, to bear-of-little-brain here…
I want to thank all who’ve responded here; your words have been extremely helpful in schooling me through another part of color management. I’m starting to get the mechanics; will need another couple hundred cups of coffee for the theory.//
What was it that felt wrong? The fact that you are not 100% sure that the profile dcraw uses is identical to the one you got from Elle? In that case you could use -o 5 to get XYZ from dcraw which is universally defined and then use lcms2 to convert that to the desired profile.
@houz, thanks, that’s a great idea I hadn’t considered. And yes,
comparing the primaries, they’re similar but not identical, and I’m not
yet familiar enough with this stuff to know how significant is the
difference.
After I conclude the on-going family project, I’m going to profile my
camera and calibrate my displays. What I want to have in rawproc is
enough CMS to accommodate those profiles, and later implement
soft-proofing and other things I come across.
I want to thank all who have helped me in this thread, particularly
those who author/work on other raw processors. What this had been for
me is 1) a significant learning experience, and 2) implementing
something that supports my own peculiar notions about workflow on
small-ish computers. I’m very much not out to compete with the more
capable offerings, rather, I want a learning-oriented platform where
basic digital image manipulation can be experimented with, for all its
power and pitfalls. G’MIC was especially influential; the whole idea of
stringing together image operations in any order prompted the tool that
eventually became rawproc.
In the learning vein, I’m going to clean up the gimage code so that the
algorithms are easier to follow. If you go to the internets for
literature and reference for a lot of these, it can be quite daunting to
assemble a working implementation of a particular operation. I hope
that code can help others grasp the fundamentals enough to write their
own rawprocs…
1 Like
@ggbutcher @Elle @Morgan_Hardwood
Following up with my post, I generated XML files from the dcraw ICCs in which I am interested; i.e. XYZ and ACES. I have included the raw and commands in case they are affected by them. I am still not quite sure on how to compare them with @Elle’s ICCs. Please excuse my Windows (gasp) syntax.
dcraw -v -w -H 2 -4 -T -o 5 -O XYZ.tif _MG_0419.CR2 && ^
exiftool -icc_profile -b -w icc XYZ.tif
dcraw -v -w -H 2 -4 -T -o 6 -O ACES.tif _MG_0419.CR2 && ^
exiftool -icc_profile -b -w icc ACES.tif
for %i in (*.icc) do IccToXml %i %~ni.xml.txt
ACES.xml.txt (2.0 KB)
XYZ.xml.txt (2.0 KB)
ACEScg-elle-V4-g10.xml.txt (2.9 KB)
ACES-elle-V4-g10.xml.txt (2.9 KB)
ACEScg-elle-V2-g10.xml.txt (2.6 KB)
ACES-elle-V2-g10.xml.txt (2.6 KB)
XYZ-D50-Identity-elle-V4.xml.txt (2.1 KB)
Context
I guess my goal is to figure out how to get the most out of dcrawa, without going the camera profiling route, generating solid RGB input data for G’MIC processinga, which apparently doesn’t concern itself with input and output color space. aProbably applicable in other contexts as well.
Hi @afre , and my apologies for taking so long to reply.
The relevant ACES profiles from my profile pack are ACES-elle-V4-g10.icc, and the V2 version of the same profile. The ACEScg profile has a smaller color gamut, close to the Rec.2020 color gamut.
If you compare the matrices for the XYZ profiles:
- My XYZ profile has the identity matrix for the Red/Green/Blue colorants.
- The dcraw XYZ profile also has the identity matrix for the colorants, but it’s been Bradford-adapted from D50 to D65.
I don’t know why the dcraw XYZ profile is the way it is. I had Marti Maria look at my own XYZ profile and he said it seemed fine (https://sourceforge.net/p/lcms/mailman/message/35693996/). I also had some people with access to image editing software that can process XYZ profiles check my XYZ profile, and again, nobody found any problems. I did these checks several months ago, precisely because a person using dcraw sent me an email saying that my XYZ profile isn’t the same as the dcraw XYZ profile. And obviously it’s not!
The dcraw profile-making code uses the D65 white point in a somewhat odd way, producing somewhat non-standard profiles with varying amounts of deviation from normal D50-adapted ICC profiles and color space conversions.
Given what you want to do, I would recommend the following, which allows you to bypass using dcraw’s ICC profiles:
-
Use PhotoFlow to export to disk a sample interpolated raw file from your camera, with the embedded color space profile still being the default dcraw input profile for your camera. @Carmelo_DrRaw can explain how to save a sample image to disk with the default camera input profile still embedded.
-
Using the image produced by step 1, use whatever program you like to extract the embedded default camera input profile and save it to disk. Perhaps the easiest to use is GIMP, which has an option to save the image’s ICC profile to disk.
-
Use these command line options to process images using dcraw:
dcraw -o 0 -4 -T <whatever white balancing and other options you want> file-name.raw-extension
-
Assign the extracted camera input profile from step 2 to the dcraw output.
-
Convert the dcraw output to your chosen RGB working space.
The first two steps only need to be done once. For steps 4 and 5, various programs can assign/convert from the command line, and of course just about all GUI editing software can assign an ICC profile and then convert to another ICC profile.
Regarding which RGB working space to choose, G’MIC doesn’t concern itself with ICC profiles, but many G’MIC operations do use hard-coded sRGB matrix values and/or TRC. So depending on the image colors and the precise G’MIC command you want to use, you might want to convert from the camera input profile to the regular sRGB color space.
Just be aware that image colors outside the sRGB color gamut will be clipped upon conversion to sRGB, if your destination sRGB profile is a matrix sRGB profile. This is true even if you use perceptual intent, because matrix ICC profiles don’t have perceptual intent tables, so if you ask for perceptual intent, what you get is relative colorimetric, which clips.
My sRGB profiles are matrix profiles, as are sRGB profiles from GIMP, RT, dt, PF, ArgyllCMS, LCMS, dcraw, and so on. The only LUT sRGB profiles I know of are from color.org, and are designed for use in a very specific print-oriented workflow.
1 Like
In case anyone might wonder, if you don’t convert from the camera input profile to an actual RGB working space such as sRGB or Rec.2020 before starting to edit the image, and instead you try to process the image while still in the camera input profile color space, various operations are likely to produce variously odd results, depending on the particular operation and the particular camera input profile.
But some operations (for example white balancing) do work quite well while the image is still in the camera input profile, though this depends on the type of input profile. My camera input profiles are well-behaved custom linear gamma simple matrix input profiles made using ArgyllCMS. I haven’t ever used other types of camera input profiles for white balancing and such, but I imagine results would be not so good.
@Elle Thanks for piecing it together for me. I read things over and over again but sometimes it doesn’t click (because of the stressful life I have ; this forum has been a good distraction). That last point about sRGB is disappointing but once I figure this out I guess it is only the last step that clips.
I sometimes use PhotoFlow but to me — maybe out of habit — it is convenient to use dcraw and G’MIC together. Maybe if @Carmelo_DrRaw can guide me through the raw development steps (or if there is a tutorial handy), I would feel more confident using PF more often; but currently I feel that I don’t know what I am doing.
Well, clipping colors to the sRGB color gamut isn’t the end of the world and in some ways makes things easier:
-
Many people have produced many wonderful images while doing all their editing in the sRGB color space.
-
Many images don’t have any colors that exceed the sRGB color gamut, so these images don’t present any problems.
-
Many of us (including me) still don’t have monitors that can display colors much beyond the sRGB color gamut. So the good news is if any “very colorful” colors do get clipped, well, probably that means at least you can actually see the colors that are still there!
-
If the colors have already been clipped, then you don’t have to worry about how to modify the images to make the more colorful colors fit inside the sRGB color gamut. Bringing “very colorful” colors into the sRGB color gamut falls into the realm of soft proofing (assuming you are willing to extend the concept of soft proofing to include preparing an image for display on the web instead of confining “soft proofing” to mean preparing an image for printing). Soft proofing is often-times a difficult editing task.
Maybe in a few years we’ll all be using Rec.2020-capable monitors, and at that point dealing with “very colorful” colors will be a lot easier.
In the meantime, the nice thing about using a GUI raw processor like darktable/PhotoFlow/RawTherapee is that these programs have “out of gamut” indicators that allow you to modify the image to do something about out of gamut colors, before the image is exported to disk as an sRGB image.
With pleasure! However it will take few days before I will have time to write long messages. Meanwhile you can maybe check some of the tutorials I have collected in the photoflow G+ commuunity, and/or the videos in the YouTube channel. I will send you links as soon as I manage…
1 Like