RawTherapee restriction on user-chosen output profile

@Hombre you might be interested to have a look at how we are dealing with soft-proofing and gamut warnings in PhotoFlow, as we are using our own method which seems to be more accurate than LCMS2. The relevant code is here. Shortly speaking, we have two different methods depending if the proofed RGB profile is of matrix or LUT type. Support for CMYK profiles is work in progress.

You can PM me if you’d like to discuss some of the details…

Hi Glenn,

My apologies, I haven’t tried your rawproc, though doing so has been on my “to do” list for a long time. Not having ever used rawproc, it’s not clear to me what “saving with working profile” actually means. But generally speaking (and I’m sure you know this as your experiment was just to see what rawproc might do) it’s not a good idea to save a jpeg to disk while the jpeg is still in the camera input profile color space.

Generally speaking, camera input profile color spaces are not good color spaces for image editing.

That’s not worded strongly enough. Generally speaking, camera input ICC profile color spaces are horrible for image editing. This is true even when the camera input ICC profile is a simple matrix profile and even more true if the camera input ICC profile is a LUT ICC profile:

  • Most camera input profiles aren’t well-behaved RGB working spaces (https://ninedegreesbelow.com/photography/well-behaved-profile.html), meaning they don’t have a neutral gray axis, meaning if R=G=B, the resulting color isn’t neutral gray, with a=b in Lab space. And sometimes they made such that R=G=B=1.0 floating point doesn’t equal Lab (100,0,0), but instead has an L value that’s >100 or < 100). So “gray” isn’t really gray and “white” really isn’t white.

  • Even if you make and use a well-behaved custom camera simple linear gamma input ICC profile (Make a better custom camera profile), these profiles define huge color spaces where much of the total volume is taken up by imaginary colors, which all by itself makes these color spaces not suitable for general editing.

  • In addition even a well-behaved camera input profile color space has primaries that are placed such that adjustments that affect just individual channels are likely to produce unpleasant results. For example, raising just the blue channel using Curves, when done in Rec.2020 makes a blue sky look more blue. But doing this in the well-behaved linear gamma input profile I made for my Canon 400D camera makes the sky look unpleasantly purple.

I use my custom camera input profiles (I made a set for my Canon 400D and another set for my Sony A7) for the following very specific editing tasks, and only when the image has been “scene-referred” interpolated (no “make pretty” post-interpolation editing algorithms) before being saved to disk from the raw processor:

  • White balancing difficult images, including images with multiple light sources, using masks and layers and Levels using GIMP-CCE or other image editor that allows to edit in user-chosen RGB working spaces using masks and layers.

  • Retrieving the camera-captured RGB channel information for use as a layer mask and for black and white processing (Blue Channel "noise" in Raw Files) - once converted to a regular RGB working space, even using unbounded ICC profile conversions, “what the camera actually captured per channel” is gone, though for most images I’ve seen this isn’t a huge factor. But sometimes it is.

  • Blending multiple (usually but not always ev-bracketed) exposures by hand.

  • Dealing with imaginary colors that are sometimes produced by applying simple linear gamma matrix input ICC profiles to interpolated image files, in particular imaginary colors on the violet-blue/yellow axis. For example, I deal with imaginary saturated violet-blue colors using masks and layers combined with a custom ArgyllCMS Lab LUT ICC profile camera input profile that (unlike simple linear gamma matrix ICC profiles) doesn’t generate imaginary colors along the violet-blue/yellow axis, but otherwise does produce somewhat anemic colors, especially in blue skies.

Offhand I can’t think of any additional editing tasks that I’d want to do before converting the image layer stack to a “normal” RGB working space such as Rec.2020.

1 Like

@Elle,

  1. I should probably have demonstrated the camera profile save with a
    TIFF. That’d been more appropriate to what I think your use case is
    after.

  2. rawproc in its current version will use whatever profile is assigned to
    the opened image for further processing, and will save the output image
    with that assigned profile if nothing else is configured in the
    Properties. When a raw is opened, if nothing else is specified, the opened
    image will have the dcraw sRGB profile, with sRGB gamma applied. You can
    modify that with various input.raw.libraw parameters mostly corresponding
    to the libraw parameters for controlling dcraw processing. I added one of
    my own, input.raw.libraw.cameraprofile, so that if you tell libraw to open
    the image -o 0 -g 1.0 (“raw” colorspace, linear (no) gamma applied), it’ll
    assign the named icc file to the image. To apply a working colorspace,
    there is a tool called “colorspace” that you’d put in as the first
    manipulation, specify the icc filename and ‘apply’, and it will transform
    the image to conform to that profile and attach that profile to the image
    for the remainder of the processing chain until either 1) you put in
    another colorspace tool, or 2) save the image. At the time of save, an
    output..cms.profile parameter (’’ is the output type, ‘jpeg’ or ‘tiff’ at
    present) is consulted, and if there’s an icc file specified it is applied
    to the image before saving, and that profile is attached to the outputted
    file. If output.*.cms.profile is blank, the image is saved with whatever
    profile followed it through the processing chain, which I referred to as
    the ‘working’ profile in the dialog. Yes, this can be done for either a
    JPEG or a TIFF file save. I can see the implications of saving a JPEG image
    with a profile that outstrips its gamut, but I’d rather allow that than to
    start putting in the sort of protections your original post to the thread
    considers.

Soooo, to get where I think you want to go in your original question using
rawproc, you’d open the camera raw with
input.raw.libraw.cameraprofile=“yourcamera.icc”, and then immediately save
to TIFF with output.tiff.cms.profile="". The resulting TIFF image would
still be in camera colorspace, and the attached profile would also be in
camera colorspace. No tools applied between open and save.

My first attempt was to put the cameraprofile.icc in output.*.cms.profile,
but that crashed the program. Debugging showed that cmsTransform had no
valid transform, LittleCMS didn’t like using the camera profile as input.
Makes sense, the rawproc save assumes a colorspace apply, not assign.
Still, I need to trap that, commit a fix to check the transform before
using it.

What I’m trying to build with rawproc is a toolbox, and the means to apply
the tools in a very flexible and straightforward manner. So, if an
operator chooses to open a raw with a camera profile and not apply a
colorspace tool with a proper working profile as the first order of
business, they reaps what they sows. Hammering screws, so to speak…

btw, I’ve spent a lot of quality time with your articles, and they’ve been
the primary source for helping me incorporate a “toolbox notion” of color
management into rawproc. Particularly, I’ve tried hard to conform to the
“apply” and “assign” semantics for colorspace usage. Now at home, I have a
single sunlight-calilbrated D7000 camera profile and two calibrated display
profiles, one each for my desktop and tablet, and things seem to be working
as intended except for opening a raw with a camera profile and applying a
working profile at the start. That seems to oversaturate the colors, far
more that what I’d expect, but I haven’t had time to completely test the
combinations of the input->working->display/output chains to see if/where
I’m doing something wrong.

Thanks for the involved reply. I’m still learning this color management
business, and your prose is extremely instructive in that regard. That
said, I hope you appreciate the angst fomented upon us hapless programmers
trying to make tools that work responsively, without crashing… :smiley:

Hi Glenn,

My apologies, I’m not sure what you mean by “apply” and “assign” semantics, but it seems you are talking about something I wrote in an article on my website. Could you give a link?

Hmm, when you say that “opening a raw with a camera profile and applying a working profile at the start . . . seems to oversaturate the colors, far more than I’d expect”, this indicates to me that somewhere there is a problem in the color management chain, but as I said, I’m not sure what you mean by “applying a working profile”.

If “applying a working profile” means “assign the camera input profile to the interpolated raw file, and then convert to the RGB working space profile”, then the image should look pretty much the same before and after the conversion, all things being equal. “All things being equal” covers things like the respective color gamuts of your monitor profile and your working space, and the actual colors in the raw file as interpreted by assigning the camera input profile to the interpolated raw file, and of course presumes that color management is being used to send the image to the screen.

This morning I tried to compile rawproc from git to get a handle on how color management is being handled, but gave up when I realized that first I’d have to download and compile wxWidgets from git. That’s a huge download. I thought wxWidgets was available from Gentoo portage, but if it is, maybe it’s under some other name?

As far as what I personally want from raw processors, that’s fairly well covered by PhotoFlow, darktable, and RawTherapee. Though I still would like a raw processor that is as fast as ufraw/nufraw, and works more or less like ufraw/nufraw, but with true exposure compensation (UFRaw blown highlights), with the AMAZE algorithm, and with floating point tiff and/or exr output. This is one of the reasons why “compile and try rawproc” is on my “to do” list - I was wondering if rawproc might be a possible replacement for ufraw/nufraw.

Oh, yes I do appreciate the angst :slight_smile: . Several years ago, when I spent some time modifying dcraw to make its internal processing be floating point and to add some additional interpolation algorithms, my appreciation for developers increased exponentially. Learning enough c code to modify dcraw and get it to run was not easy!

While working on the dcraw code I learned about projects that developed various interpolation algorithms not in dcraw, such as the people working on Guillermo Luijk’s PerfectRaw (GUILLERMO LUIJK >> PROGRAMAS >> PERFECT RAW - not to be confused with the totally unrelated PhotoShop colorneg/perfectcolor/perfectraw plug-ins) and the followup coding efforts by Rawness, RawTherapee, LibRaw and other groups involved with writing raw processing code. Wow! so much work and creativity has gone into and continues to go into our raw processing programs!

On the same line of thought, free/libre software depends on people volunteering their time and effort for coding, and also for providing feedback, making good bug reports and following through with testing, reading and responding to bug reports (that’s a huge task all by itself), helping to write user manuals, helping each other on mailing lists and in forums like pixls.us, writing articles and making videos on free/libre software, setting up and maintaining forums and mailing lists, and on and on.

It’s the sum total of everyone’s efforts that makes free/libre software possible. But without the time and effort of all the developers who write the actual code for free/libre software, there’d be nothing at all for the rest of us to contribute to.

Ack, that’s what I get for relying on my memory, should be “convert” vs “assign”, per your article here: https://ninedegreesbelow.com/photography/convert-assign.html

I very much came to appreciate the journey you took with dcraw-undnged, dcraw-unDnged, which is why I added the input.raw.libraw.cameraprofile parameter. Long story short, if you set the following parameters in rawproc:

input.raw.libraw.colorspace=raw
input.raw.libraw.gamma=linear
input.raw.libraw.cameraprofile=yourcalibratedcameraprofile.icc

libraw delivers the raw linear camera image just like dcraw would with -o 0 -g 1.0, and my software assigns the specified .icc profile to it. That’s the starting point in rawproc; to proceed from there ‘color-managed’, you’d first add a colorspace tool and use it to apply (convert to) a working profile speciied as a .icc file. Or, if your camera profile is well-behaved, you can just start adding tools and they’ll work on the raw-linear image. The image displayed during work at each step is converted using a display profile, another .icc file. When you go to save, the image at the end of the chain of applied tools is either converted to a specified ouput profile, or assigned the profile that followed it through processing.

This is a thread hijack, but here I go: when open a raw as described above, the colors look right to me, but when I add the colorspace tool and apply a working profile, I get rather garish colors. Thing is, if I specify the camera profile in the colorspace tool, the resulting display image looks the same as before the colorspace tool. That’s where I need to spend some time picking apart my cmsTransform chains…

Back to the thread, the reason I’m describing all this is to put forward one approach to getting the sort of flexible, ‘do it my way’ raw processing toolbox you’re looking for. The DT and RT programmers have a quandry in this regard, judging just how much flexibility their user base can stand. Me, with a user base of one, not so much. I would be curious to hear how the AppImage works with Gentoo, if you are so inclined, one way to avoid dealing with wxWidgets…

To avoid further hijacking :slight_smile: I replied to your comments over in this thread: