RawTherapee restriction on user-chosen output profile

Software RawTherapee
7

Hmm, the above comment somewhat implies that the reason why the RawTherapee devs have made it impossible for users to select an input class profile as the user-chosen output profile is to protect the user from making a choice that will lead to wrong results for users who don’t know what they are doing.

In my opinion, writing code specifically to protect users from making mistakes based on a developers’ opinions about what users do and don’t understand is a road down which developers of image editing programs should not go.

But I suspect that the real reason for leaving input class profiles out of the list of allowed profile classes for saving an image to disk is that the RawTherapee devs didn’t realize there might be workflow reasons for outputting an interpolated image file that’s still in the camera input color space.

I like RawTherapee quite a lot - it has some amazing editing algorithms not available in other image editors. But normally I don’t use RawTherapee for actually interpolating raw files, because of one specific issue with the RT processing pipeline - there is a bounded ICC profile conversion - possibly to LAB, possibly to the RGB working space, maybe through both of these color spaces, that clips out-of-gamut colors in the interpolated raw file (Raw Processor Review, Part 1).

When I do use RawTherapee for interpolating a raw file, my workaround for avoiding clipped colors is to use an sRGB profile from disk as the input and output profile - thus avoiding any clipped colors from the RT processing pipeline - and then assign the camera input profile after opening the interpolated file with GIMP-2.9.

In other words, personally it doesn’t affect my raw processing workflow at all, regardless of whether RT does or doesn’t allow saving an image to disk in a user-chosen input class ICC profile.

However, if the real reason for making it impossible for users to choose an input class profile for the output profile is to protect users from doing something the RT devs think is not in the best interest of users who don’t know what they are doing, then perhaps a similar constraint should be put on the user’s ability to select a custom camera input profile from disk. After all, an sRGB profile from disk is not an “input class” profile. Rather it’s a “display class” profile.

1 Like
8

IIRC the restriction was put there to prevent a crash, probably related to CMYK profiles.

I prefer when the software lets the user choose what the user wants, so +1 to remove the filter if a crash can be prevented.

Could you upload some sample ICC profiles which can’t currently be selected?

2 Likes
9

That’s easy :slight_smile:

1 Like
10

Hi @Morgan_Hardwood - here is a (temporary) link to a sample camera input ICC profile that can’t currently be selected: https://ninedegreesbelow.com/files/sony98-am.icc

The relevant part of the profile is that it’s an RGB profle of the “input” class. When I made the profile I didn’t put in a proper copyright, but please consider it as CC0 if a copyright is needed.

11

@Elle @Morgan_Hardwood Where do you expect to select the ICC from ? There’s the Input profiles bundled with RT, which is not accessible to the Output profile combobox, and the system ICC profiles dir which is where RT get its output profiles. I don’t mind adding the “cmsSigInputClass” to the filter so you could select it, but looking into the bundled Input profiles would add too much profiles to that combo for 95% of the userbase I guess.

To prevent a crash, I’m only listing profiles with RGB channels, no CMYK profiles since we don’t handle that colorspace. Condition would then be :

  • Device class == Display OR Input OR Output
  • AND color space == RgbData
12

In Linux, users and software can put ICC profiles in a variety of places. Personally I keep the profiles that I’m actually using in /usr/share/color/icc, which is where RT looks when the user is selecting an output ICC profile, same as when selecting an input ICC profile.

13
  1. Is there an interest of soft-proofing the output profile instead of the printer profile, i.e. Lab → Output → Monitor ?
  2. Does it make any sense to make a triple conversion Lab ->Output → Printer → Monitor soft-proofing ?
14

If anyone wants access to the bundled ICC profiles for a given camera(s), the best thing for that user to do probably would be to copy the desired profie(s) to a folder where RT already looks. For some/many systems this probably does require being root or administrator. I can’t imagine anyone wanting to wade through all the bundled profiles every time they choose an output profile.

15

@Elle Filter removed in this changeset.

PS : Small correction : the profiles that are listed are already the bundled and personal ones…

16

Thank you thank you!

17

Yes, please. In particular I’d like to be able to see some indicators for out of gamut colours.

18

+1 - if the user plans to do any additional editing in the output color space, and given that RT output currently is integer, it would be nice to see what colors might be clipped before actually saving the image to disk.

19

Yes please. We had some discussion of this some weeks back here Monitor profile, rendering intent and soft-proofing tab · Issue #3054 · Beep6581/RawTherapee · GitHub
and there was support at the time.

20

Well, this is pretty much the idea behind rawproc, my raw processor hack. To your quest, I just did what I think you need by opening the NEF with input.raw.libraw.cameraprofile=Nikon_D7000_Sunlight.icc, then saved it to a JPEG with no additional processing and removing the sRGB profile from output.jpeg.cms.profile. Ta-da, even got the dialog, “No output profile specified, saving with working profile.”

Now, that this works in rawproc is due more to my ignorance than actually knowing how this is all supposed to work. But it makes sense to me, in that if there is no other colorspace transformation after a raw is opened with an assigned input profile, it should be saved with that profile as input for downstream work.

21

@Hombre you might be interested to have a look at how we are dealing with soft-proofing and gamut warnings in PhotoFlow, as we are using our own method which seems to be more accurate than LCMS2. The relevant code is here. Shortly speaking, we have two different methods depending if the proofed RGB profile is of matrix or LUT type. Support for CMYK profiles is work in progress.

You can PM me if you’d like to discuss some of the details…

22

Hi Glenn,

My apologies, I haven’t tried your rawproc, though doing so has been on my “to do” list for a long time. Not having ever used rawproc, it’s not clear to me what “saving with working profile” actually means. But generally speaking (and I’m sure you know this as your experiment was just to see what rawproc might do) it’s not a good idea to save a jpeg to disk while the jpeg is still in the camera input profile color space.

Generally speaking, camera input profile color spaces are not good color spaces for image editing.

That’s not worded strongly enough. Generally speaking, camera input ICC profile color spaces are horrible for image editing. This is true even when the camera input ICC profile is a simple matrix profile and even more true if the camera input ICC profile is a LUT ICC profile:

  • Most camera input profiles aren’t well-behaved RGB working spaces (https://ninedegreesbelow.com/photography/well-behaved-profile.html), meaning they don’t have a neutral gray axis, meaning if R=G=B, the resulting color isn’t neutral gray, with a=b in Lab space. And sometimes they made such that R=G=B=1.0 floating point doesn’t equal Lab (100,0,0), but instead has an L value that’s >100 or < 100). So “gray” isn’t really gray and “white” really isn’t white.

  • Even if you make and use a well-behaved custom camera simple linear gamma input ICC profile (Make a better custom camera profile), these profiles define huge color spaces where much of the total volume is taken up by imaginary colors, which all by itself makes these color spaces not suitable for general editing.

  • In addition even a well-behaved camera input profile color space has primaries that are placed such that adjustments that affect just individual channels are likely to produce unpleasant results. For example, raising just the blue channel using Curves, when done in Rec.2020 makes a blue sky look more blue. But doing this in the well-behaved linear gamma input profile I made for my Canon 400D camera makes the sky look unpleasantly purple.

I use my custom camera input profiles (I made a set for my Canon 400D and another set for my Sony A7) for the following very specific editing tasks, and only when the image has been “scene-referred” interpolated (no “make pretty” post-interpolation editing algorithms) before being saved to disk from the raw processor:

  • White balancing difficult images, including images with multiple light sources, using masks and layers and Levels using GIMP-CCE or other image editor that allows to edit in user-chosen RGB working spaces using masks and layers.

  • Retrieving the camera-captured RGB channel information for use as a layer mask and for black and white processing (Blue Channel "noise" in Raw Files) - once converted to a regular RGB working space, even using unbounded ICC profile conversions, “what the camera actually captured per channel” is gone, though for most images I’ve seen this isn’t a huge factor. But sometimes it is.

  • Blending multiple (usually but not always ev-bracketed) exposures by hand.

  • Dealing with imaginary colors that are sometimes produced by applying simple linear gamma matrix input ICC profiles to interpolated image files, in particular imaginary colors on the violet-blue/yellow axis. For example, I deal with imaginary saturated violet-blue colors using masks and layers combined with a custom ArgyllCMS Lab LUT ICC profile camera input profile that (unlike simple linear gamma matrix ICC profiles) doesn’t generate imaginary colors along the violet-blue/yellow axis, but otherwise does produce somewhat anemic colors, especially in blue skies.

Offhand I can’t think of any additional editing tasks that I’d want to do before converting the image layer stack to a “normal” RGB working space such as Rec.2020.

1 Like
23

@Elle,

  1. I should probably have demonstrated the camera profile save with a
    TIFF. That’d been more appropriate to what I think your use case is
    after.

  2. rawproc in its current version will use whatever profile is assigned to
    the opened image for further processing, and will save the output image
    with that assigned profile if nothing else is configured in the
    Properties. When a raw is opened, if nothing else is specified, the opened
    image will have the dcraw sRGB profile, with sRGB gamma applied. You can
    modify that with various input.raw.libraw parameters mostly corresponding
    to the libraw parameters for controlling dcraw processing. I added one of
    my own, input.raw.libraw.cameraprofile, so that if you tell libraw to open
    the image -o 0 -g 1.0 (“raw” colorspace, linear (no) gamma applied), it’ll
    assign the named icc file to the image. To apply a working colorspace,
    there is a tool called “colorspace” that you’d put in as the first
    manipulation, specify the icc filename and ‘apply’, and it will transform
    the image to conform to that profile and attach that profile to the image
    for the remainder of the processing chain until either 1) you put in
    another colorspace tool, or 2) save the image. At the time of save, an
    output..cms.profile parameter (’’ is the output type, ‘jpeg’ or ‘tiff’ at
    present) is consulted, and if there’s an icc file specified it is applied
    to the image before saving, and that profile is attached to the outputted
    file. If output.*.cms.profile is blank, the image is saved with whatever
    profile followed it through the processing chain, which I referred to as
    the ‘working’ profile in the dialog. Yes, this can be done for either a
    JPEG or a TIFF file save. I can see the implications of saving a JPEG image
    with a profile that outstrips its gamut, but I’d rather allow that than to
    start putting in the sort of protections your original post to the thread
    considers.

Soooo, to get where I think you want to go in your original question using
rawproc, you’d open the camera raw with
input.raw.libraw.cameraprofile=“yourcamera.icc”, and then immediately save
to TIFF with output.tiff.cms.profile="". The resulting TIFF image would
still be in camera colorspace, and the attached profile would also be in
camera colorspace. No tools applied between open and save.

My first attempt was to put the cameraprofile.icc in output.*.cms.profile,
but that crashed the program. Debugging showed that cmsTransform had no
valid transform, LittleCMS didn’t like using the camera profile as input.
Makes sense, the rawproc save assumes a colorspace apply, not assign.
Still, I need to trap that, commit a fix to check the transform before
using it.

What I’m trying to build with rawproc is a toolbox, and the means to apply
the tools in a very flexible and straightforward manner. So, if an
operator chooses to open a raw with a camera profile and not apply a
colorspace tool with a proper working profile as the first order of
business, they reaps what they sows. Hammering screws, so to speak…

btw, I’ve spent a lot of quality time with your articles, and they’ve been
the primary source for helping me incorporate a “toolbox notion” of color
management into rawproc. Particularly, I’ve tried hard to conform to the
“apply” and “assign” semantics for colorspace usage. Now at home, I have a
single sunlight-calilbrated D7000 camera profile and two calibrated display
profiles, one each for my desktop and tablet, and things seem to be working
as intended except for opening a raw with a camera profile and applying a
working profile at the start. That seems to oversaturate the colors, far
more that what I’d expect, but I haven’t had time to completely test the
combinations of the input->working->display/output chains to see if/where
I’m doing something wrong.

Thanks for the involved reply. I’m still learning this color management
business, and your prose is extremely instructive in that regard. That
said, I hope you appreciate the angst fomented upon us hapless programmers
trying to make tools that work responsively, without crashing… :smiley:

24

Hi Glenn,

My apologies, I’m not sure what you mean by “apply” and “assign” semantics, but it seems you are talking about something I wrote in an article on my website. Could you give a link?

Hmm, when you say that “opening a raw with a camera profile and applying a working profile at the start . . . seems to oversaturate the colors, far more than I’d expect”, this indicates to me that somewhere there is a problem in the color management chain, but as I said, I’m not sure what you mean by “applying a working profile”.

If “applying a working profile” means “assign the camera input profile to the interpolated raw file, and then convert to the RGB working space profile”, then the image should look pretty much the same before and after the conversion, all things being equal. “All things being equal” covers things like the respective color gamuts of your monitor profile and your working space, and the actual colors in the raw file as interpreted by assigning the camera input profile to the interpolated raw file, and of course presumes that color management is being used to send the image to the screen.

This morning I tried to compile rawproc from git to get a handle on how color management is being handled, but gave up when I realized that first I’d have to download and compile wxWidgets from git. That’s a huge download. I thought wxWidgets was available from Gentoo portage, but if it is, maybe it’s under some other name?

As far as what I personally want from raw processors, that’s fairly well covered by PhotoFlow, darktable, and RawTherapee. Though I still would like a raw processor that is as fast as ufraw/nufraw, and works more or less like ufraw/nufraw, but with true exposure compensation (UFRaw blown highlights), with the AMAZE algorithm, and with floating point tiff and/or exr output. This is one of the reasons why “compile and try rawproc” is on my “to do” list - I was wondering if rawproc might be a possible replacement for ufraw/nufraw.

Oh, yes I do appreciate the angst :slight_smile: . Several years ago, when I spent some time modifying dcraw to make its internal processing be floating point and to add some additional interpolation algorithms, my appreciation for developers increased exponentially. Learning enough c code to modify dcraw and get it to run was not easy!

While working on the dcraw code I learned about projects that developed various interpolation algorithms not in dcraw, such as the people working on Guillermo Luijk’s PerfectRaw (GUILLERMO LUIJK >> PROGRAMAS >> PERFECT RAW - not to be confused with the totally unrelated PhotoShop colorneg/perfectcolor/perfectraw plug-ins) and the followup coding efforts by Rawness, RawTherapee, LibRaw and other groups involved with writing raw processing code. Wow! so much work and creativity has gone into and continues to go into our raw processing programs!

On the same line of thought, free/libre software depends on people volunteering their time and effort for coding, and also for providing feedback, making good bug reports and following through with testing, reading and responding to bug reports (that’s a huge task all by itself), helping to write user manuals, helping each other on mailing lists and in forums like pixls.us, writing articles and making videos on free/libre software, setting up and maintaining forums and mailing lists, and on and on.

It’s the sum total of everyone’s efforts that makes free/libre software possible. But without the time and effort of all the developers who write the actual code for free/libre software, there’d be nothing at all for the rest of us to contribute to.

25

Ack, that’s what I get for relying on my memory, should be “convert” vs “assign”, per your article here: https://ninedegreesbelow.com/photography/convert-assign.html

I very much came to appreciate the journey you took with dcraw-undnged, dcraw-unDnged, which is why I added the input.raw.libraw.cameraprofile parameter. Long story short, if you set the following parameters in rawproc:

input.raw.libraw.colorspace=raw
input.raw.libraw.gamma=linear
input.raw.libraw.cameraprofile=yourcalibratedcameraprofile.icc

libraw delivers the raw linear camera image just like dcraw would with -o 0 -g 1.0, and my software assigns the specified .icc profile to it. That’s the starting point in rawproc; to proceed from there ‘color-managed’, you’d first add a colorspace tool and use it to apply (convert to) a working profile speciied as a .icc file. Or, if your camera profile is well-behaved, you can just start adding tools and they’ll work on the raw-linear image. The image displayed during work at each step is converted using a display profile, another .icc file. When you go to save, the image at the end of the chain of applied tools is either converted to a specified ouput profile, or assigned the profile that followed it through processing.

This is a thread hijack, but here I go: when open a raw as described above, the colors look right to me, but when I add the colorspace tool and apply a working profile, I get rather garish colors. Thing is, if I specify the camera profile in the colorspace tool, the resulting display image looks the same as before the colorspace tool. That’s where I need to spend some time picking apart my cmsTransform chains…

Back to the thread, the reason I’m describing all this is to put forward one approach to getting the sort of flexible, ‘do it my way’ raw processing toolbox you’re looking for. The DT and RT programmers have a quandry in this regard, judging just how much flexibility their user base can stand. Me, with a user base of one, not so much. I would be curious to hear how the AppImage works with Gentoo, if you are so inclined, one way to avoid dealing with wxWidgets…

26

To avoid further hijacking :slight_smile: I replied to your comments over in this thread: