Feature request: save as floating-point

[agriggio (Alberto)]

Hi all, some further progress on this. I’ve just pushed a couple of commits to the unbounded-processing branch to not clip negative values. The basics seem to work, but I haven’t done much testing yet.

Here’s a quick demo using the yellow flower provided by @Elle:

https://filebin.net/51evd9es82j62p7n/Peek_2018-02-19_16-30.mp4

Note that it is important to use one of @Elle’s linear gamma ICC profiles as output profile (I was using LargeRGB-elle-V2-g10), as the default ones of RT will clip negative values, even if you set the output gamma to linear (I’m still not sure why though).

[Elle (Elle Stone)]

OK, here are the raw files:

raw-files-for-rt.zip (61.5 MB)

The zip file has a note with copyright and white balance information. I also included a custom camera input profile for the raw files for the two cameras.

[agriggio (Alberto)]

Thanks @Elle for the files. I’ve developed them in RT using the unbounded-processing branch, and I’ve just put the resulting TIFFs (together with the pp3’s) at https://filebin.net/061gfuguhz4zx4z7
Everything seems fine to me – though I basically used neutral + your custom input profiles, and, importantly, your LargeRGB-elle-V2-g10 output profile, so it might still be the case some tools clip, particularly on the 0 side.

[Elle (Elle Stone)]

Hi @agriggio - thanks! for checking the files. A couple days ago I did notice a problem when processing the “very yellow flower” raw file (not from the tiffs you uploaded to filebin.net, rather I processed the raw file using the GIMP plug-in).

Double-checking today, I updated rawtherapee tthis morning, still on the floating point branch, recompiled, and opened the yellow flower raw file from within GIMP-2.9. I used the custom camera input profile (that’s in the zip file with the raw files) for my old Canon 400D as the input profile and also the output profile.

The image was transferred to GIMP without any problems, and still in the camera input profile color space, which is what I wanted. The blue channel is just right, still has all the original blue channel detail. But the red channel is clipped over substantial parts of the image:

In the original raw file there is a tiny spattering of clipped pixels in the green channel, but no pixels are clipped in the red channel. However, after applying the daylight white balance, a considerable portion of the red channel is driven to above 1.0 floating point. Figure 4 in this article shows the clipped green pixels: UFRaw blown highlights, and Figure 2 shows a similar-to-RT pattern of clipped pixels in the red channel. The clipped pixels in UFRaw are from a lack of true exposure compensation - UFRaw clips before doing exposure compensation, but I don’t think this is what RT is doing, leastways I’ve never seen this problem in RT before.

For 16-bit output, I would set the raw exposure compensation to 0.5 or even 0.25, to avoid clipping the red channel. But for floating point output I’d expect these “above 1.0f” channel values to be brought along without clipping (what darktable and PhotoFlow do). Is there a setting in RawTherapee that I didn’t set correctly? Here’s the pp3 file:

090916-1718-109-1543.cr2.pp3 (10.5 KB)

[agriggio (Alberto)]

strange…what output profile are you using? do you have highlight reconstruction turned on? anyway I’ll double check as soon as I can

[Elle (Elle Stone)]

For the output profile I used the camera input profile for the Canon 400D, the same profile as was included in the zip file with the other raw files. This camera input profile is a well-behaved RGB working space with a linear gamma TRC (not a point curve).

Reopening the raw file via the GIMP plug-in and searching all the tabs, no, I don’t see anything activated that’s related to highlight reconstruction.

[agriggio (Alberto)]

Hi @Elle,
I finally had the time to take a look:

@Elle

no, I don’t see anything activated that’s related to highlight reconstruction.

This is the problem. Just enable highlight reconstruction and it should work:

I’ll try to change the code to make this not necessary, as soon as I have some time…

EDIT: here’s a little demo
https://filebin.net/kihrde1yjpiemple/Peek_2018-02-25_21-21.mp4

[agriggio (Alberto)]

@agriggio

I’ll try to change the code to make this not necessary

should work now even without highlight reconstruction enabled

[agriggio (Alberto)]

Hi, a quick update on this. Yesterday I’ve merged the unbounded-processing branch (yes, I wrote the u-word into dev, so now it is possible to save as 32-bit float TIFF without any clipping. Interested people are encouraged to test this extensively, as there might still be some parts of RT that do not support this properly.

Note though that you will need an output profile that allows lcms2 to work in unbounded mode. For instance, RT_sRGB is not one of them. I suggest the profiles of @Elle, though any other “matrix-shaper” profile should do.

Finally, on a related note, now it is also possible to specify custom RGB working spaces. Instructions on how to do this are on RawPedia.

[Morgan_Hardwood]

Maybe @jdc could help get the RT_* profiles to work with the new mode?
@agriggio could you explain why our current profiles cannot be used and what it is about @Elle’s profiles that makes them compatible?

[Elle (Elle Stone)]

If I recall correctly existing RT profiles all use point curves, even the ones with pure gamma TRCs. Also (again from recall) existing RT profiles are V2 profiles, so the sRGB/LAB companding/etc TRCs are point curves by necessity. Parametric curves require V4 profiles.

[agriggio (Alberto)]

as @elle wrote, RT profiles (maybe not all of them though) use tables to implement TRCs, that’s the problem

[Morgan_Hardwood]

Where can I read about alternatives to using tables for TRCs?

[agriggio (Alberto)]

lcms API, page 135 (“Tone Curves”) and following, look for “parametric curves”

[Elle (Elle Stone)]

Here are the more commonly used parametric curves, set up using the LCMS API:

  cmsToneCurve *tonecurve;

  linear gamma:
  tonecurve = cmsBuildGamma (NULL, 1.00);
  
  gamma=1.8:
  tonecurve = cmsBuildGamma (NULL, 1.80078125); 
  
  gamma=2.2:
  tonecurve = cmsBuildGamma (NULL, 2.19921875);

  the sRGB TRC:
  cmsFloat64Number srgb_parameters[5] = 
  { 2.4, 1.0 / 1.055,  0.055 / 1.055, 1.0 / 12.92, 0.04045 };
  tonecurve =  
  cmsBuildParametricToneCurve(NULL, 4, srgb_parameters);

  Rec709:
  cmsFloat64Number rec709_parameters[5] = 
  { 1.0 / 0.45, 1.0 / 1.099,  0.099 / 1.099,  1.0 / 4.5, 0.081 };  
  tonecurve = 
  cmsBuildParametricToneCurve(NULL, 4, rec709_parameters);

  LAB companding curve:
  cmsFloat64Number labl_parameters[5] = 
  { 3.0, 1.0 / 1.16,  0.16 / 1.16, 2700.0 / 24389.0, 0.08000 };
  tonecurve = 
  cmsBuildParametricToneCurve(NULL, 4, labl_parameters);

If you use gamma=2.2 or gamma=1.8 instead of the odd values I give above, the resulting TRC won’t match the TRCs in V2 profiles, and in the case of AdobeRGB1998, won’t match the actual AdobeRGB1998 specs from Adobe.

[jdc (Desmis)]

If the problem is to generate ICC V4 instead of ICC V2, I don’t know how to do
Buu if there is profile done by @Elle that match these specifications , I think - if no objections - we can used

[Morgan_Hardwood]

How does one generate v2 or v4 ICC profiles using open-source software?

[Morgan_Hardwood]

There is DisplayCAL’s Synthetic ICC program, which I used to generate RT’s ACES profile, but I see nothing about LUT points or v2 vs v4:

[Elle (Elle Stone)]

https://ninedegreesbelow.com/photography/lcms-make-icc-profiles.html

ellelstone/elles_icc_profiles

[agriggio (Alberto)]

@elle has code on github to do that, using lcms2. But:

@jdc

if there is profile done by @Elle that match these specifications , I think - if no objections - we can used

I agree with Jacques, why can’t we just ship those profiles?