Hmm, after pulling in today’s code and recompiling, I wasn’t able to trigger the same behavior. I did recompile using “debug” instead of “RelWithDebInfo”, and maybe that made a difference, but I’m not planning to recompile and check any time soon.
The amount of RAM used by RT did continually go up per additional save, regardless of whether saving at 16-bits or 32-bits. But the maximum it reached even saving 32f, even saving 7 or 8 times, was less than 3GB, and the RAM readout on my taskbar never did get high enough to notice.
Oh, wait, is “-DCMAKE_BUILD_TYPE=” case-sensitive? Because I typed “debug”, not “Debug”.
That is why I don’t build anything anymore. It is too much work. All I found regarding your question is https://cmake.org/cmake/help/latest/manual/cmake-language.7.html#variables but it just says that variable names are case-sensitive. No mention of their values. The docs say certain things are case sensitive and other things aren’t so it is a world of confusion.
Regarding Linux, it’s literally a copypasta of two code blocks (one for getting the dependencies and toolchain, one for compiling RT): http://rawpedia.rawtherapee.com/Linux
Infinite loop confirmed. The trick is to save the f32 TIFF under a new name. One thread then loops in ExifManager::parse() while consuming more and more RAM. I’m digging deeper…
diff --git a/rtexif/rtexif.cc b/rtexif/rtexif.cc
index 47d885e50..b6a60bbd8 100644
--- a/rtexif/rtexif.cc
+++ b/rtexif/rtexif.cc
@@ -3077,7 +3077,7 @@ void ExifManager::parse (bool isRaw, bool skipIgnored)
root->printAll ();
#endif
- } while (ifdOffset && !onlyFirst);
+ } while (ifdOffset > 0 && !onlyFirst);
// Security check : if there's at least one root, there must be at least one image.
// If the following occurs, then image detection above has failed or it's an unsupported file type.
@floessie could you give some more clues as to how to reproduce? I tried saving a 32-bit TIFF, compressed and uncompressed, to /tmp but didn’t notice any leaking memory or strange CPU activity. Does the file browser need to have the output folder open? Does the TIFF need to be compressed? Are any metadata changes necessary?
@agriggio@Morgan_Hardwood@Elle@snibgo Do we need to wait for the engine update to remove the clamping when reading floating point image data ? Scaling the channels seem to make the colors wrong, so maybe having clipped HL is preferable ?
Does “remove the clamping when reading floating point image data” refer to reading a floating point image that was saved to disk by some other image processing software such as GIMP-2.9?
Does “scaling the channels” mean normalizing the channel data to fit within the display range 0.0f to 1.0f?
What does “clipped HL” mean?
Let’s say someone opens with RawTherapee a floating point image that’s in an RGB color space that’s not one of RT’s “Working Profiles” color spaces. What’s the first thing that happens with the current RawTherapee code. Is the image immediately converted to one of RT’s Working Profiles? Or maybe to CIELAB?
Does “remove the clamping when reading floating point image data” refer to reading a floating point image that was saved to disk by some other image processing software such as GIMP-2.9?
Yes
Does “scaling the channels” mean normalizing the channel data to fit within the display range 0.0f to 1.0f?
Yes
What does “clipped HL” mean?
Clipped highlights
Is the image immediately converted to one of RT’s Working Profiles? Or maybe to CIELAB?
Actually, for 32 bits floats images, the data is scaled to 0.0-1.0 based on the detected min/max values, then converted to the embedded profile.
Hmm, in that case instead of normalizing the data, I’d recommend just clamping it and letting people know it will be clamped to the channel range 0.0-1.0. That way the user can (is forced to) make their own decisions about how to deal with out-of-display-range channel values before they bring the image into RawTherapee.
As you already noted, simply normalizing the channel values can do funny things to the colors. I think it would be easier to tell users “for now, until the processing pipe line is unclamped, channel values are clamped on import”, than to explain to users why all of a sudden the image colors are all different than they were before opening the image with RawTherapee.
I agree with Elle about not normalizing the data based on the min/max values.
Have a look at this file, which obviously lacks highlights
First the raw in rt:
Now the same image saved to float tiff, loaded and normalized to [0;1] based on min/max values.
Edit: as (technically) expected the normalizing stretches the histogram in this case. But I don’t think that’s a user’s expection.
Edit: Here’s a screenshot of the same tif without normalizing based on min/max values.
When using the dev version of RawTherapee (updated yesterday, I think) via the RT-GIMP plugin, is there a way to tell RawTherapee to send a 32f instead of a 16i image?
Hi @Morgan_Hardwood and thanks! That particular edit didn’t work. But replacing all references in the RT code to “tiffBits =8” and “tiffBits = 16” to “tiffBits = 32”, did the trick. Of course this wouldn’t be suitable for general usage, but for my purposes it works just fine. So I’m guessing the “single place” (if there is one) that would require changing to provide 32f output via the GIMP plugin is in one of these files:
I also looked at the actual GIMP-RT plug-in code, but nothing looked promising. Though that doesn’t mean much as I don’t find the plug-in code very easy to read, and I didn’t look very hard to see if the plug-in code could control the bit depth of the returned file.
One thing that will need to be done as part of unclamping 32-bit floating point throughput and output is making sure all the ICC profile transforms and also the output ICC profiles support unbounded floating point conversions.
Not all ICC profile TRCs fully support unbounded floating point conversions, and some don’t even partially support unbounded floating point conversions:
Point curves don’t support unbounded floating point conversions. Everything is clipped to the display range.
True gamma TRCs (other than gamma=1.0) allow for unbounded positive channel values, but clip shadow channel values to 0.0. Which means “out of gamut” colors are clipped to zero in the shadows, which in turn means the potential for data loss, and what’s left of the original out of gamut data has undergone LCH Hue and Chroma changes.
Linear gamma (gamma = 1.0) ICC profiles do fully support unbounded floating point conversions, but of course only if the TRC is actually a gamma curve instead of a point curve.
The sRGB, Rec.709, and LAB L TRCs do support fully unbounded floating point conversions but only if these curves are encoded using a parametric curve, which means using a V4 profile, and of course requires not asking LCMS to output a point TRC.
I’m assuming any other Type 4 parametric curve that has a small "linear portion in the shadows also will fully support unbounded floating point profile conversions (and maybe also Type 5? and 6? see the LCMS API documentation table of parametric curve types). But the “point curve” equivalents of these parametric equations don’t support unbounded ICC profile conversions.
Looking at RawTherapee-supplied output profiles, the only ones that support unbounded floating point conversions are ACES.icc (linear gamma TRC) and I’m guessing the two DCI-P3 profiles, which seem to have nonlinear true gamma curve TRCs, and so support channel values > 1.0 but not channel values < 0.0.
All of the remaining RawTherapee output profiles have point curves, including the linear gamma version of sRGB and the linear gamma version of LargeRGB, and so don’t support unbounded floating point profile conversions. Instead RGB values are clipped.
There are some other problems with RawTherapee ICC profiles: Most of them aren’t completely well-behaved, and most of them have slightly wrong red, green, blue, and V2 white point tags:
“Rec2020.icm” is a copy of the ArgyllCMS (public domain) Rec2020.icm. All the ArgyllCMS ICC profiles are well-behaved.
The second well-behaved profile (“RT_Medium_gsRGB.icc”) brings up a problem that affects most of the RawTherapee output profiles, which is that the white point, red, green, and blue XYZ tags deviate from what you’d get if you made the profiles exactly according to the various color space specs, and also don’t match the ArgyllCMS equivalent profiles (when there is one), and also don’t match existing old proprietary versions, for those profiles for which I could find such profiles.
According to the profile description “RT_Medium_gsRGB.icc” is supposed to be a variant of the AdobeRGB1998 profile, except with the sRGB TRC. But the profile XYZ tags aren’t correct. Here are the white, red, green, and blue tags from the RawTherapee profile, compared to the same tags from the ArgyllCMS “ClayRGB.icm” profile (the ClayRGB.icm profile exactly matches the AdobeRGB1998 specs):
TagSignature>wtpt
RT XYZNumber X=“0.95016479” Y=“1.00000000” Z=“1.08842468”
AR XYZNumber X=“0.95045471” Y=“1.00000000” Z=“1.08905029”
TagSignature>rXYZ
RT XYZNumber X=“0.60961914” Y=“0.31103516” Z=“0.01947021”
AR XYZNumber X=“0.60974121” Y=“0.31111145” Z=“0.01947021”
TagSignature>bXYZ
RT XYZNumber X=“0.14916992” Y=“0.06320190” Z=“0.74453735”
AR XYZNumber X=“0.14918518” Y=“0.06321716” Z=“0.74456787”
TagSignature>gXYZ
RT XYZNumber X=“0.20541382” Y=“0.62576294” Z=“0.06089783”
AR XYZNumber X=“0.20527649” Y=“0.62567139” Z=“0.06086731”
Similar problems obtain for many of the other RawTherapee output profiles, including the sRGB profile. The problems have a couple of different sources:
Using the wrong values for D65 and D50
Not correcting for hexadecimal rounding during the profile-making process.
Using the LCMS built-in sRGB profile-making function (“cmsCreate_sRGBProfile()”).
For details see these articles:
Will the Real sRGB Profile Please Stand Up?
From sRGB color space to sRGB profile: how to calculate the ICC sRGB profile primaries from the sRGB color space specifications
Survey of Free and Open Source ICC RGB Working Space Profiles
Are Your Working Space Profiles Well Behaved?
In Quest of Well Behaved Working Spaces
Elle Stone’s Well-Behaved ICC Profiles and Code
I think it’s important that free/libre software have properly made ICC profiles, and from what I can tell, an awful lot of profile suppliers for free/libre software don’t actually provide completely correct profiles.
If someone says “well, our profiles are close enough”, that’s a statement that just leaves me confused - why not be “exactly right” instead of “sort of right”, when there’s no particular cost associated with using the right parameters to make the profiles?
If someone wants to say “well, we prefer using this other value for D65 and/or for D50”, then that’s fine, but the resulting profile shouldn’t be passed around as a profile for the intended color space, as color space specifications include not just the primaries, but also the white point values.
My profile-making code is GPL’ed and so anyone can borrow and modify any or all of it to incorporate into their own projects, and of course if anyone finds errors, please let me know! The most important parts of my profile-making code are the “spec-based” white points and the back-calculated matrices for profiles that otherwise wouldn’t be well-behaved. To calculate these matrices I use one of the ArgyllCMS source code files modified separately for each profile, plus a spreadsheet to calculate these matrices. ArgyllCMS has code for correcting for hexadecimal rounding, to make well-behaved profiles. LCMS doesn’t have such code.
I’m a satisfied customer of your profiles. I use the Rec2020-g18 for working, and the sRGB-srgbtrc for JPEG output. Those, and my calibrated camera and display profiles and my color appears consistent among all my home monitors. Need to find time to bring my colorimeter to work…
Are you sure about those numbers? To me they look an awful lot like the wrong ones I recently fixed in darktable: D50 adapted primaries with a D65 white point!
