Purpose
The linear workflow is an important part of processing that has been discussed and debated throughout the forum. I would like this thread to collect the current understanding of devs and researchers on how they see linearity should be maintained and used in the processing pipeline.
Post Guidelines
By survey, I mean the formal sense, not opinion or endless debate.
I am sure there will be disagreement. If that is the case, you may state your case in 1 post; the person to whom you are replying gets to reply once. Of course, some will warrant further exchanges, but I am putting it this way to emphasize restraint and respect among peers.
State the approach or topic (1/post), its merits and demerits, and show examples of its use. Please include links, files and references.
PS Also address edge cases: negative, high, inf, nan, imaginary values, high gamut, etc.
Here are some preliminary thoughts; I may edit this if I have more well-formed opinions later.
When the underlying mathematical model of some algorithm is based on a linear space (e.g. uses a Poisson noise model, such as LucyāRichardson deconvolution), then I believe strongly it is appropriate to perform it in a linear space.
On the topic of performing other operations in a linear space, I do not know.
Here is a discussion of compression. I realize RT does not implement its own compression algorithm, but the JPEG compression discussed at the link is wavelet-based. So this suggests the wavelet algorithms should be applied after the gamma correction/tone curve is applied.
Resizing and sharpening may be better done after gamma correction; see that link for sample pictures.
Dan Margulis writes: āWhile blurring might be better at an ultra-low gamma, sharpening and addition of noise are worse, so having filter operations take place at a low gamma generally is a bad idea.ā I guess by filter he means things like the guided filtering used in the RT highlights/shadows module.
edit: To mimic the effect of a color filter on a camera shooting black and white film, any āchannel mixerā for black and white conversion should operate on the linear data.
Firstly, I want to lay down my comprehension of the definition of ālinearā with respect to digital imaging. A lot of what I think I understand is based on this comprehension, so clearing up any misconceptions would help me personally, as well as might beneficially shape the conversation. So, my understanding of ālinearityā is prepended with the word āradiometricā, which identifies the characteristic I think is important to consider in maintaining the ālinearā relationship of light measurements. At each location on the sensor array there is collected a value representing the intensity of the light focused on that location, and itās each measurementsā relationship to each other that I think is important to maintain as long as possible.
My proof workflow does the following:
convert each 16-bit integer number from the raw file to an equivalent value in the 0.0-1.0 floating point space. This act would foment a lot of ancilliary discussion, but suffice to say that there is no net effect on linearity, each number in multiplicative terms still has the same relationship to its siblings.
assign a camera space to the image - no change of the values, but will be useful for display/output conversion.
blackpoint subtraction. No change of the multiplicative relationships, all data moves down by the same camera-specific value. Effectively establishes the place where the color black is anchored in the data.
whitebalance. This one i struggle with a bit, as while the essential operation is a mulitplication, different multipliers are applied to each channel, radically changing the channelsā radiometric relationship. Iām doing it this early because dcraw does it about here, and I havenāt done any comparative analysis to assess its placement. Well, I did look at baking the white balance correction into the profile transform some time ago, but I never had the time to figure out how to work per-session camera profiles into my workflow.
demosaic. For proofing, I use the half algorithm, which just builds each pixel from each quad mosaic, no change to the measurements. I just end up with a half-size RGB image with the same measured intensities. I donāt understand the ārealā demosaic algorithms enough to comment on their radiometric intrusion.
Okay, this gets me to a RGB image whose white point is still where the camera comprehended it as the saturation limit, for my cameras thatās 16383 (14-bit raw) in 16-bit integer, or about 0.249948 in 0.0-1.0 floating point. So, the next tool I apply is a black/whitepoint scaling, in GāMIC thatād be the -normalize operator (or -cut, I canā recall). I think some folk use exposure compensation to do the same thing, but the transform is equivalent: a multiplication of each measurement to put the very highest measurements at 1.0 floating point. 1.0 is eventually mapped to the display notion of white. For my images with small to medium dynamic range, the proof image at this point is usually sufficient for regarding, no basecurve/filmic/loggamma pet tricks required. Someone tell my why this isā¦ ??
For proofing, at this point I just resize to a decent size (~800x600), minimum USM sharpen, and save to JPEG. WB aside, bear-of-little-brain here thinks the only departure from linear in the above workflow was the conversion to sRGB gamut and tone for the output, to comply with the display norms @troy_s attempted to explain some time ago.
Now, dynamic range of the scene is what Iāve come to realize vexes this simplistic workflow. Indeed, to preserve highlights at capture, ETTR would compel one to underexpose a high-DR scene, which will put most of the image tones way down toward black. This, to my experience, is where departure from linear is required, in order to move those lower tones into the mid-range while keeping from pushing the highlights into oblivion.
Note the workflow described above is captured in a group tool; this is new for the upcoming 0.9 release. The blackwhitepoint is set to scale the data from its min and max values to 0.0-1.0. The tone tool is next; I pulled the parameters pane out of the dock to show the complete tone tool settings- the filmic tone curve is selected, and itās plot is at the bottom of the parameters pane. This curve is a very non-linear transform, and until this tool the data to my maybe warped thinking, was āradiometrically linearā. The display is generated from the sharpen tool, last in the chain, and is approximately sRGB, color and tone.
If I were to select the blackwhitepoint tool for display, the image would be quite dark overall. I used the Z6 highlight-weighted matrix metering mode to preserve the highlights; noteably, the in-scene light sources are blown, but the sky above the building is not. The filmic curve pulls the dark majority to comfortable visibility.
Iāve recently had to rework my blackpoint handling, e.g., the subtract tool is new to accomodate my new camera. Also, some operations push data negative, Iām still working on how to informatively consider this, but for now the data min/max of the blackwhitepoint tool just clips at 0,0.
With respect to high values, I tend to let my processing push data past 1.0, then I look at that outcome in display, and sometimes Iāll modify a curve to pull things back. Using floating point internal representation makes this trivial to handle.
With respect to gamut, until recently my practice was to, immediately after demosaic, convert the image from camera space to a Rec2020 working space. Iām not doing that right now, because Iāve seen artifacts I donāt like, and they donāt occur if I save the colorspace conversion for final display/output. Not understanding that, to date.
@afre, a good thread, Iāve been musing over such recently with the new camera, and trying to make pragmatic sense of all that āunboundedā discourse from a while back.
Edit: So, @afre pointed out something Iād not captured in my original post, that I needed to elucidate what Iāve learned. Okay, what Iāve learned is that departure from scene-linear is really only needed for two reasons:
Accommodation of non-linear displays. @troy_ sās lesson.
Compression of a sceneās dynamic range that challenges our camera and/or display capabilities.
The purpose of this thread (and forum) is to share with one another what we have learnt. Thank you for participating and pointing out your gaps of knowledge. Hopefully, after a dozen more posts, we will have a better appreciation of the challenges involved.
Thanks for that, @ggbutcher. You might include denoising before the demosaic.
I donāt quite follow. If you subtract a constant from two different values, the ratio of the new values will be different to the ratio of the old values. So there is a change of multiplicative relationship, isnāt there? Hence the values are no longer āradiometric linearā.
Ah, thanks for straightening out a math-adverse goofballā¦
Yes, Iām a bit out of my field with the mathematics. Indeed, one of my on-going objectives is to precisely understand āradiometrically linearā in these terms. My general feeling about all this is that the original measurements of light energy are important to the colorimetric and tonal interpretation of the scene in an image, and each manipulation should be carefully considered, and limited to the minimum required to achieve the desired rendition.
For instance, Iāve been messing with the original filmic function, trying to better-understand its effect. In my current tool chain, it is the first significant departure from linear, things like blackpoint subtraction and whitebalance notwithstanding. There are discussions out there that implore a subsequent application of an S-shaped curve after filmic āliftsā, and that to me seems to be undue whipsawing of the tonality. So, Iāve been playing with the A, B, C, and D coefficients, and I find that I can bake most of what Iād do in an S-curve into them. I donāt fully understand their interaction, but for instance, the B coefficient is key to shaping the curve toe and the tonality of the lowest shadows. In the recent āinfinite flowersā playraw i posted a filmic-Scurve rendition, but I could probably have baked it all into the filmic toolā¦
@afre, I know you want to capture our thinking in single posts, but itās hard not to discuss itā¦
@ggbutcher Which means, permission granted. Hint: It is okay to list things in your first post but try to be focused in subsequent posts, separating topics into manageable chunks.
Although I have things to say about the replies so far, I will wait until more people get a chance to contribute. That said, question: is normalizing data to [0.0-1.0] radio-metrically acceptable?
A very good question, one that Iāve pondered incessantly of recent. Iād surmise, yes, as the numbers used to represent energy measurements are arbitrary, but the ādistanceā between them expresses their energy relationship. Proportionately, I think thatās maintained in a normalization, as I believe it is with an exposure multiplication. That, from a person with three university degrees, containing four math courses, totalā¦ - FWIW!
I suspect this has been brought up before; just donāt remember where.
ā @Elle has spoken about arithmetic with respect to blending, and gamut; uncertain if she addressed this. Unfortunately, she isnāt here anymoreā¦
ā I recall @David_Tschumperle telling me to divide by a factor to get to the [0.0,1.0] range and then multiply to restore the original range. Out of habit, I still normalize more than I divide and multiply.
In rawproc I handle normalization a bit oddly, it seems. When I read raw ADU values, unsigned 16-bit integers from libraw, I scale them to equivalent numeric values in 0.0 - 1.0 floating point. That is, say, 16383, which is the maximum 14-bit value, gets converted to 0.249984741 in 0.0-1.0 floating point. Thatās where raw images start in rawproc; I consider that representation to be ālinearā with respect to the original energy.
At some point in the chain, usually right after demosaic, Iāll stick a blackwhitepoint tool. Thatās where the data is scaled such that the original saturation point is brought to 1.0. āblackā usually doesnāt mean anything here, as the data has already been anchored to 0.0 with blackpoint subtraction, if needed. Out of habit, I use a slope-based ālinear curveā; I used to use the curve tool but thatās computationally expensive. If just moving the saturation point, it could just be a multiplication, I thinkā¦
After that operation is where Iāll put non-linear tone curves, which I have coded to optionally normalize to the 0.0-1.0 range. That is, the curve tops off at 1.0, so data thatās already < 1.0 will not be pushed past 1.0. Iāve assembled a little āzooā of tone operators: gamma, reinhard, loggamma, and filmic, and Iām currently messing with filmic to see if it has utility working with the highlight-weighted metering of my new camera. So far, emphatically yesā¦
Thing is, my proof workflow uses an option of the blackwhitepoint tool that pushes the black and white points āinto the histogramā, that is, arbitrarily clips the extreme values to a preset proportion. Often, that is sufficient to render a useable image, particularly if the scene dynamic range was small. So, a lot of my images go out the door with only the sRGB TRC to āmake niceā for the wild web. Linear, from start to almost finishā¦
It seems to me there are two related but different concepts:
(1) Linearity, which I take to mean pixel values are proportional to the energy received by the camera, or hypothetical camera.
(2) Scene-referred values, which are linear but also have no maximum because whatever illumination the original scene has, it can always be increased. By contrast, output-referred values do have a maximum because screens and paper have a maximum brightness.
Scene-referred also implies that a collection of such images (including videos) could be edited together in some well-defined manner to work together seamlessly, regardless of different cameras, software, and so on. Thatās a large topic.
I deal in photos and videos, where achieving (1) is fairly easy. Modern cameras seem to be fairly linear. If they arenāt, a correction isnāt difficult. And this may be where @ggbutcherās black-level subtraction comes in: for such cameras, a value is subtracted in order to make values linear. Or, at least, values are then assumed to be linear.
Multiplying or dividing linear values by a constant will retain their linearity. But capping values at some limit will destroy linearity. Many operations to improve aesthetics, āthe lookā, will also destroy linearity.
But (2) is more difficult. There is no maximum value, there is no whitest white. (Or there is, but it is at infinity.) Many operations we are familiar with donāt exist or need to be revised. For example, a scene-referred image canāt negated, interchanging black with white.
A low-level non-interactive tool such as ImageMagick handles linear data with ease. Almost nothing in IM cares whether the data is linear. The arithmetic doesnāt change. Scene-referred is messier because IM originated in the days of integer-only processing, and we need to switch off mechanisms that clamp values beyond āblackā or āwhiteā.
A GUI is more difficult because we need to see results on the screen, which is non-linear. So a GUI might work on a linear version of the image, and convert on the fly to whatever the screen needs. I donāt know if Gimp works this way. I hope it does.
A GUI for scene-referred images has the extra complication of reducing the infinite range of scene luminescences to the finite range of a screen. It compresses (by an infinite amount!) or just shows a small range of the image. Perhaps Natron etc does this. Gimp doesnāt.
In the OP, @afre mentioned āhigh gamutā, but I think that is a different topic to linearity or scene-referred. I donāt do much cross-gamut work. IM doesnāt expose all the functionality of LCMS, and I suspect LCMS doesnāt have all the tools I would like, for example fine control over squishing wide gamuts into smaller ones.
@afre, apologies, but you got me going with this threadā¦
Thing is, at the scene, the camera has a hard maximum in its saturation point. Everything in front of it has to be mapped somewhere below that. So, the recording you bring home is pegged there, for any post processing. Now, CGI and other synthetic imaging do not have that restriction, but Iām not making movies hereā¦
Yeah, Iām not sure where it comes from, but with the Z6 if you donāt do it, blacks are just not right. Just pull the value from the metadata, apply it, and go on. My D7000 didnāt have one, so it seems to be sensor-dependent. Anyone more-informed on that dynamic should feel free to school meā¦
BTW, as far as black level - Iām a bit rusty, but the camera ADCs are set up with an offset such that if there is 0 input signal (e.g. no photons), you get electrical noise centered around the black level.
I BELIEVE the rationale here is such that internal noise is evenly distributed around the black point, as opposed to getting āmirroredā up into the positive signal region. Iām a bit rusty here though. I sort of recall an analysis (Iām fairly certain it was by JimK whose blog nik cited, may have been Horschack who does a lot of technical analysis on DPReview including deducing the underlying mechanism of Sonyās āstar eaterā and some apparent RAW cooking -Sony A7xxx Posterization and Colored Banding (Part 3): Sony Alpha Full Frame E-mount Talk Forum: Digital Photography Review for example ) that showed that cameras which didnāt have a black level offset tended to have more read noise in those lower few bits due to negative noise values getting mirroredā¦ Again, this is based on partial memory and I could be misremembering it. It was either on the same blog nik cited or in the DPR forums, both of them are not particularly search-friendly.
The idea is that the sensor data is a linear function of the input PLUS a fixed constant offset, which is the black level. So when you subtract the black level, the result is now a linear function of the number of input photons. At least from most discussions Iāve seen, modern digital sensors are very linear with very little deviation right up to the point where they clip.
Having the internal noise sources accurately represented as opposed to being mirrored around 0 also helps if you want to average multiple exposures.
As to linear vs. nonlinear workflows - Obviously some algorithms work better on linear data and some work better on nonlinear data. Interestingly, the analysis JimK did involving the effect of gamma-compressed vs. linear data for resizing and sharpening algorithms in the post cited by nik is in direct conflict with claims Iāve seen made here recently that such operations should NEVER be performed on anything other than linear data. Personally - I almost universally trust Jimās judgement over that of most others for a variety of reasons.
Also, one must keep in mind that in most cases, an evaluation of the final result of a processing workflow is a subjective one - does it look good to the viewer? What may look good or unique may fundamentally NOT be what was present in the original scene.
In my case - some of what Iāve been working on lately is exactly this, specifically to handle display limitations. Right now, almost any camera on the market has dynamic range recording capabilities that are WAY beyond a typical display. (Current HDR displays are a whole other story - Iāve found that the exposure fusion approach which is enormously beneficial for someone viewing on an SDR display is completely unnecessary if you take the same image and feed it to an HDR display without any dynamic range compression. Sadly, weāre years away from having such displays be more common than SDR displays AND having proper content delivery infrastructure for it - in my case, the only way I can get content to my HDR display is to take an image and encode it to a 10-bit HEVC video, which is enormously wasteful.)
Itās no secret that I have a strong preference for the Mertens exposure fusion algorithm. Interestingly, fusing synthesized exposures generated from a high dynamic range RAW image is not exactly how it was initially designed, but even if itās an indirect approach to local tonemapping, it still generally seems to outperform most other local tonemapping approaches, rarely ever exhibiting the flaws (such as haloing) that can arise from local tonemapping. Nearly everything about the algorithm is nonlinear, but in the end - the perceptual results are such that products which use the algorithm as a fundamental building block of their default mode of operation (Such as the Pixel 3) routinely get praised in reviews of their camera functionality.
While your last paragraph is absolutely and irrefutably correct, there is room for a right and wrong here. Some operations can and are being performed on gamma corrected data, but that does not take away the fact that theoretically they actually definitely should not be. There are multiple sources that clearly explain and show that things like blurring and resizing should be done on linear data. Please see here and here for example.
Edit: just to add, as @anon41087856 would probably reiterate too, there is physics and mathematics (en electronics) underlying digital photography. There is little sense to abandon or ignore that, at least, for me.
Edit 2: I see I got some double negatives mixed up. Tried to rephrase.
At least from a quick read of your two links, section D1 of the second one:
āWhether you get āprettierā results when using a gamma=1.0 or a gamma=2.2 RGB color space is an entirely subjective call, and in my opinion, the artist is always right.ā
That doesnāt look like a āclear explanationā that blurring and resizing should always be done on linear data to me.
Edit: Should the ātechnically incorrectā approach be the default approach used? No. But should it be completely forbidden for anyone to ever use that approach? Also, I firmly believe, no.
For clarity then please also quote the first part of section D1 from @Elleās article:
āIn all comparisons above, the colors in the images on the left, edited in the linear gamma version of the sRGB color space, are technically correct. The colors in the images on the right, edited in the regular sRGB color space are technically wrong (ā¦).ā
Thereās always a distinction between technically correct and artistically pleasing.
On the note of processing in linear vs gamma compressed, I really do not have much to add to this discussion, as I am not a developer, except for a specific preference for linear for generating sharpening masks, such as Rawtherapeeās new-ish contrast threshold feature.
Regardless if the actual sharpening is done in linear vs gamma compressed (donāt have an educated opinion on that debate), I think that any sharpening mask should be created in the linear space, since the signal to noise ratio tends to be poorer in the shadows, and lots more fine detail is above the noise floor in the highlights, when observed as a gamma encoded image. This observation is supported by the discussion of photon shot noise linked here. http://www.photonstophotos.net/Emil%20Martinec/noise.html
Given this correlation, then the S/N ratio should be less variant across the tonal range in linear color space, so setting a minimum threshold for sharpening would be able to be set just above the noise level (optimum setting) for a greater tonal range of the image, compared to where I have to do now, where I set the threshold tuned for midtones, and faint detail in the highlights doesnāt get sharpened, and higher nose in the shadows gets sharpened.
Edit INB4 someone responds āJust export two versions with different threshold values and blend with luminance maskā . That would be a waste of time, and raw processing would be more intuitive and quick and I could just tune S/N ratio for one patch with noise and detail in the midtones and assume I wonāt make the highlights look waxy, or the shadows noise be amplified.
Edit INB4 someone responds āJust export two versions with different threshold values and blend with luminance maskā . That would be a waste of time, and raw processing would be more intuitive and quick and I could just tune S/N ratio for one patch with noise and detail in the midtones and assume I wonāt make the highlights look waxy, or the shadows noise be amplified.
Isnāt this really the right answer though? (That, is the raw converter should be able to tune the sharpening threshold to scale with luminosity. I agree doing it by hand is annoying.) The shadows will always be noisier than the highlights just by the virtue of how the physics works out. In a linear space, if a pixel collects N photons, itās going to have SNR proportional to N^{1/2}. Highlights have many stops (say 2^6 photons, at least) more input than the shadows. So they are guaranteed dramatically less SNR.
Further, Iām not sure what doing it in a linear space gets you compared to a gamma-corrected one. The gamma correction is a monotonic map of N and hence of the noise levels, so for any threshold in linear space you should be able to find another one in gamma-corrected space that gives similar results.
This is all heuristic, so maybe Iām missing something? Iād be happy to write out something mathematically more rigorous if you want.
Actually, now that I think of it, you do have a point. With a sufficiently wide ranged gamma correction for the threshold weighting, whether the mask is being created in linear or gamma corrected space becomes irrelevant. My initial gut sense was that linear space would have a more even amount of noise across the tonal spectrum, but after doing some mental thought experiments of noise SNR of āSNR proportional to N^{1/2}ā, echoed on the Martinec article, realized that the range of values in a linear perspective from a small number of photons would be less than that of a large number of photons, even though it is the other way around from a proportional perspective. Therefore, creating an edge mask in a linear space would target the shadows too much. This is confirmed by my experience of gamma settings of 1.0 for RT noise reduction targeting the shadows too much. Then there is variables such as read noise, and it becomes pretty apparent that there is no one size fits all choice, and that a user defined weighting curve is always going to be likely necessary.
I think a good general point is surfacing here, in that preserving the energy relationship has more value in some operations than others. Endeavors that are concerned with color and tonality definitely have a concern with the energy relationship, endeavors that are concerned with edges not so much. Indeed, certain tone mapping may facilitate edge detection in things like convolution kernels, where its the difference in adjacent values that facilitates the transform.
Hint: It is okay to list things in your first post but try to be focused in subsequent posts, separating topics into manageable chunks. 
- FWIW!
