Due to annoying bugs in LuminanceHDR I am building my own HDR compositor, and for this I need to find out how to transform parts of a brighter image so that they can replace parts of a darker image. For this, I transform the input image data to a linear color space so that I can apply a constant factor to the samples which should be independent of channel and value.
To get my own view on the problem, I compare two (perfectly aligned but differently exposed) input images: I iterate over different small areas of the images, transform their pixels to a linear color space (RGB without gamma or XYZ), take the area average for each channel and then I compute the quotient of the values for each area of the image for each channel. Example: for two input images with light values 19 and 20, I would expect sampleAvg(area(darkerImage))/sampleAvg(area(lighterImage)) to be always 0.5, except for black or overexposed areas. The brighter image has the double exposure time than the dark one.
The problem is that the factor depends on the relative brightness of the analyzed area (of the brighter image): For very bright areas, the factor comes close to 1, while it is below 0.5 for darker parts. I get a good exponential approximation for the quotient:
The x axis shows the Y channel (normalized 0 to 1) of an sRGB image (transformed to XYZ), while the y axis shows the average factor needed to transform a Y sample from the 1EV brighter image to the darker image.
I get the same result with input images in linear ACES color space and without color space transformation:
I even changed the camera input profile (DCP) to include a flat tone curve:
I would instead expect the graph to be a horizontal line independent of x: y = 0.5
Why is the quotient dependent of the value? I performed analysis in linear color spaces, I used the DNG’s default matrix and tone curve in Lightroom 5.6 to convert to TIFF format.
Perhaps we share the same problem. I encounter non-linearity with my slide scanner data, i.e. the data as they come from the scanner without gamma-correction and without any transformation. If I plot the density of a target with known density values against the measured data, the data approach a constant value when the density approaches large values.
I interpret this as a constant value added to the data from the detector, which is common practice for CCD cameras as I know them (in astronomy). If this bias is not subtracted before any arithmetic operation, the detector exhibits a non-linear behaviour.
Apparently the values are not linear. By “linear”, I mean values should be proportional to the light energy received, so opening up a stop will double values.
What camera did you use? Have you tested the camera (EDIT and the workflow) for linearity? See Linear camera raw.
I assume you use raw values, debayered, perhaps white balanced but with no auto-white-balance, and no auto-brighten. Is that correct? How are you reading the raw image?
If you are using OOC JPEGs, that would explain the problem. There is no chance of getting linear values from OOC JPEGs.
You say:
The x axis shows the Y channel (normalized 0 to 1) of an sRGB image (transformed to XYZ), …
Why are you going via sRGB? This creates potential for error.
You have no idea what’s going on inside of Lightroom. There’s no way to be sure that LR is keeping everything linear from capture to output.
Try using linear output from dcraw (after black subtraction) and then see if things are linear.
In my experience making an HDR compositor for the old CLI Filmulator, raw files very much are linear. Why not just use LibRaw to read your raw files directly?
Thank you for your answers so far! I try to reply to them as soon as possible. The images were made with a Samsung Galaxy S7 back camera.
Some additions to my research:
RawTherapee export from DNG to reference TIFF without any color profile or gamma (with “fast” demosaicing):
Now the graphs (for two darker images with -1 and -2 EV) are more linear but still depend on the values.
Lightroom does not accept profiles with identity matrixes for forward and color matrix, so there is no way to get a reference image from LR.
I opened the DNGs in RawDigger, and the sample values correlate perfectly with the expected EVs. The origin of the problem must be located in the processing pipeline.
Questions:
Could the DCP matrixes with negative factors be responsible for this result? My intuition tells me that only factors with the same sign should result in a constant factor … Example for Samsung Galaxy S7:
Next I will compare with other cameras and provide you some RAW files.
PS - to reply to @CarVac’s question: My compositor will accept rendered files and not RAW files because it will call image_align_stack before starting which needs TIFFs for processing, so my approach to regress an exponential function will be an acceptable solution. Additionally the input images should include the basic corrections for lens and denoising. I will try dcraw next.
I just want to understand why the processing pipeline makes the sample non-linear.
If you select ‘no color profile’ on output, RT applies a sRGB gamma.
If you want linear output, use a profile with a “linear gamma” (apologies for the sloppy terminology)
Could the DCP matrixes with negative factors be responsible for this result?
No. Why not? A 3x3 matrix calculates new R’G’B’ values thus:
R’ = aR + bG + cB
G’ = dR + eG + fB
B’ = gR + hG + i*B
If we multiply input RGB values by a factor such as 2, then output values R’G’B’ will be multipled by the same factor. This is true for all values of abcdefghi.
My guess it that Lightroom is doing some other processing, more than a simple power (“gmma”) curve. Perhaps something adaptive.
LR is definitely not linear, not even its default neutral settings. You could try older process versions to see if you could find one that is linear. However, with antiquated versions come antiquated processing algorithms. It might be better to use FLOSS apps instead where you know exactly what you are getting into.
In LR, I cannot switch off WB. It takes by default the value the camera saved to the DNG, equal for all input images. All irrelevant settings switched off, no EV or tonecurve changes.
Even more strict for the RT reference export: all settings reset, for safety reset the working space gamma to 1.0
I have exported to other - linear - formats with higher gamut. As you see in my analysis, the results are the same.
I agree with @snibgo : Take a look at the curve with encoding gamma 1/2.2:
For linear x value 0.2 the y value is already 0.5. This corresponds to a relative exposure value of < -2. This gamma may be perfect for the fitting viewing conditions, but it may be set lower to let the human eye focus on a lower relative EV (e. g. as RawDigger proposes: -3 EV) as the normal brightness. An internal gamma < 1/2.2 may be the reason for the curves.
I already set the gamma to 1.0 (to be sure) and export the reference TIFF.
No conspiracy here. My compositor will accept rendered (but not tonemapped) images as they are and will find the appropriate curves to make a graded mapping from input to output image. It does not need to know any exposure data.
I never believed that a tool that fits perfect to my demands already exists: GitHub - jcelaya/hdrmerge: HDR exposure merging by Javier Celaya-Alastrue – and it goes beyond it: It produces a compact DNG, corrects exposure if the darkest image is too dark, adds the correct metadata and is very fast.
Thank you @paperdigits for pointing me to it and all the other guys for your constructive comments!
Yes, it is great. The downside is that it is missing some useful features. It could borrow them from existing raw processors; however, no one is motivated to do it.
Think of all the things that could be done before alignment or demosaicing.
Depends on the raw files. Alignment is only up to a few pixels, so you need a very still sensor. Masking could be more sophisticated and have more tools.
Not again. It doesn’t change the original pattern. No demosaicing at all.
You mean the usual optimzations like denoising and lens correction? This is what keeps me to pursue my own algorithm. Some optimizations beforehand will surely give better results.
I could not see any problems yet, only if the images were not shot in burst mode.
And isn’t the blurred hard mask a problem? This is probably not the best approach. The blurred edges seem to be the only parts where the different layers are mixed. I would instead expect an approach applied pixel by pixel: include a grading in the bright areas of the input images where the brightest parts are not included and the slighly darker parts are a mix of both images down to a certain threshold. The brightness grading could be from 0.75 to 0.95.
This can only be true for the one image that is not aligned, all the others will be aligned, and so the mosaic somehow must be mapped to the first image. After the transformation, a red sample could be over a green sample. A certain demosaicing is needed to mix in samples of the correct channel
Not necessarily. One could restrict the alignment to be horizontal and/or vertical translation (crop) only by a multiple of 2 pixels, thus preserving the Bayer pattern matching. Not ideal, but demosaicing and then subsampling again back to raw suffers from a different set of problems due to multiple interpolations.
One alternative to produce a merged raw without demosiaicing is what Google does for HDR+: mix on a block by block basis in the frequency domain. (Also implemented here.)
I thought HDRMerge would do something like image_align_stack, which is necessary for handheld pictures according to my experience: In LuminanceHDR you can optionally align the images pixel-wise by hand, and using this approach I always had ghost edges somewhere in the image. More transformations are necessary to get satisfying results.
But it seems you are right: HDRMerge User Manual v0.6 . The only reason why my results are sharp is probably that I captured my latest images with very fast bracketing.
Another observation: The resulting HDR has a little bit more noise in the dark parts than the brightest input image.
Thank you for the other link. Some idea like “Raised Cosine Window” was in mind …
