Highlight reconstruction darktable vs. rawtherapee

Iain · July 24, 2022, 1:41am

Here is the information about the highlight reconstruction algorithm that is in development for Darktable

github.com/darktable-org/darktable

Segmentation based highlights recovery (V4)

darktable-org:master ← jenshannoschwalm:gl_recovery_v4

opened 05:12AM - 20 Jul 22 UTC

jenshannoschwalm

+1367 -36

This highlights recovery algorithms is based on ideas from Iain from the gmic te…am and has been developed in collaboration with Iain and garagecoder (also from gmic). It only works for bayer sensors and handles both spatial highlights and larger clipped areas equally. No other external libraries are used, the code has been tuned for performance using omp, no OpenCL codepath. The user interface has two sliders (both with a knob to visualize the effect for user feedback), the highlights module version has **not** changed so you can try out this pr without hassle. Personally i have been using the new algorithm for months now, results (for me) are better than what the other algos do in most cases, performance (on my 5yrs 8core intel) is kept well under control taking less than 1 sec on 45Mpix images in all cases. **How does the algo work** We understand the bayer data as superpixels, each having one red, one blue and two green photosites, we analyse all data on the channels independently so resulting in 4 color-**planes**. In all 4 color planes we look for isolated areas being clipped (segments), these segments also include the unclipped photosites at the borders. To combine small segments we use a morphological closing operation, the radius of that UI op can be chosen interactively between 0 and 8. Inside these segments (well - in fact precisely at unclipped locations at the borders) we look for a **candidate** to represent the value we take for restoration. Choosing the **best** candidate is done via a weighing function based on a) the local standard deviation in a 5x5 area as we want a candidate from a smooth part, b) the median value of unclipped positions also in a 5x5 area as candidates from a dark border section are often bad and c) a weighing threshold presented in the user interface as **candidates**. After finding the best candidate for every segment in every plane we inpaint data on every clipped photosite. The core principle is to inpaint pseudo-chromacity, calculated by subtracting opponent channel means rather than luminance. (For better stability we calculate data in cube roots for an estimate btw.) The inpainted data are not in the normalized 0->1 range any more, so in dt speech or as AP would probably say are scene-referred. **Problems and possibly how to treat** 1. As you might imagine the segmentation is based on the clipping threshold. Thus also the candidate selection and inpainting is based on this. So you might have to play with the clip value for best results. (there is some code increasing border stability but still needs UI feeding) 2. Due to dt's region-of-interest concept the inpainted colors might be vastly wrong while zooming in. 3. As we don't do any AI on content (i tested an additional felzenzwalb segmentation btw) connecting segments via the morphological combine function might not work good enough, if you find such a problem you will have to find other ways to handle highlights. 4. Segments surrounded by 'wide' dark borders (or in other words can't be connected via the morphological combining) always have to rely on some basic inpainting without a good candidate. This is mostly "better than nothing" but in some cases it simply doesn't work well. You should try to tune the candidates slider here. A 'classic' situation would be bright lights like color lamps in a mostly very dark image. 5. Images with difficult lighting situations (see 'bar and bottles' in the testset) are heavy stuff but the other algos also struggle badly here. 6. The segmentation analyses the whole region-of-interest so tiling does not work in this algorithm. The memory footprint of the algo thus might be too high for low-mem systems (<8GB) **Slider knobs** Both give false-color representations in the darkroom display 1. combine: basically shows the segment borders. As these grow with combining segments you get an idea how close segments are connected. 2. candidates: highlighted are segments without a 'good candidate'. **Footnotes:** 1. Iain idea can be seen in https://discuss.pixls.us/t/highlight-recovery-teaser/17670 or in a late video here https://discuss.pixls.us/t/next-live-meeting-saturday-16-july-12-00-utc/31507 2. A testset for severely highlights impaired images has been presented at https://discuss.pixls.us/t/highlight-recovery-test-set/28801 3. The here used segmentation algorithm is a novel algo based on a standard floodfill extended to a) fill in segment identifiers, b) mark border locations and c) keep information of the bounding rectangle of a segment. I also tried out a classic felzenzwalb but the chosen algo was safest. Please note that Iain used a watershed algo in his prototype code. 4. The morphological operations have not been implemented in a 'general' way but are restricted to a radius of 8 for performance reasons. 5. The 'reference plane' in Iain prototype used to be the minimum of all planes, as garagecoder suggested we now use a color coeff based weighed average reference leading to much better color cast control. The cost in quality is a subtle loss of details. To reduce chroma noise the values taken from individual channels are slightly gaussian blurred. 6. Missing from Iain original idea is the 'recovery' in clipped areas, that might be added in the next version. (That part of the code is surprisingly tricky if we want it to handle well all real-world situations. Existing preliminary solutions are based on distance and border gradients.)

If you are familiar with testing development builds, please give it a go.

Terry · July 24, 2022, 2:08am

I have to agree that I wanted to try masking and using multiple instances of highlight reconstruction. I presume there are logical reasons this is not an option.

Terry · July 24, 2022, 2:11am

Would there be value in offering two options for highlight reconstruction where one is before and the other after demosaicing? Could the code from RT be adapted to DT for this purpose?

123sg · July 24, 2022, 2:14am

I presume there’s no windows build yet? Maybe @priort’s next weekly(?) version will have it? Looking forward to trying it out

priort · July 24, 2022, 2:40am

Ha you are fishing…maybe have a go at getting set up to do WIndows builds… not too hard …

Build it from here until it gets merged…

If I have some time tomorrow I will build and post a version

123sg · July 24, 2022, 3:32am

Hi Todd, yes I was hoping you might bite…
I should look into building! Not sure when I will though
Don’t hurry to build a version on my account - but it would be great

priort · July 24, 2022, 3:39am

123sg · July 24, 2022, 4:20am

Wow!!! that was quick now I want to try it instead of doing the work I was going to do… decisions…

Thanks Todd!

priort · July 24, 2022, 4:21am

Quick attempt with new HLR mode…

Second one with a bit more color…

You will need a build with this mode to open this…
2014-05-30_19-47-01_01.cr2.xmp (10.9 KB)

123sg · July 24, 2022, 4:25am

That looks good… one of the best DT versions yet IMO. (viewed on my laptop again - which is calibrated but has only 68% sRGB…colors sometimes look a bit off)
Edit: I might actually say the best yet.
Edit no2: At this point I think it’s a toss up between this one and one of @kofa’s using laplacians HR

123sg · July 24, 2022, 4:31am

Obviously I can’t make my mind up

priort · July 24, 2022, 4:36am

Mine are a bit crushed…I used the tone eq to raise the foreground so tweaking that would likely improve that and some masking to protect or enhance some aspects… also I just pushed around the sliders on the new mode I really have not control over it or idea what I am doing

hannoschwalm · July 24, 2022, 7:08am

I didn’t try to do a “best edit” at all. A few comments though

Unfortunately the “whitepoint” is not exposed in a proper exif tag in this image, instead canon chose to use a makernote tag dt does not use so far. You can read that for example like

[hanno@fedora recovery]$ exiftool 2014-05-30_19-47-01.CR2 | grep White
White Balance                   : Auto
White Balance Red               : 0
White Balance Blue              : 0
Normal White Level              : 11767
Specular White Level            : 12279

and you should set the white point before doing fancy stuff. This will aide any hl handling algorithm (and more btw).

You can now check much better the blown out sections, either by the “raw overexposed” button or the button in highlights module beside clipping threshold.
To check for effect of any hl algorithm or compare i would switch off modules like filmic, color balance rgb, and reduce exposure so everything if easy to be evaluated. Your image will loke like

Bildschirmfoto vom 2022-07-24 08-54-591379×608 871 KB

, the xmp is
2014-05-30_19-47-01.CR2.xmp (6.1 KB)
Now try hl recovery or other algos and see how the colour cast is handled or details from unblown channels are used to keep details.

You may of course do how you like but this would be the technical way and would be the basis on all following modules. Masking stuff or whatever method you choose would always like “best available” in the input stage. Plus: colours are “calculated” in the demosicer and all algos used in that module would like best input data too.

Popanz · July 24, 2022, 7:32am

Thanx for all the input. Judging from the shown results, the new tool look very promising. Will have a look on my desktop how well it works the next days. Is there a snap version for unstable, or do I have to build it on my own? And special thanx are going to the dt developers, for making a brilliant piece of software better and better!

jorismak · July 24, 2022, 10:44am

I merged the new highlight recovery mode into R-Darktable, and wanted to show what it did by setting white-level and just enabling it… @hannoschwalm beat me to it by showing his own work .

I get a bit of red ‘hints’ around the brightest clipped area on the right, but this might be fixed by twiddling with the sliders (which I didn’t). Filmic’s ‘maxrgb’ default mode will bring that color more to the foreground, so I choose ‘luminance Y’. A bit of extra local contrast to the bright sky (not the dark clouds), a bit of diffuse-dehazing on the mountain in the background, and other regular ‘fixers’… not saying you have to like the edit, but it shows what can be done right from Darktable after @hannoschwalm’s work is available.

2014-05-30_19-47-01_01.cr2.xmp (17.9 KB)

Another - time consuming - trick, I tried using Rawtherapee as a preprocessor. Opened the file, cleared everything to ‘neutral’. Switched demosaicing to RCD to match Darktable’s defaults, enabled chromatic fix in demosaicing, switch color profile to Rec2020, and export to Rec2020 16bit tif. Enable color propagation, lower exposure so nothing is clipped (but touch nothing else).

Load this tif into Darktable, raise exposure where you want it (to expose the foreground properly), enable filmic and other tools as you wish, the highlights are now already recovered in Rawtherapee so DT doesn’t has to deal with them. Things like denoise-profiled and lens-corrections pick up the original metadata so that appears to be working. Only white-balance changes with ‘color calibration’ is a bit of a PITA, but there are other ways to correct white balance if you need.

priort · July 24, 2022, 4:33pm

I see you set WP value using normal…just curious what is specular threshold meant to be used for?? I also thought I saw a linear sensor threshold at 10000. I wonder if that is some of the OEM secret sauce if they apply some non-linear functions to the channels at high values to manage clipping?? Really I have no idea but I guess as you get further from 10000 then linear wb corrections might create overshoots in some channels?? I really have absolutely no idea just curious…

priort · July 24, 2022, 4:35pm

For your second approach your export might not have a linear gamma but you could make one in RT…I haven’t used it for a bit so it might have one by default now

hannoschwalm · July 24, 2022, 5:24pm

I don’t know. It’s not documented. Canon will know for sure…

Peter · July 24, 2022, 8:13pm

12279-11767=512. Black point?

jorismak · July 24, 2022, 9:51pm

It’s not, it’s gamma-corrected Rec2020… but it’s neatly tagged so Darktable loads the file just fine and puts it in a linear rec2020 working space again.

Yes, this means effectively RT applies some sort of gamma curve and DT tries to undo it again, but I’m not that bothered by it. If there was a ‘Linear rec2020’ in the RT profile box, I would’ve picked it. But this is fine as an example (and more).

Maybe having worked with film negative scans, where each channel gets gamma-corrected and inverted and multiplied to hell and back to still get a nice looking image at the end… maybe because of the grain…, but I’m not bothered by ‘mathematical inaccuracy’ of doing ‘pow 0.55556’ and then a ‘pow 1.18’ again :). In 16bit precision, it’s close enough to not be noticed.