Highlight reconstruction darktable vs. rawtherapee

Iain · July 24, 2022, 1:41am

Here is the information about the highlight reconstruction algorithm that is in development for Darktable

github.com/darktable-org/darktable

Segmentation based highlights recovery (V4)

darktable-org:master ← jenshannoschwalm:gl_recovery_v4

opened 05:12AM - 20 Jul 22 UTC

jenshannoschwalm

+1367 -36

This highlights recovery algorithms is based on ideas from Iain from the gmic te…am and has been developed in collaboration with Iain and garagecoder (also from gmic). It only works for bayer sensors and handles both spatial highlights and larger clipped areas equally. No other external libraries are used, the code has been tuned for performance using omp, no OpenCL codepath. The user interface has two sliders (both with a knob to visualize the effect for user feedback), the highlights module version has **not** changed so you can try out this pr without hassle. Personally i have been using the new algorithm for months now, results (for me) are better than what the other algos do in most cases, performance (on my 5yrs 8core intel) is kept well under control taking less than 1 sec on 45Mpix images in all cases. **How does the algo work** We understand the bayer data as superpixels, each having one red, one blue and two green photosites, we analyse all data on the channels independently so resulting in 4 color-**planes**. In all 4 color planes we look for isolated areas being clipped (segments), these segments also include the unclipped photosites at the borders. To combine small segments we use a morphological closing operation, the radius of that UI op can be chosen interactively between 0 and 8. Inside these segments (well - in fact precisely at unclipped locations at the borders) we look for a **candidate** to represent the value we take for restoration. Choosing the **best** candidate is done via a weighing function based on a) the local standard deviation in a 5x5 area as we want a candidate from a smooth part, b) the median value of unclipped positions also in a 5x5 area as candidates from a dark border section are often bad and c) a weighing threshold presented in the user interface as **candidates**. After finding the best candidate for every segment in every plane we inpaint data on every clipped photosite. The core principle is to inpaint pseudo-chromacity, calculated by subtracting opponent channel means rather than luminance. (For better stability we calculate data in cube roots for an estimate btw.) The inpainted data are not in the normalized 0->1 range any more, so in dt speech or as AP would probably say are scene-referred. **Problems and possibly how to treat** 1. As you might imagine the segmentation is based on the clipping threshold. Thus also the candidate selection and inpainting is based on this. So you might have to play with the clip value for best results. (there is some code increasing border stability but still needs UI feeding) 2. Due to dt's region-of-interest concept the inpainted colors might be vastly wrong while zooming in. 3. As we don't do any AI on content (i tested an additional felzenzwalb segmentation btw) connecting segments via the morphological combine function might not work good enough, if you find such a problem you will have to find other ways to handle highlights. 4. Segments surrounded by 'wide' dark borders (or in other words can't be connected via the morphological combining) always have to rely on some basic inpainting without a good candidate. This is mostly "better than nothing" but in some cases it simply doesn't work well. You should try to tune the candidates slider here. A 'classic' situation would be bright lights like color lamps in a mostly very dark image. 5. Images with difficult lighting situations (see 'bar and bottles' in the testset) are heavy stuff but the other algos also struggle badly here. 6. The segmentation analyses the whole region-of-interest so tiling does not work in this algorithm. The memory footprint of the algo thus might be too high for low-mem systems (<8GB) **Slider knobs** Both give false-color representations in the darkroom display 1. combine: basically shows the segment borders. As these grow with combining segments you get an idea how close segments are connected. 2. candidates: highlighted are segments without a 'good candidate'. **Footnotes:** 1. Iain idea can be seen in https://discuss.pixls.us/t/highlight-recovery-teaser/17670 or in a late video here https://discuss.pixls.us/t/next-live-meeting-saturday-16-july-12-00-utc/31507 2. A testset for severely highlights impaired images has been presented at https://discuss.pixls.us/t/highlight-recovery-test-set/28801 3. The here used segmentation algorithm is a novel algo based on a standard floodfill extended to a) fill in segment identifiers, b) mark border locations and c) keep information of the bounding rectangle of a segment. I also tried out a classic felzenzwalb but the chosen algo was safest. Please note that Iain used a watershed algo in his prototype code. 4. The morphological operations have not been implemented in a 'general' way but are restricted to a radius of 8 for performance reasons. 5. The 'reference plane' in Iain prototype used to be the minimum of all planes, as garagecoder suggested we now use a color coeff based weighed average reference leading to much better color cast control. The cost in quality is a subtle loss of details. To reduce chroma noise the values taken from individual channels are slightly gaussian blurred. 6. Missing from Iain original idea is the 'recovery' in clipped areas, that might be added in the next version. (That part of the code is surprisingly tricky if we want it to handle well all real-world situations. Existing preliminary solutions are based on distance and border gradients.)

If you are familiar with testing development builds, please give it a go.