Highlight reconstruction darktable vs. rawtherapee

jorismak · July 25, 2022, 10:09pm

It’s been stated a few times in the thread that the white-level for this file is wrong in DT, making the highlight-clipping-indicator and the highlight-recovery modules not really work. Set the correct white-level, and you’ll notice what’s clipped and not.

I’ve also posted screenshots to RT seeing the same magenta cast and clipped raw, and FastRawViewer (libraw / rawdigger) to show what is clipped and not.
This file clearly needs some reconstruction to get the sky to show OK.

edit: @hannoschwalm beat me to it again (Discuss not showing new replies for me only after a reload, so I don’t see replies from others before I post myself )

emmel · July 25, 2022, 10:25pm

Dear Joris, Thanks for yr reply. I must admit that I read the thread not thoroughly enough And I didn’t really understood the consequence of issue with the whitepoint.

123sg · July 25, 2022, 10:27pm

It should be an 4th (EDIT mistake should be 5th) option in the existing highlight reconstruction module.

emmel · July 25, 2022, 10:36pm

OK, I see. You refer to the ‘guided laplacians’ option. But wasn’t that already in DT 3.9?

123sg · July 25, 2022, 10:40pm

Sorry, I’m not on my pc so couldn’t check, I should have said, 5th option It’s simply called highlight reconstruction (or similar)

jorismak · July 26, 2022, 7:34am

It wasn’t meant to be pointing fingers or anything! Just checking that the information we started with is still true , or that something new is discovered

emmel · July 26, 2022, 9:16am

I

Dear Steven, I managed to install this version of DT (407f53c). I did : “git clone --recurse-submodules --depth 1 GitHub - jenshannoschwalm/darktable: darktable is an open source photography workflow application and raw developer” and followed the instructions further down that page on how to install a test version, alongside my ‘normal’ version.

There’s only the 4 options in HLR.
Do I have to do something extra, to have the new module activated in this version?

123sg · July 26, 2022, 9:34am

Oh… it’s certainly not there! I’m sorry not to be of more help, but I have very little technical knowledge when it comes to compiling and so on. I’m a windows user, so I installed the build from @priort and it just worked…
The only thing I can think of to check - is to look on the top left corner of the darktable window, it should say what version it is below the logo. The screenshot below is ‘normal’ DT 4.0, the release version, whereas when I’m running the new build it reads 4.1.0 followed by a string of characters. It might be different on windows, but you could at least maybe check that your system is actually running the new version. I’m sure someone else will have a more useful suggestion!

kmilos · July 26, 2022, 9:36am

After cloning, you have to check out the relevant feature branch, not the master one.

emmel · July 26, 2022, 9:52am

I’m not sure what you mean by ‘check out’. But I’ll start a new Topic, because this has nothing to do anymore with HLR. It’s more related to an new Linux-, former Mac user taking his first steps in the world of github, cloning, compiling, building, … and I don’t really understand it all. Yet. grrrr.

RawConvert · July 26, 2022, 10:02am

@emmel , here’s some info I put together on building dt -

emmel · July 26, 2022, 10:33am

Hi Andrew,
I wanted to start a new topic, and when I typed in a subject, your topic came up as a suggestion. I’m reading it at the moment and it’s just what I need. So, no new topic from me needed.

ltr21 · July 26, 2022, 10:45am

Well, you are right, of course. The sky is heavily clipped, in all three RGB channels somewhere, so clipping tools are necessary. There are different tools, so there are different results, of course - according to individual tastes. I wanted to point out only the presence of the Dynamic Range Compression tool in the RT, which can be used here in a fast and effective way, although not the only one. The DRC in the RT compresses the data so that it fits the destination range using R.Fattal and Co. algorithm. It cures here almost completely the magenta problem, due to chromatic aberration. This “almost” and so on, would need some extra work, but this is beyond the scope of our discussion devoted mainly to the DT, which is out of my experience.

jorismak · July 26, 2022, 3:23pm

The new highlight recovery code is not yet in the official Darktable sourcecode. It’s ‘proposed, but not yet accepted’ at this stage, so to speak.

On the Darktable github, there is a section with ‘pull request’, proposals for code to be ‘pulled into Darktable’. This is the one for the highlight recovery:

github.com/darktable-org/darktable

Segmentation based highlights recovery (V4)

darktable-org:master ← jenshannoschwalm:gl_recovery_v4

opened 05:12AM - 20 Jul 22 UTC

jenshannoschwalm

+1378 -39

This highlights recovery algorithms is based on ideas from Iain from the gmic te…am and has been developed in collaboration with Iain and garagecoder (also from gmic). It only works for bayer sensors and handles both spatial highlights and larger clipped areas equally. No other external libraries are used, the code has been tuned for performance using omp, no OpenCL codepath. The user interface has two sliders (both with a knob to visualize the effect for user feedback), the highlights module version has **not** changed so you can try out this pr without hassle. Personally i have been using the new algorithm for months now, results (for me) are better than what the other algos do in most cases, performance (on my 5yrs 8core intel) is kept well under control taking less than 1 sec on 45Mpix images in all cases. **How does the algo work** We understand the bayer data as superpixels, each having one red, one blue and two green photosites, we analyse all data on the channels independently so resulting in 4 color-**planes**. In all 4 color planes we look for isolated areas being clipped (segments), these segments also include the unclipped photosites at the borders. To combine small segments we use a morphological closing operation, the radius of that UI op can be chosen interactively between 0 and 8. Inside these segments (well - in fact precisely at unclipped locations at the borders) we look for a **candidate** to represent the value we take for restoration. Choosing the **best** candidate is done via a weighing function based on a) the local standard deviation in a 5x5 area as we want a candidate from a smooth part, b) the median value of unclipped positions also in a 5x5 area as candidates from a dark border section are often bad and c) a weighing threshold presented in the user interface as **candidates**. After finding the best candidate for every segment in every plane we inpaint data on every clipped photosite. The core principle is to inpaint pseudo-chromacity, calculated by subtracting opponent channel means rather than luminance. (For better stability we calculate data in cube roots for an estimate btw.) The inpainted data are not in the normalized 0->1 range any more, so in dt speech or as AP would probably say are scene-referred. **Problems and possibly how to treat** 1. As you might imagine the segmentation is based on the clipping threshold. Thus also the candidate selection and inpainting is based on this. So you might have to play with the clip value for best results. (there is some code increasing border stability but still needs UI feeding) 2. Due to dt's region-of-interest concept the inpainted colors might be vastly wrong while zooming in. 3. As we don't do any AI on content (i tested an additional felzenzwalb segmentation btw) connecting segments via the morphological combine function might not work good enough, if you find such a problem you will have to find other ways to handle highlights. 4. Segments surrounded by 'wide' dark borders (or in other words can't be connected via the morphological combining) always have to rely on some basic inpainting without a good candidate. This is mostly "better than nothing" but in some cases it simply doesn't work well. You should try to tune the candidates slider here. A 'classic' situation would be bright lights like color lamps in a mostly very dark image. 5. Images with difficult lighting situations (see 'bar and bottles' in the testset) are heavy stuff but the other algos also struggle badly here. 6. The segmentation analyses the whole region-of-interest so tiling does not work in this algorithm. The memory footprint of the algo thus might be too high for low-mem systems (<8GB) **Slider knobs** Both give false-color representations in the darkroom display 1. combine: basically shows the segment borders. As these grow with combining segments you get an idea how close segments are connected. 2. candidates: highlighted are segments without a 'good candidate'. **Footnotes:** 1. Iain idea can be seen in https://discuss.pixls.us/t/highlight-recovery-teaser/17670 or in a late video here https://discuss.pixls.us/t/next-live-meeting-saturday-16-july-12-00-utc/31507 2. A testset for severely highlights impaired images has been presented at https://discuss.pixls.us/t/highlight-recovery-test-set/28801 3. The here used segmentation algorithm is a novel algo based on a standard floodfill extended to a) fill in segment identifiers, b) mark border locations and c) keep information of the bounding rectangle of a segment. I also tried out a classic felzenzwalb but the chosen algo was safest. Please note that Iain used a watershed algo in his prototype code. 4. The morphological operations have not been implemented in a 'general' way but are restricted to a radius of 8 for performance reasons. 5. The 'reference plane' in Iain prototype used to be the minimum of all planes, as garagecoder suggested we now use a color coeff based weighed average reference leading to much better color cast control. The cost in quality is a subtle loss of details. To reduce chroma noise the values taken from individual channels are slightly gaussian blurred. 6. Missing from Iain original idea is the 'recovery' in clipped areas, that might be added in the next version. (That part of the code is surprisingly tricky if we want it to handle well all real-world situations. Existing preliminary solutions are based on distance and border gradients.)

On the top of that page:

So the code is currently in @hannoschwalm’s github repository.

So the easiest way to explain (there are other ways), is to check out his github:

Just as you did with git clone. I would leave out the --depth 1 for now. That means you get only the ‘latest’ code, without it, you get ‘everything’.

Then check out the correct branch (‘switch branch’ to the new code), and then update the submodules.

git clone https://github.com/jenshannoschwalm/darktable.git
cd darktable
git checkout gl_recovery_v4
git submodule init
git submodule update

Maybe this can be combined in a one-liner as

git clone -b gl_recovery_v4 --recurse-submodules https://github.com/jenshannoschwalm/darktable.git

Then continue your build as usual. Since you managed to build v4 before, this should work out just fine.

priort · July 27, 2022, 12:58am

This is all you need to do and then build…

git switch -c new_branch
git pull https://github.com/<user>/<forked-repo>.git <branch>

No cloning needed just add the branch…it will track changes …you can remove it easily later if you want…

Terry · July 27, 2022, 5:17am

I hammer home the point about not clipping highlights with the ETTR suggestion. Also if you push the ETTR too much and don’t clip you can still get very desaturated highlights. That could be a problem for some images. I agree, that I would much rather deal with noise than reconstruct a clipped highlight. However, sometimes the dynamic range of the lighting means even the best of us will get unavoidable clipping.

Terry · July 27, 2022, 5:30am

@kofa I like that you describe LR exposure merge as an in camera solution. Because I have a perpetual LR 6 licence I defer to it for my exposure blending. I like that it handles RAW files and even handheld shots. What really amazes me is that if there are tourists (bloody photographer’s curse) walking around the scene it can handle that problem as well. LR works most of the time, but even it can have a hissy fit sometimes.

I have played with Hugin a little but not invested enough time into it yet because of LR. I have tried the HDR function in DT with tripod aligned shots and have been underwhelmed by its performance. I am not sure if DT version has improved in that module.

kofa · July 27, 2022, 5:57am

I don’t, but I think you do.

I rarely do HDR merges; I stitch (hand-held) panoramas. For scenes with a high dynamic range, I underexpose to protect the highlights, and deal with the noise. I don’t think the noise is a serious problem, not even from my 10-year-old LX7 (1/1.7") compact, which is much like the G16 you used in your example.

Terry · July 27, 2022, 6:02am

I feel most of the HDRs I see on the internet are horrible, unnatural and overcooked. If the scene permits and I have a tripod I like to do auto exposure bracketing, process the separate images in DT and output as 16 bit tiffs which I then open as layers in Gimp and blend with layer masks. A gradient mask will often work nicely for landscapes and replaces the need for a gradient lens filter which can be a pain to use anyway.

emmel · July 27, 2022, 6:50am

Perefect! And the “one liner” worked. Thanks for you support!