Sounds really interesting 
Thanks for the answer!
nice, thanks for posting all these command lines here again.
you are using 12 curves, right? i looked at these images for too long, i can see diffusion/blur artifacts at the mountain ridge to the right when flipping between the accentuated coarse and previous images.
i was excited about the possibility to use the coarse scales as a kind of dynamic range compression here but never succeeded to make that look good. do you have good experience with that?
instead iām using the extremes of the curve to affect blacks and highlights separately. with mixed results. it only affects isolated blacks/highlights, it needs the opponent around it
also i keep changing my mind which shape exactly should the curves have for particular images. sometimes iām thinking maybe a couple of arbitrary curves and a way to merge it would be better (aka exposure fusionā¦).
No, it is definitely an issue and happens in both all vs coarse. I notice it in the indoors photo as well, though it is thankfully much less apparent.
That said, it is a big improvement from @Carmelo_DrRawās previous attempts. 
Some sort of curve smoothing in order?
I realize this is pretty heavy thread necromancy, but this comment has always stuck in my head (were you trying to lead me somewhere mathematically? 
), and Iāve finally had some time to start poking at it.
The local laplacian paper provides one particular definition of a remapping function - but it states that it just requires a monotonic function. Most global tonemapping curves (and IMO any sane one) are monotonic.
What would be the mathematical result if you took the limit as num_exposures with Mertens reaches infinity with the hardmask set? That seems to me like it would begin to look almost identical to the Paris et al 2011 approach, where each entry in the Laplacian pyramid is taken from what is effectively an āoptimally exposedā input.
Thereās also https://people.csail.mit.edu/hasinoff/pubs/AubryEtAl14-lapfilters.pdf - which instead of calculating the āoptimalā remapping, calculates in discrete steps, and interpolates values that fall in between discrete steps. Again - doesnāt this mathematically look VERY similar to Mertens fusion with many (but not infinite) exposures and hardmasks?
Iām starting to experiment with GitHub - LSTM-Kirigaya/localaplace: implement of Local Laplace Filter algorithm (slow as hell but easy to modify) to see what happens if you use @jandren 's sigmoidal tonemapping as the remapping function. (Iāve gotten it working with the āoriginalā approach first, it needed some fixes to not choke on logarithmic inputā¦), and then try to fix up GitHub - KerouichaReda/Fast_LLF_python: implementing the algorithm of fast llf into python to make it less painful to experiment. Iām very curious to see what difference there is (if any) between Mertens and an LLF with the remapping being a ātypicalā global tonemapping function.
Iāll hopefully have sigmoidal kanged into a remapping function later this week, while Iām on vacation, this cruise is quite relaxing/quiet and some of my shots have challenging dynamic range so Iāve been fiddling with LLF on the observation deck.
Edit: Hmmā¦ The Paris 2015 paper indicates that this only applies to single-channel images. This could be problematic as far as optimizing things. Although it looks like what is most important is that the Gaussian pyramid with g coefficients is single-channel - this roughly corresponds to an exposure compensation for that entryās āoptimal source imageā, but individual channels should be OK for the Laplacian pyramid. Or I could be completely barking up the wrong tree here.
5 Likes
Nothing better than long running projects that can simmer in the back of ones head for a long time. Iām intrigued two what the sigmoid curve could add to this!
3 Likes
Yup. So far despite the mathematical similarities, using sigmoid as a replacement for the monotonic functions called out in the paper isnāt working too well despite the significant structural similarities to Mertens fusion with hardmask enabled and many exposures, but the issue may remain between chair and keyboard.
The āFastā version is indeed SIGNIFICANTLY faster, and itās far more memory-efficient than Mertens (which needs to store a Gaussian AND Laplacian pyramid for each exposure, while the āFast LLFā algorithm only needs one Gaussian pyramid, the output Laplacian, and one temporary Laplacian pyramid at a time. Funny thing is that AP criticized Mertens fusion as being slow, but the 2011 LLF filter is actually significantly slower despite the theoretical optimizations. The 2015 version will probably be similar in speed for a similar number of exposures but the memory efficiency benefits are massive if it can be made to work similarly to Mertens fusion.
However the implementations Iām starting from (linked above) have vastly different remapping functions. I decided to try to rework the āslowā version into āfastā - it is working however Iām experiencing severe banding artifacts. Iām not sure if it is due to a mistake in my implementation or a fundamental limitation. Iāll play a bit more for a day or two, post what I have, and then take a break until I return home if I hit a wall.
p.s. thanks for the Python implementation in the tonecurve explorer, it made it REALLY easy to pull in as an alternative remapping function. No hue preservation now, but frankly thatās a ānice to haveā as a finishing touch and not a hard requirement for experimentation.
Edit: Iām pretty sure Iām running into the restrictions that the remapping function r() needs to be Nyquist-sampled. Itās going to take me a while to wrap my head around this since Iām used to spatial and time sampling, but this isā¦ The fourier transform of a luminance function? Either way I definitely need to revisit my math because the assumption that this would be mathematically similar to Mertens fusion with many exposures and hard masks seems to be flawed somewhere, because using traditional tonemapping curves such as sigmoid seems to have little to no dynamic range compression effect.
Edit 2 (heh didnāt even post my first edit) - The remapping functions used in Paris et al are significantly more aggressive than any traditional tonemapping curve, especially since the functions described in Paris et al are commonly applied to log-encoded data. Clearly some of my assumptions in potential parallelisms between Mertens fusion with a hardmask and LLF were incorrect. This is going to take a bit of digging into the math to figure out where I went wrong.
For comparison, this is the comparison of a common LLF remapping function (from the paper, with alpha=0.25, beta=0, sigma=0.8) vs. taking the logarithmic values, linearizing them (2**x), feeding that to sigmoid with display black = 0.1 and display white = 0.34 and contrast=3, taking the logarithm of that, and subtracting log(0.1845) to shift the curve upwards to be centered at 0:
nice experiments! i think i can still see the banding for extreme settings in the productive darktable/vkdt implementations (they use the 2015 approach). while weāre necroing, hereās my old writeup on the topic: local laplacian pyramids: draft blog post
i think my conclusion is the same as yours. the curve has to be bandlimited in some sense, i did not think any deeper about this.
1 Like
Iāve seen banding even at 100 iterationsā¦ mehā¦ But your analysis does confirm my results.
F[x f (x)] ā F[ f ]ā implies to me that taking the Fourier transform of the derivative might make analysis a bit easier than what Iāve been doing so far. Time to install jax on this machine! The āexampleā for the fast version Iāve found did indeed use a different mapping function that I have yet to plot, and now I understand why.
The remap function used in Fast_LLF_python/llf.py at master Ā· Entropy512/Fast_LLF_python Ā· GitHub is still REALLY weird though compared to the original Paris functions:
The derivative of the function sure as hell is bandlimited though:
If Iām reading the darktable source correctly, itās operating on linear data instead of doing a log transform? The original Paris paper strongly hints that operating on log-encoded data is preferable. (Similar to how feeding linear data to Mertens exposure fusion often leads to undesirable results, including the only time Iāve ever seen Mertens halo.)
Interestingly, all of APās frothing about Mertens being slow and wasteful was BS.
Typical Mertens implementation: 3 exposures, leading to:
1 output Laplacian pyramid
3 Gaussian pyramids for weights
3 Laplacian pyramids (1 per exposure)
1 output Gaussian pyramid to calculate normalization at the end
While dtās implementation isnāt very memory efficient from what I remember, it can be implemented similarly to the LL setup where you only need to store one exposureās worth of pyramids at a time
The darktable LL implementation:
1 output Laplacian pyramid
1 input Gaussian pyramid
6 temporary Laplacian pyramids
And for some settings you might need more than 6! (with APās frothing at the mouth that LL could never break and anything that has a possibility of ever breaking is garbageā¦ But the paper literally says that not only must the function be bandlimited but you must sample it frequently enoughā¦)
Time for lunchā¦
Nice experiments indeed, but please refrain from mentioning your good friend AP.
2 Likes
Hope you find some more time for this.
I have some thoughts that has been growing in my mind for some days now.
First is about the curve. Is you want to do ev to ev then itās probably better to just use a normal logistic curve rather than the skewed log logistic from the sigmoid module. Second, is that the weird difference between the two remapping functions. Is it so that the one that is just two bumps in the end I added to a linear function? That would make them quite similar.
Finally, would it make sense and be possible to implement a windowing function in ev space? Like adding a weight to the data from the different exposures that for example prioritizes greyās and ramps down as the compensated exposure grows too large. That would put a nice and smooth limit on how bright/dark data we care about.
i really like this idea. the banding artifacts are quite a nuisance. even for moderate noise, these bands turn into very messy stripes of noise, and i donāt want to pay for more pyramidsā¦
would have to experiment with your thought here a bit though. you donāt usually have the full exposure difference in one step, but only one level of laplace coefficients as you collapse the pyramid. might still work to limit the difference there though. and probably itās the difference between laplace coefficients from different pyramids that needs limiting, not absolute difference to source value.
How does it work when you do the multiple exposures case?
I guess you know what exposure it comes from even tough you are at some pyramid level. The number of exposures could be defined by the window function and a stepping argument.
Like exposure every 1 EV under the bell curve with 2 EV std would need something like 15 exposures if we want to go to 3 sigma.
I honestly havenāt read up how these algorithms work in detail but a window function of some sort feels right as a smooth way of reaching a end to how many exposures that are needed.
Some interesting limits here as well.
inf wide window and inf number of exposures would yield the same result as Paris et al 2011 (if I understand the explanation above correct)
0 width window should be simply the normal tone mapping result.