@agriggio I am compiling your branch atm. In the meanwhile, I found that setting the Tone curve to ‘Luminance’ mode gives an image much more true to the embedded JPEG. Could this simply be due to the fact that the matching itself is done on luminance values?
fuji-histmatching branch, the curve and image look like this (please forgive the Dutch interface):
I would say this is already closer to the JPEG, in particular in the highlights.
in short: no
in length: omg no, it’s probably language barrier, what’s wrong with you that you thought so ?
sorry, you’ve told me above that this is forum and bugs should go on github page … I didn’t mean that bad nor that I wanted teach somebody …
@Thanatomanic that’s not the curve I’m expecting to see though… I forgot to mention you need a recent version of exiftool to see its intended behaviour. (And maybe also try clearing the cache and arp just to be sure)
@sigsegv111 no worries, I think there was a bit of misunderstanding. My tentative fix is done in ART, not in rawtherapee. While it can be certainly ported over, it can’t be immediately copy/pasted, and so it wouldn’t belong in rawtherapee’s GitHub tracker I think
@agriggio With exiftool properly enabled and the cache cleared, I get this in ART:
So there are two curves now, smart idea. I’m inclined to say this performs slightly worse than before unfortunately.
FWIW, I tried some histogram matching outside of RawTherapee by manually doing it in Wolfram Mathematica. I also tried to match on different things: luminosity, mean RGB value and B only (because of the sky). If I plot those curves overlaid on the curve in RawTherapee, I get this. Blue dots = luminosity, orange = mean RGB, green = B channel only.
I would say the algorithm in RT gives very comparable curves. The one matched on the blues gives a slight push in the second half of the histogram, which could be an improvement for the match of this particular image if the tone curve is set to Standard instead of Luminance.
Mathematica actually makes less of a nice curve in the upper regions, but that’s not surprising since the histogram is pretty flat there and I didn’t like optimizing the interpolation process further.
thanks much for your outstanding help … Could you please try also this one (it is also linked above)
this picture has even more fancy curve …
what you have outlined above looks really promising … also I’ve learned about ART which I didn’t know that it exist …
best regards, dan
For this image I see more differences between the curve in RT and my calculated curves. The colors are as before.
fwiw, I’ve tweaked this a bit more, and tested on several images taken with different fuji cameras. I’m happy with the results, so I’ve merged it into master (for ART). Of course, you are totally free to disagree and think it’s not good enough (it is certainly not perfect…), but I think I’ll move on for now.
Thanks Alberto, maybe @sigsegv111 can find what he looks for in ART if the auto-tone curve in Luminance-mode in RT doesn’t suit him.
One question though @agriggio, since I had been looking over your matching code briefly yesterday. I couldn’t quite figure out how I could force the curve to contain more reference points. Simply changing a few hard-coded values for
npoints and a few others, didn’t do much. Can you give me a pointer?
which version? ART or RT? In ART there’s quite a bit of code that tries to make sure that you don’t add too many points, because the more points you have, the higher the risk of overfitting. So the curve is first built, and then simplified as much as possible at the end of
mappingToCurve (there’s probably also some redundancy in the code, relics of my many half-failed attempts to get this as robust as possible ). I don’t remember now what the RT version does, it’s probably quite a bit older if nobody else has touched it recently…
Oh, and I forgot: auto-matched is definitely not mandatory: If someone prefers to have a more deterministic starting point (i.e. a fixed curve for all the images), there’s certainly nothing wrong with that…
thanks for help … I will use ART for now … I am fine with that … I have opened an *issue on github so this problem is already reported …
could you be please so kind and try to explain me (only briefly) what’s the problem in mainline RT that FUJI x-trans RAFs are processed very inaccurately like that ?.. I have to repeat, I’ve never hit this kind of issue with thousands of Nikon NEFs edited in RT since version 4.x over past years …
thanks and regards, dan
the problem is in the auto-matching algorithm, not in RT itself. It seems that some Fuji cameras (X100 models especially, from what I’ve seen) tend to underexpose quite a lot and then compensate this when generating the embedded JPEG. This confuses the auto-matching algorithm, which sometimes generates “bad” curves. If you switch to using a fixed curve instead of auto-matching (I think RT ships with a profile called “standard film curve” which might be suitable), then there is no problem.
Thanks, I’ll see what I can find and possibly improve upon.
In addition to this, as I try to show with my Mathematica graphs, the algorithm actually performs very well. The point is that it computes the matching curve based on the luminance of the pixels, and this may or may not give good results if the mode of the tone curve is set to ‘Film like’ or ‘Standard’. But it may be better if you set it to ‘Luminance’.
Other than that, I am inclined to say that there is not an immediate ‘broken’ thing in RT that can be fixed for Fuji files.
great !! thanks much for answers … understood … d
Seems my guess was not so wrong
spot on, indeed
One final thought on this after experimenting a little more: you can get a much better match with the embedded JPEG if you match on the RGB channels separately. Of course this is not how our tone-curves work atm. It might be a worthwhile feature request for the future.
Proof of principle.
Matched RAW through separate channel matching
Matched RAW through separate channel matching