The image caching logic implemented so far in PhotoFlow was rather slow and unstable, and it was not possible to easily control the total amount of memory used by the cache buffers. It could hence induce heavy swapping in some cases (ping @gadolf on this) .
In the past couple of weeks I have been working hard to re-write the image caching code, and provide something which is more stable and has a tunable maximum memory footprint.
The new caching should provide faster processing of the preview images, while it might give longer export times with some images. At the moment the total amount of memory reserved for caching is approximately 1GB, but it will soon become user-configurable.
I have been doing tests of the new code for few days now, and hopefully I have spotted and fixed all the bugs related to the new caching logic, but… who knows?
So I need your help to test this new feature, report crashes or glitches, and provide me some feedback wether you prefer or not the new code!
Packages will be available for download from the continuous release page (look for the new-caching label).
That’s good to hear! The speed-ups concern mostly the preview image, and therefore are more difficult to quantify…
I am not very much surprised, unfortunately. The changes I made were quite substantial, and I suspect they introduced some thread synchronization bugs that are difficult to chase.
Could you remind me what operating system are you using?
The best would be to use a version built in RelWithDebInfo mode, and run the program through valgrind, but that’s only possible on some systems.
For the Windows case I am looking into adding support for Dr. Mingw, but that’s not ready yet…
Meanwhile, it would be already helpful to provide me the pfi file that triggers the crash and a short description of what you have been doing…
I should have talked about memory, not speed, actually. Regarding memory use, from my perspective, it seems the same behavior as your last simplified pipeline release, that already ported changes to memory use. In summary, no swapping and no memory fill up.
The speed up on the preview image is really nice - the Shadows/Highlights layer is now almost keeping up with slider adjustments, whereas before I was typing numbers and waiting.
Threading bugs are the worst to find. Have you tried the various ‘sanitiser’ modes in clang? The address sanitiser and the thread sanitiser are good for tracking this sort of stuff down, and they don’t slow the application down too much.
I have a crash which I thought was reproducable, but it has gone away now!
I was loading a .pfi file which contains a Richardson-Lucy sharpening layer, then zooming to 1:1 and scrolling. This provokes an assertion failure which terminates photoflow (except when it doesn’t).
Richardson-Lucy deconvolution looks really bad in the preview - this is probably unavoidable if it is working from a low-resolution version of the image.
Maybe you could add an option for ‘calculate this layer at full resolution’ or a full resolution cache layer or something.
I’m trying to reproduce the capture sharpening feature in RawTherapee 5.7dev by using:
Optical Corrections
RL Sharpen (0.8-1.0 sigma, 10-30 iterations)
Raw Developer (with lens distortion correction turned off, but CA correction on)
This seems to be quite effective.
I can just turn sharpening off if viewing at small sizes.
This is exactly the kind of bug I have been chasing (and I hoped to have fixed)…
I have started to check the code with CLang’s address sanitizer. and it reports some problem with the RL deconvolution code (which is not related to the new caching logic). Could you check if you have crashes also without using RL sharpening?
Yes. Is it all about cloning the right branch from git and then just following the steps described on github PF main page? I believe I should add some extra settings while compiling, which ones?
Just use it like you normally do, possibly using tools that require caching (local contrast, shadows/highlights, relight, etc…). If it crashes, provide me the terminal output so that I can possibly see the reason.
My laptop is definitely lower end and older than the other two testers’. At 1:1, shadows/highlights definitely makes the scrolling refresh much slower. I cranked up local contrast, which was incredibly slow on my machine before; now, changing the settings is much more responsive.
Okay, just as I was typing this message, PF finally crashed. I have a feeling it has to do with cache memory allocation; i.e., no issues until I scrolled more at 1:1.
I did some scrolling – no crash – and then left PF running while doing something else on the laptop. Eventually, it crashed. Here is the log: log.txt (74.9 KB).
This was when scrolling at 1:1 scale. I’ve also seen the crash when changing the settings on shadows/highlights with the preview zoomed to fit the window.
I have prepared a new version with more checks of the tile reference counting, this should give a more detailed location of the reference counting error… the MacOS package should be ready in less than one hour.
I am testing the code on MacOS as well, and so far I cannot trigger any crash when zooming, panning or moving sliders…
@afre the log you pasted corresponds to a clean program shutdown, not a crash… are you sure it is the good one?
Yes, that is what I read too… Maybe I did close it unintentionally. Anyway, my laptop stalled / crashed soon after, showing its age.
What I did was test RL, save the PFI, close PF, open PFI, zoom and scroll, open raw, close RL tab, test highlights/shadows, zoom and scroll THEN crash. (Perhaps, the cache didn’t clear between app or tab closures? Also, closing tab didn’t crash PF, which is new and good for me!)
Second time testing highlights/shadows with logging, I may have unintentionally closed the console but I am not sure if I did or it crashed by itself.