Filter Auto-Balance eating all RAM with large images.

Software G'MIC
1

Hello,

I have some very large images like this one ( Sentinel-2 satellite images ) :
2018-08-08_085118

In gimp 2.10.4 trying to use “Auto Balance” filter in Gmic QT 2.3.3 the preview works, but when i try to export as new active layer Gmic eats quickly around 7-8GB of Ram before crashing and giving me this message :

2018-08-08_084030
( same if i choose to apply the filter on the input layer, " Sur place" in French for the default output mode, not sure how it is named in English )

I tried with disabling the preview before exporting in case it could free up more RAM but no dice, filter eats all free RAM and crash, all i can see is Gmic-QT reaching 7-8GB of memory and crash, maybe eats a bit more than 8 but it goes too fast to see it reported

Got 16GB of Ram on my machine, (3 of them are assigned to a small Ramdisk ), no windows Virtual Memory swap for reasons… all non necessary programs not running, reduced Gimp swap file size to 1 GB only instead of usual 6-8GB assigned, keeping only the final layer and removing all others and before starting Gmic plugin i clean the history in Gimp to free up some more memory but no way to get such large image rendered.

Didn’t tried a lot of other plugins to compare but some are not consuming so much RAM and can render this large image.

There isn’t probably much that can be done apart from activating the windows swap/Paging Virtual Memory… but i’d like to know how Gmic manage memory, if there is a limit we can set maybe or if like gimp it can use a fast swap folder when Ram limit is reached, well anything that could prevent it to eat all the RAM and just crash.

Or maybe someone have an idea or trick i could try to bypass this.

Thanks in advance.

2

This is based on my limited experience, so my remarks may or may not be applicable.

The reality is that images take a lot of RAM to process, even when they are medium sized. And you have extremely large ones! G’MIC does calculations with 32-bit float precision. Unsure how to reduce that. ImageMagick, e.g., allows you to use a lower precision and / or manage load for more demanding tasks.

What I don’t get is why you would turn off Windows’ and reduce GIMP’s swap. I might be generalizing but swap is there to compensate when RAM is insufficient and to store temp files. Photoshop, e.g., uses your whole disk drive for swap by default!

Another thing to note is that if you set Windows’ swap to something other than default, i.e., to auto mode, the OS becomes unpredictably unstable.

1 Like
3

Thank you for the answer,

I did not activated windows paging itself since years, when win XP was still the top OS, done the same when win7 was released, i’m still on win7 and system runs way better without pagging/swap managed by windows.

For gimp and reducing the size:

Well at first i thought that this setting was the size of the swap folder on the SSD/HDD gimp will use, i had it set to a value around 12GB for long time and never had issues with gimp, never monitored the RAM usage either.

Then lately when starting to mess with those large Sentile-2 stattelite images i run out of memory in Gimp itself and crashed it numerous times… until i discovered that the setting you set there is the amount of RAM allocated to Gimp, the number set is not the swap folder size on the SSD/HDD but the amount of RAM after which gimp will start to use the swap folder, so i reduced it from 6-8 GB i have set since my discovery to only 1GB to keep a max of free ram for Gmic to process but still isn’t enough for this filter.

Image is in 16 bits i might try to set gimp to 8 bits but if Gmic process everything in 32 bits it won’t do any difference.

Last solution that i can try is set the things in Gmic without rendering, get command line from log file and then close gimp to free up some memory and then try Gmic CLI but not sure if it will be enough, i’ll give a try tomorrow.

I don’t think there is much else i can do beside activating windows swap itself after that.

Thank you.

4

I haven’t made a suggestion yet. Use one or more of the following:

1. Use CLI.

2. Let Windows manage its own swap. As I said above, this makes the OS and the apps it runs more stable.

3. a) Do a test run to get the parameters first.
– Resize (reduce) the image. Use the nearest neighbour method.
– And / Or, Crop to a (smaller) sample.
b) Then separate the full image into tiles and then combine them after processing. This might take coding effort with which I cannot help you.

4. a) Modify the code or use only the chunks that you need. I doubt you need all of it.

b) Ask @garagecoder. He is nice. :slight_smile:

2 Likes
5

Thanks for the suggestions,

your forgot #5 buy more RAM, but machine is not that young ( DDR3 still ) and RAM cost a kidney nowadays :confused:

1 : CLI like i said in previous post is what i will try tomorrow and see, without Gimp + gmic opened it might be enough memory gain for this filter.

2 : I’ll probably enable it just for this task and delete it after work sessions being done if method 1 does not work.

This said Windows pagefile is not really something i like, it takes space on the SSD that is already well filled, it does a lot of I/O for nothing, even when you have 10 GB of free RAM remaining and doing tasks not demanding more than 1GB… It have a tendency at moving a lot of stuff preemptively for nothing. If the pagefile.sys if on your system drive you can notice it during heavy access to big files on the drive or during things like gaming with High res textures getting a tiny delay for loading.

If you set windows to clear pagefile.sys on shutdown it takes literally minutes to shut down the PC, it’s instantaneous with swap off, of course you can set it to not clear the pagefile.sys on exit but that depends on the environment where the PC is located, it can be a security problem…

That’s why it is disabled since many years and until today i never reached my full RAM capacity and never encountered a single problem despite doing quite a lot of relatively heavy things done on this machine.

3 : Not really possible to crop or reduce size for this one and quite a few others sadly, will be possible with some others tho.

Might end up doing tiles, it will be a longer process but if there is no other choice… but i’m not sure how things like this Auto balance filter and some others i plan to use will end up, I mean since one tile could result in having a lot of greens for example and another have a tint pushing more with blue for the sea borders or brown or grayish for inland soil and mountains etc etc some filters that ( i assume, i am not sure how things work exactly ) base them calculations on the image content so it might end up in a ugly patchwork picture at the end :stuck_out_tongue:

4 : I’m not a coder, beside some basic batch and a bit of powershell I’m afraid there isn’t much i can do by myself in this case, and i will certainly not try to bother someone to see if he can do something ( even if he is very nice ^^ ). Might bother someone irl and see if i can borrow one 32-64GB of RAM equipped machines, but probably won’t happen either…

Well… i first asked the question because i wasn’t sure if there was a way for Gmic to manage this by telling it to not use full RAM, to limit it somehow but apparently there isn’t so i will have to explore something else, will try solution 1 & 2 tomorrow.

Thanks again for the help.

6

Hi @Yalba, I’m reading this now to see if something can be done.
@afre very persuasive :blush: thankyou

1 Like
7

The new-fangled internet solution is: spin up a VM on a cloud provider that has a ton of RAM, rsync your stuff up, install gmic, run your gmic scripts, rsync the data down, destroy the VM. Might cost you a dollar or two :wink:

1 Like
8

Possibilities that I won’t consider. :wink:

2. You know more about it than me. I am only speaking from my limited experience. I think the problem has more to do with Windows doing a lot of other things in the background and not necessarily to do with swap. This has worsened now that I am using Windows 10, even though I try to disable everything and go as light as possible. To make things worse, MS always secretly turns them back on.

PS Does Windows 7 even have TRIM support for SSDs?

PPS Another reason for an overactive SSD may have to do with an overactive Antivirus or anti-something app that actively scans every process and byte. It could be a number of other things such as your hardware. Some hardware such as your GPU or WIFI may take some of your bandwidth. All these things may drain your resources at different frequencies and intensities, making it difficult for you to do something dedicated and demanding.

3. Interpolating tiles is an advanced topic but definitely possible. What I meant by resizing or sampling is to gather weight information so when you process the tiles you have prior info. Are you doing remote sensing and GIS type work? If so, there should be software that handles that type of thing. :sunny:

1 Like
9

That filter creates at least 5 “layers” just by itself, if we also take into account any layers created by the other commands it seems likely to end up with at least 6 (or more). End result:

10980^2 * 3 * 4 * 6/2^30 = 8gb

Then assuming some memory fragmentation due to lack of paging file, windows probably can’t assign a large enough contiguous block. I’ll see if I can alter the filter to reduce total layers (5 seems a bit high anyway).

2 Likes
10

@garagecoder : Oh thanks a lot , wasn’t expecting this :star_struck:, but don’t bother too much with this if it’s too much hassle or a headache, thank you very much.

@paperdigits : Something i never think about as not really fan of cloud based solutions for hosting stuff or machines but will keep this in mind, might serve for a lot of other things indeed. Thanks

@afre : windows 7 itself idk to be honest, never used the windows inbuilt defrag tools, it’s disabled on this machine since years, Trim is now managed by my Intel SSD toolbox software for the Intel SSD, used O&O defrag for a long time before this but Intel tool is enough now .

Edit : with AV disabled it is the same things i noticed each time i got it pagefile activated for short periods of time.

It’s for a kind of semi-personal project, can’t say much more than this, nothing secret tho :stuck_out_tongue:
I just started recently messing with those satellite images, saw there was some dedicated and powerful tools but they won’t achieve what i want to do, but there is some with really cool possibilities for the little i saw, i’ll give a try to some of those when i’ll have finished and some free time to burn :stuck_out_tongue:

Edit : @garagecoder Wow, i see now, not surprising it ens up crashing for me when i see it reaching 7-8GB.

Like i said no need to get a headache with this, if anything take your time.

And thanks again :slight_smile:

11

@Yalba Would definitely like to see what you can accomplish, if you are willing to share. Sorry for the PSs. I tend to edit my comments more than I should; I rather not have multiple posts…

12

Once i’ll will be done i’ll try ( with a different location than the one i work on :stuck_out_tongue: ) , if i manage to achieve what i have in mind, i’m not sure yet if I can achieve my goal to start with :stuck_out_tongue:

edit :
And no worries for the PS’s, was weird as all replying posts started to accumulate while i was replying to the first one ^^

Forgot to reply about the interpolating tiles, i’ll need to dig into this because for now i don’t really have an idea about this method, will have to learn more about this and try.

13

Where are you located?
I have six matched 2 gig DDR3 sticks @ 1600MHz, i.e. in total 12 gig
that I could be persuaded to part from…

Would your motherboard accept six sticks? And would that make
your situation any better?

Have fun!
Claes in Lund, Sweden

1 Like
14

@Yalba it turns out the maths is irreducable without using the g’mic math parser, in which case the whole command may as well be in it. I rewrote it and added an option “Reduce ram” for that - you’ll be able to refresh filters and get the new version within about 30 mins probably.

3 Likes
15

Wow you guys amaze me,

Thanks a lot @Claes, I’m in France but those sticks won’t marry well with the ones i already have, G.Skill Kit Extreme3 2x 8Go PC19200 Trident X CAS10 running @2400Mhz in XMP profile, and on the top of this it used to be a picky motherboard when released, many ram brands had issues with this mobo.

Thanks again a lot for the offer, really appreciate it.

I’m really amazed by all replies trying to help, very kind from your part guys, just tremendous.

16

@garagecoder : i have no more words, can’t thank you enough for that , I’ll update the filter and will give a quick try later at night, will work on this project more intensively tomorrow and see how it goes.

Thank you very much.

Thanks again for all the help guys, sincerely a big thank you to all of you.

Edit : updated and tested quickly, it can render it without crashing now, not using as much RAM as before, absolutely wonderful.

Thanks a million @garagecoder for making it possible.

Now I’ll be busy the next days and for a little while spending all my free time messing with those pics until hopefully achieving what i want :slight_smile:

Thank you all :hugs:

1 Like