Memory limit for store & input command

[helour]

Hello everyone,

I have a problem with GMIC “save” and “input” commands for large image - 30289x6511x1x3. Is there a memory limit for the “store” command?

I’m getting an error message: “Command ‘input’: Variable ‘img’ has not been assigned with command ‘store’”.

Thank you for your answers.

[grosgood (Garry R. Osgood)]

Hi helour,

This is on an 8 GB linux laptop; about 7.7 GB available for user processes:

gosgood@gwen /run/media/gosgood/SneakersOne/Beginner_s_Cookbook $ gmic input 30289,6511,1,3,255*u -name. foo -d store[foo] Foo -d input \$Foo
[gmic]-0./ Start G'MIC interpreter.
[gmic]-0./ Input image at position 0, with values '255*u' (1 image 30289x6511x1x3).
[gmic]-1./ Set name of image [0] to 'foo'.
[gmic]-1./ Display image [0] = 'foo', from point (15144,3255,0).
[0] = 'foo':
  size = (30289,6511,1,3) [2256 Mio of floats].
  data = (249.361,45.4798,233.304,200.868,38.1821,110.322,229.878,127.091,203.867,2.21882,223.635,85.6301,(...),209.724,243.997,4.71862,32.8247,6.70649,28.4258,250.175,126.506,9.63048,195.307,160.334,80.2144).
  min = 2.96859e-07, max = 255, mean = 127.497, std = 73.6128, coords_min = (28490,6451,0,1), coords_max = (21660,677,0,0).
[gmic]-1./ Store image [0] as  variable 'Foo'
[gmic]-0./ Display image [].
[gmic]-0./ Input image from variable 'Foo', at position 0
[gmic]-0./ *** Error *** Command 'input': Variable 'Foo' has not been assigned with command 'store'.

The image itself is ≈2.26 GB, but immediately before the first -display command memory requirements for G’MIC was very nearly ≈7 GB, and during the execution of the store command it exceeded 8 GB. During the restoration of the image to the image stack, via input, G’MIC aborted after some swapping and thrashing with an error message very similar to yours. I think, in my demonstration here, the root cause is simple memory exhaustion. Unless you have substantially more memory, I would surmise a similar root cause for you as well. As far as I know, store has no intrinsic memory limit. I have successfully run similar commands on platforms with 64 GB and 256 GB; store and retrieval by input work fine. Your memory requirements can be double or more than the image itself. G’MIC frequently needs to use temporary internal buffers.

[helour]

I do not know where is the problem. My computer has 16GB RAM + 16GB swap (on SSD). HTOP displays the 6.4 GB of memory used when the GMIC is running.

[grosgood (Garry R. Osgood)]

Hello helour,

Houston, I think we have a problem. First off — eliminate memory exhaustion. I’ve run the command on a 64 GB machine, the maximum process size was 9.8 GB, but the same error message
prevailed. Interestingly, while monitoring the process memory size, it dropped dramatically to a mere 2.2 GB immediately after the execution of the store command, but before input posted to the shell log.

I ran a more extensive pipeline with a few -display commands to print image list statistics, and an eval command to run a trivial bit of diagnostics in the math evaluator. Two trials, one with a much smaller trial image; the pipeline ran as expected. The pipeline failed with the 30289 × 6511 → 197,211,679 pixel image. The error message differs, and, I think furnishes some insights as to how the internals are running off the rails. Gory debug mode details follow for the two trials.

Trial 1: 3028 × 651 → 1,971,228 pixels. Works as expected.

gosgood@bertha ~ $ gmic -debug input 0 name. Dummy input 3028,651,1,3,255*u -name. foo vlen='{wh}' -d , store[foo] bar eval "{ref(get('bar',"\$vlen"),noise);print(size(noise))}" -d[Dummy] , input \$bar
[gmic]-0./ Start G'MIC interpreter (in debug mode).<gmic>-0./ Initial command line: 'cli_start , -debug input 0 name. Dummy input 3028,651,1,3,255*u -name. foo vlen={wh} -d , store[foo] bar eval {ref(get('bar',$vlen),noise);print(size(noise))} -d[Dummy] , input $bar'.
<gmic>./ Decompose command line into 22 items: 
<gmic>./   item[0] = 'cli_start'
<gmic>./   item[1] = ','
<gmic>./   item[2] = '-debug'
<gmic>./   item[3] = 'input'
<gmic>./   item[4] = '0'
<gmic>./   item[5] = 'name.'
<gmic>./   item[6] = 'Dummy'
<gmic>./   item[7] = 'input'
<gmic>./   item[8] = '3028,651,1,3,255*u'
<gmic>./   item[9] = '-name.'
<gmic>./   item[10] = 'foo'
<gmic>./   item[11] = 'vlen={wh}'
<gmic>./   item[12] = '-d'
<gmic>./   item[13] = ','
<gmic>./   item[14] = 'store[foo]'
<gmic>./   item[15] = 'bar'
<gmic>./   item[16] = 'eval'
<gmic>./   item[17] = '{ref(get('bar',$vlen),noise);print(size(noise))}'
<gmic>./   item[18] = '-d[Dummy]'
<gmic>./   item[19] = ','
<gmic>./   item[20] = 'input'
<gmic>./   item[21] = '$bar'

<gmic>-0./ Enter scope './'.
<gmic>-0./ Item[0]: 'cli_start', selection [].
<gmic>-0./ Found custom command 'cli_start: ' (takes no arguments).
<gmic>-0./ Expand command line for command 'cli_start' to: ''.
<gmic>-0./cli_start/ Return from empty command 'cli_start/'.
<gmic>-0./ Item[2]: '-debug' -> 'debug', selection [].
<gmic>-0./ Item[3]: 'input', selection [].
<gmic>-0./ Command 'input': arguments = '0'.
[gmic]-0./ Input empty image at position 0 (1 image 0x0x0x0).
<gmic>-1./ Item[5]: 'name.', selection [0].
<gmic>-1./ Command 'name': arguments = 'Dummy'.
[gmic]-1./ Set name of image [0] to 'Dummy'.
<gmic>-1./ Item[7]: 'input', selection [0].
<gmic>-1./ Command 'input': arguments = '3028,651,1,3,255*u'.
[gmic]-1./ Input image at position 1, with values '255*u' (1 image 3028x651x1x3).
<gmic>-2./ Item[9]: '-name.' -> 'name.', selection [1].
<gmic>-2./ Command 'name': arguments = 'foo'.
[gmic]-2./ Set name of image [1] to 'foo'.
<gmic>-2./ Item[11]: 'vlen={wh}' -> 'vlen=1971228', selection [].
[gmic]-2./ Set local variable 'vlen=1971228'.
<gmic>-2./ Item[12]: '-d' -> 'd', selections [0,1].
<gmic>-2./ Command 'display': arguments = ','.
[gmic]-2./ Display images [0,1] = 'Dummy, foo', from point (0,0,0).
[0] = 'Dummy':
  size = (0,0,0,0) [0 b of floats].
  data = (nil) = () [float].
[1] = 'foo':
  size = (3028,651,1,3) [22 Mio of floats].
  data = 0x7f9e01e22010 = (35.5049,168.419,252.065,193.653,59.9958,168.669,142.534,56.1951,196.325,66.4267,14.5031,78.2318,(...),4.21824,113.592,175.492,224.546,207.438,136.655,190.332,38.8058,149.642,240.614,38.8795,122.476).
  min = 2.98047e-05, max = 255, mean = 127.552, std = 73.6039, coords_min = (2527,501,0,0), coords_max = (1593,524,0,1).
<gmic>-2./ Item[14]: 'store[foo]', selection [1].
<gmic>-2./ Command 'store': arguments = 'bar'.
[gmic]-2./ Store image [1] as  variable 'bar'
<gmic>-1./ Item[16]: 'eval', selection [0].
[gmic_math_parser] size(noise) = 1971228 (compiled as 'const scalar', memslot = 38)
[gmic_math_parser] size(noise) = 1971228
<gmic>-1./ Command 'eval': arguments = '{ref(get('bar',$vlen),noise);print(size(noise))}' -> '1971228'.
[gmic]-1./ Evaluate expression '1971228' and assign it to status.
<gmic>-1./ Item[18]: '-d[Dummy]' -> 'd[Dummy]', selection [0].
<gmic>-1./ Command 'display': arguments = ','.
[gmic]-1./ Display image [0] = 'Dummy', from point (0,0,0).
[0] = 'Dummy':
  size = (0,0,0,0) [0 b of floats].
  data = (nil) = () [float].
<gmic>-1./ Item[20]: 'input', selection [0].
<gmic>-1./ Command 'input': arguments = '$bar' -> '*store/bar'.
[gmic]-1./ Input image from variable 'bar', at position 1 (1 image 3028x651x1x3).
[gmic]-2./ Display images [0,1] = 'Dummy, foo'.
[0] = 'Dummy':
  size = (0,0,0,0) [0 b of floats].
  data = (nil) = () [float].
[1] = 'foo':
  size = (3028,651,1,3) [22 Mio of floats].
  data = 0x56139defd6a0 = (35.5049,168.419,252.065,193.653,59.9958,168.669,142.534,56.1951,196.325,66.4267,14.5031,78.2318,(...),4.21824,113.592,175.492,224.546,207.438,136.655,190.332,38.8058,149.642,240.614,38.8795,122.476).
  min = 2.98047e-05, max = 255, mean = 127.552, std = 73.6039, coords_min = (2527,501,0,0), coords_max = (1593,524,0,1).
<gmic>-2./ Exit scope './'.

[gmic]-2./ End G'MIC interpreter.

Trial 2: 30289 × 6511 → 197,211,679 pixels. Fails with error.

gosgood@bertha ~ $ gmic -debug input 0 name. Dummy input 30289,6511,1,3,255*u -name. foo vlen='{wh}' -d , store[foo] bar eval "{ref(get('bar',"\$vlen"),noise);print(size(noise))}" -d[Dummy] , input \$bar
[gmic]-0./ Start G'MIC interpreter (in debug mode).<gmic>-0./ Initial command line: 'cli_start , -debug input 0 name. Dummy input 30289,6511,1,3,255*u -name. foo vlen={wh} -d , store[foo] bar eval {ref(get('bar',$vlen),noise);print(size(noise))} -d[Dummy] , input $bar'.
<gmic>./ Decompose command line into 22 items: 
<gmic>./   item[0] = 'cli_start'
<gmic>./   item[1] = ','
<gmic>./   item[2] = '-debug'
<gmic>./   item[3] = 'input'
<gmic>./   item[4] = '0'
<gmic>./   item[5] = 'name.'
<gmic>./   item[6] = 'Dummy'
<gmic>./   item[7] = 'input'
<gmic>./   item[8] = '30289,6511,1,3,255*u'
<gmic>./   item[9] = '-name.'
<gmic>./   item[10] = 'foo'
<gmic>./   item[11] = 'vlen={wh}'
<gmic>./   item[12] = '-d'
<gmic>./   item[13] = ','
<gmic>./   item[14] = 'store[foo]'
<gmic>./   item[15] = 'bar'
<gmic>./   item[16] = 'eval'
<gmic>./   item[17] = '{ref(get('bar',$vlen),noise);print(size(noise))}'
<gmic>./   item[18] = '-d[Dummy]'
<gmic>./   item[19] = ','
<gmic>./   item[20] = 'input'
<gmic>./   item[21] = '$bar'

<gmic>-0./ Enter scope './'.
<gmic>-0./ Item[0]: 'cli_start', selection [].
<gmic>-0./ Found custom command 'cli_start: ' (takes no arguments).
<gmic>-0./ Expand command line for command 'cli_start' to: ''.
<gmic>-0./cli_start/ Return from empty command 'cli_start/'.
<gmic>-0./ Item[2]: '-debug' -> 'debug', selection [].
<gmic>-0./ Item[3]: 'input', selection [].
<gmic>-0./ Command 'input': arguments = '0'.
[gmic]-0./ Input empty image at position 0 (1 image 0x0x0x0).
<gmic>-1./ Item[5]: 'name.', selection [0].
<gmic>-1./ Command 'name': arguments = 'Dummy'.
[gmic]-1./ Set name of image [0] to 'Dummy'.
<gmic>-1./ Item[7]: 'input', selection [0].
<gmic>-1./ Command 'input': arguments = '30289,6511,1,3,255*u'.
[gmic]-1./ Input image at position 1, with values '255*u' (1 image 30289x6511x1x3).
<gmic>-2./ Item[9]: '-name.' -> 'name.', selection [1].
<gmic>-2./ Command 'name': arguments = 'foo'.
[gmic]-2./ Set name of image [1] to 'foo'.
<gmic>-2./ Item[11]: 'vlen={wh}' -> 'vlen=197211679', selection [].
[gmic]-2./ Set local variable 'vlen=197211679'.
<gmic>-2./ Item[12]: '-d' -> 'd', selections [0,1].
<gmic>-2./ Command 'display': arguments = ','.
[gmic]-2./ Display images [0,1] = 'Dummy, foo', from point (0,0,0).
[0] = 'Dummy':
  size = (0,0,0,0) [0 b of floats].
  data = (nil) = () [float].
[1] = 'foo':
  size = (30289,6511,1,3) [2256 Mio of floats].
  data = 0x7fb121656010 = (80.4135,221.374,73.2544,194.962,48.5448,66.2645,135.319,242.799,142.155,90.4665,150.762,254.957,(...),61.3581,135.49,94.5716,43.9799,105.917,38.6727,88.7924,7.80504,112.064,88.717,196.653,210.172).
  min = 7.12462e-07, max = 255, mean = 127.499, std = 73.6133, coords_min = (26309,819,0,2), coords_max = (13980,987,0,0).
<gmic>-2./ Item[14]: 'store[foo]', selection [1].
<gmic>-2./ Command 'store': arguments = 'bar'.
[gmic]-2./ Store image [1] as  variable 'bar'
<gmic>-1./ Item[16]: 'eval', selection [0].
[gmic_math_parser] size(noise) = 197211679 (compiled as 'const scalar', memslot = 38)
[gmic]-1./ *** Error *** Item substitution '{ref(get('bar',197211679),noise);print(size(noise))}': CImg header not found in serialized buffer.

gosgood@bertha ~ $ 

There was also a dramatic drop in process memory consumption during, or immediately after the store command executed, again to about 2.2 GB. Interestingly, the math evaluator did report on the size of the data set as it appeared in the math evaluator. The error message suggests, along with process size dropping, that the buffer was de-allocated prematurely.

This, I think, may be a two-pipe problem…

[afre (Alan)]

This peeves me, when my infamous laptop rejects slightly larger images (which are very small by today’s standards ). If you can punch this problem into submission or explain-fu away, I will rejoice. PS - I never use store.

[David_Tschumperle (G'MIC staff)]

That is interesting. I’ll investigate to see what happens.
Just to know… What OS are you using?

[helour]

@David_Tschumperle GMIC Version 3.0.2, Linux 5.10.105-1-MANJARO SMP PREEMPT x86_64

[grosgood (Garry R. Osgood)]

gosgood@bertha $ gmic -version

gmic: GREYC’s Magic for Image Computing: command-line interface
Version 3.1.0 (pre-release #220306)
(https://gmic.eu)

    Copyright (c) Since 2008, David Tschumperlé / GREYC / CNRS.
    (https://www.greyc.fr)

gosgood@bertha $ uname -a
Linux bertha 5.15.11-gentoo-bertha #2 SMP Sun Jan 16 11:17:01 EST 2022 x86_64 Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz GenuineIntel GNU/Linux

[David_Tschumperle (G'MIC staff)]

I confirm the strange behavior here too (Ubuntu Linux). I’ll try to understand what happens.

[KaRo (Karsten R)]

It is on MacOS 11.6.5 with Version 3.1.0 (pre-release #220319) the same behavior .

[grosgood (Garry R. Osgood)]

The operation succeeds with requests for compressed storage:
Trial 1: request store[foo] 1,bar — Command succeeds.

gosgood@bertha ~/git_repositories/gmic-community/tutorial/Beginner_s_Cookbook $ gmic -debug input 0 name. Dummy input 30289,6511,1,3,255*u -name. foo vlen='{wh}' -d , store[foo] 1,bar eval "{ref(get('bar',"\$vlen"),noise);print(size(noise))}" -d[Dummy] , input \$bar
[gmic]-0./ Start G'MIC interpreter (in debug mode).<gmic>-0./ Initial command line: 'cli_start , -debug input 0 name. Dummy input 30289,6511,1,3,255*u -name. foo vlen={wh} -d , store[foo] 1,bar eval {ref(get('bar',$vlen),noise);print(size(noise))} -d[Dummy] , input $bar'.
<gmic>./ Decompose command line into 22 items: 
<gmic>./   item[0] = 'cli_start'
<gmic>./   item[1] = ','
<gmic>./   item[2] = '-debug'
<gmic>./   item[3] = 'input'
<gmic>./   item[4] = '0'
<gmic>./   item[5] = 'name.'
<gmic>./   item[6] = 'Dummy'
<gmic>./   item[7] = 'input'
<gmic>./   item[8] = '30289,6511,1,3,255*u'
<gmic>./   item[9] = '-name.'
<gmic>./   item[10] = 'foo'
<gmic>./   item[11] = 'vlen={wh}'
<gmic>./   item[12] = '-d'
<gmic>./   item[13] = ','
<gmic>./   item[14] = 'store[foo]'
<gmic>./   item[15] = '1,bar'
<gmic>./   item[16] = 'eval'
<gmic>./   item[17] = '{ref(get('bar',$vlen),noise);print(size(noise))}'
<gmic>./   item[18] = '-d[Dummy]'
<gmic>./   item[19] = ','
<gmic>./   item[20] = 'input'
<gmic>./   item[21] = '$bar'

<gmic>-0./ Enter scope './'.
<gmic>-0./ Item[0]: 'cli_start', selection [].
<gmic>-0./ Found custom command 'cli_start: ' (takes no arguments).
<gmic>-0./ Expand command line for command 'cli_start' to: ''.
<gmic>-0./cli_start/ Return from empty command 'cli_start/'.
<gmic>-0./ Item[2]: '-debug' -> 'debug', selection [].
<gmic>-0./ Item[3]: 'input', selection [].
<gmic>-0./ Command 'input': arguments = '0'.
[gmic]-0./ Input empty image at position 0 (1 image 0x0x0x0).
<gmic>-1./ Item[5]: 'name.', selection [0].
<gmic>-1./ Command 'name': arguments = 'Dummy'.
[gmic]-1./ Set name of image [0] to 'Dummy'.
<gmic>-1./ Item[7]: 'input', selection [0].
<gmic>-1./ Command 'input': arguments = '30289,6511,1,3,255*u'.
[gmic]-1./ Input image at position 1, with values '255*u' (1 image 30289x6511x1x3).
<gmic>-2./ Item[9]: '-name.' -> 'name.', selection [1].
<gmic>-2./ Command 'name': arguments = 'foo'.
[gmic]-2./ Set name of image [1] to 'foo'.
<gmic>-2./ Item[11]: 'vlen={wh}' -> 'vlen=197211679', selection [].
[gmic]-2./ Set local variable 'vlen=197211679'.
<gmic>-2./ Item[12]: '-d' -> 'd', selections [0,1].
<gmic>-2./ Command 'display': arguments = ','.
[gmic]-2./ Display images [0,1] = 'Dummy, foo', from point (0,0,0).
[0] = 'Dummy':
  size = (0,0,0,0) [0 b of floats].
  data = (nil) = () [float].
[1] = 'foo':
  size = (30289,6511,1,3) [2256 Mio of floats].
  data = 0x7f055acd8010 = (49.4669,171.222,220.399,89.017,28.4953,143.995,94.947,216.46,176.521,27.8731,186.533,31.1956,(...),176.772,153.338,87.1895,8.61013,116.001,234.646,229.843,118.401,120.916,169.243,251.669,17.7868).
  min = 8.90577e-07, max = 255, mean = 127.499, std = 73.6092, coords_min = (8159,4897,0,0), coords_max = (754,1181,0,0).
<gmic>-2./ Item[14]: 'store[foo]', selection [1].
<gmic>-2./ Command 'store': arguments = '1,bar'.
[gmic]-2./ Store image [1] as compressed variable 'bar'
<gmic>-1./ Item[16]: 'eval', selection [0].
[gmic_math_parser] size(noise) = 197211679 (compiled as 'const scalar', memslot = 38)
[gmic_math_parser] size(noise) = 197211679
<gmic>-1./ Command 'eval': arguments = '{ref(get('bar',$vlen),noise);print(size(noise))}' -> '197211679'.
[gmic]-1./ Evaluate expression '197211679' and assign it to status.
<gmic>-1./ Item[18]: '-d[Dummy]' -> 'd[Dummy]', selection [0].
<gmic>-1./ Command 'display': arguments = ','.
[gmic]-1./ Display image [0] = 'Dummy', from point (0,0,0).
[0] = 'Dummy':
  size = (0,0,0,0) [0 b of floats].
  data = (nil) = () [float].
<gmic>-1./ Item[20]: 'input', selection [0].
<gmic>-1./ Command 'input': arguments = '$bar' -> '*store/bar'.
[gmic]-1./ Input image from variable 'bar', at position 1 (1 image 30289x6511x1x3).
[gmic]-2./ Display images [0,1] = 'Dummy, foo'.
[0] = 'Dummy':
  size = (0,0,0,0) [0 b of floats].
  data = (nil) = () [float].
[1] = 'foo':
  size = (30289,6511,1,3) [2256 Mio of floats].
  data = 0x7f055acd8010 = (49.4669,171.222,220.399,89.017,28.4953,143.995,94.947,216.46,176.521,27.8731,186.533,31.1956,(...),176.772,153.338,87.1895,8.61013,116.001,234.646,229.843,118.401,120.916,169.243,251.669,17.7868).
  min = 8.90577e-07, max = 255, mean = 127.499, std = 73.6092, coords_min = (8159,4897,0,0), coords_max = (754,1181,0,0).
<gmic>-2./ Exit scope './'.

[gmic]-2./ End G'MIC interpreter.
gosgood@bertha ~/git_repositories/gmic-community/tutorial/Beginner_s_Cookbook $ 

Trial 2: request store[foo] bar — Command fails with “CImg header not found in serialized buffer.”

gosgood@bertha ~/git_repositories/gmic-community/tutorial/Beginner_s_Cookbook $ gmic -debug input 0 name. Dummy input 30289,6511,1,3,255*u -name. foo vlen='{wh}' -d , store[foo] bar eval "{ref(get('bar',"\$vlen"),noise);print(size(noise))}" -d[Dummy] , input \$bar[gmic]-0./ Start G'MIC interpreter (in debug mode).<gmic>-0./ Initial command line: 'cli_start , -debug input 0 name. Dummy input 30289,6511,1,3,255*u -name. foo vlen={wh} -d , store[foo] bar eval {ref(get('bar',$vlen),noise);print(size(noise))} -d[Dummy] , input $bar'.
<gmic>./ Decompose command line into 22 items: 
<gmic>./   item[0] = 'cli_start'
<gmic>./   item[1] = ','
<gmic>./   item[2] = '-debug'
<gmic>./   item[3] = 'input'
<gmic>./   item[4] = '0'
<gmic>./   item[5] = 'name.'
<gmic>./   item[6] = 'Dummy'
<gmic>./   item[7] = 'input'
<gmic>./   item[8] = '30289,6511,1,3,255*u'
<gmic>./   item[9] = '-name.'
<gmic>./   item[10] = 'foo'
<gmic>./   item[11] = 'vlen={wh}'
<gmic>./   item[12] = '-d'
<gmic>./   item[13] = ','
<gmic>./   item[14] = 'store[foo]'
<gmic>./   item[15] = 'bar'
<gmic>./   item[16] = 'eval'
<gmic>./   item[17] = '{ref(get('bar',$vlen),noise);print(size(noise))}'
<gmic>./   item[18] = '-d[Dummy]'
<gmic>./   item[19] = ','
<gmic>./   item[20] = 'input'
<gmic>./   item[21] = '$bar'

<gmic>-0./ Enter scope './'.
<gmic>-0./ Item[0]: 'cli_start', selection [].
<gmic>-0./ Found custom command 'cli_start: ' (takes no arguments).
<gmic>-0./ Expand command line for command 'cli_start' to: ''.
<gmic>-0./cli_start/ Return from empty command 'cli_start/'.
<gmic>-0./ Item[2]: '-debug' -> 'debug', selection [].
<gmic>-0./ Item[3]: 'input', selection [].
<gmic>-0./ Command 'input': arguments = '0'.
[gmic]-0./ Input empty image at position 0 (1 image 0x0x0x0).
<gmic>-1./ Item[5]: 'name.', selection [0].
<gmic>-1./ Command 'name': arguments = 'Dummy'.
[gmic]-1./ Set name of image [0] to 'Dummy'.
<gmic>-1./ Item[7]: 'input', selection [0].
<gmic>-1./ Command 'input': arguments = '30289,6511,1,3,255*u'.
[gmic]-1./ Input image at position 1, with values '255*u' (1 image 30289x6511x1x3).
<gmic>-2./ Item[9]: '-name.' -> 'name.', selection [1].
<gmic>-2./ Command 'name': arguments = 'foo'.
[gmic]-2./ Set name of image [1] to 'foo'.
<gmic>-2./ Item[11]: 'vlen={wh}' -> 'vlen=197211679', selection [].
[gmic]-2./ Set local variable 'vlen=197211679'.
<gmic>-2./ Item[12]: '-d' -> 'd', selections [0,1].
<gmic>-2./ Command 'display': arguments = ','.
[gmic]-2./ Display images [0,1] = 'Dummy, foo', from point (0,0,0).
[0] = 'Dummy':
  size = (0,0,0,0) [0 b of floats].
  data = (nil) = () [float].
[1] = 'foo':
  size = (30289,6511,1,3) [2256 Mio of floats].
  data = 0x7fbc83b9d010 = (91.726,67.7343,245.684,96.4213,18.9277,32.8585,18.9377,39.5857,118.034,91.7279,149.185,145.83,(...),186.021,56.1959,86.2835,121.414,67.1728,116.707,10.4969,102.863,128.678,45.1618,66.6846,232.6).
  min = 7.12462e-07, max = 255, mean = 127.499, std = 73.6124, coords_min = (22849,1716,0,1), coords_max = (10894,392,0,0).
<gmic>-2./ Item[14]: 'store[foo]', selection [1].
<gmic>-2./ Command 'store': arguments = 'bar'.
[gmic]-2./ Store image [1] as  variable 'bar'
<gmic>-1./ Item[16]: 'eval', selection [0].
[gmic_math_parser] size(noise) = 197211679 (compiled as 'const scalar', memslot = 38)
[gmic]-1./ *** Error *** Item substitution '{ref(get('bar',197211679),noise);print(size(noise))}': CImg header not found in serialized buffer.

gosgood@bertha ~/git_repositories/gmic-community/tutorial/Beginner_s_Cookbook $ 

This furnishes a workaround. And as an added advantage, the extra compression delays supplies time for potty breaks.

[David_Tschumperle (G'MIC staff)]

Well, currently investigating, and what I see here is that I store the size of the buffer into an unsigned int.
In the case of a 30289x6511x1x3 image, we have 591635037 float values, so a buffer size of 2366540148 bytes (1 float = 4 bytes), which is actually a bit more than 2^31 (2147483648). I certainly have some cast from uint32 to int32 somewhere in the code, which could lead to an overflow.
This has to be improved BTW

[grosgood (Garry R. Osgood)]

Ah. Likely, then, that an ever-larger compressed image will overflow too.

[David_Tschumperle (G'MIC staff)]

Yes, that’s actually why I’m thinking about changing the way serialize works (it is used internatlly by store).
Currently, serialize always return a 1xNx1x1 image, where N is basically the buffer size (+ some header containing the information of the original image size).
Doing this will limit the buffer size N to be around 2^{31}, which is not a good idea.

I think serialize could rather return a 2D image MxNx1x1 so that MxN could store the buffer size if it’s larger than 2^{31}. And even more, if the buffer size is larger than 2^{2*31} , I could use the depth and the spectrum to allow even larger sizes.
I guess allowing a buffer size of 2^{4*31} should be more than enough.

[David_Tschumperle (G'MIC staff)]

The following commits should solve your issue.

Allow command 'store' to manage bigger images.

and

Allow command 'store' to manage bigger images.

At least it works now on your case. Not tested thoroughly, so feel free to test by yourself!
I’m not able to build new prerelease packages right now, because I don’t have access to my PC at work with SSH right now. I’ll post new binaries with the fix tomorrow!

[KaRo (Karsten R)]

@grosgood

gmic -debug input 0 name. Dummy input 30289,6511,1,3,255*u -name. foo vlen='{wh}' -d , store[foo] bar eval "{ref(get('bar',"\$vlen"),noise);print(size(noise))}" -d[Dummy] , input \$bar

I get a new error:
[gmic]-2./ Store image [1] as variable ‘bar’
[gmic_math_parser] size(noise) = 197211679 (compiled as ‘const scalar’, memslot = 38)
[gmic]-1./ *** Error *** Item substitution ‘{ref(get(‘bar’,197211679),noise);print(size(noise))}’: Invalid request of a shared-memory subset (4131389441->1,0,0,0).

[David_Tschumperle (G'MIC staff)]

@KaRo

I get a new error:
[gmic]-2./ Store image [1] as variable ‘bar’
[gmic_math_parser] size(noise) = 197211679 (compiled as ‘const scalar’, memslot = 38)
[gmic]-1./ *** Error *** Item substitution ‘{ref(get(‘bar’,197211679),noise);print(size(noise))}’: Invalid request of a shared-memory subset (4131389441->1,0,0,0).

The amount of memory for the variables of the math parser are limited to 2^31 = 2Gb.
I’ll try to detect when this happens and print a more clear error message.

[David_Tschumperle (G'MIC staff)]

A few information about what kind of limitations we can expect from G’MIC:

• In G’MIC, an image has 4 coordinates : width, height, depth and spectrum , each being stored as an unsigned int (aka uint32, 32-bits integer).
• So theoretically, each axis can have a dimension up to 2^{32}-1 = 4294967295.
• But, as many commands do “signed” calculus with these dimensions, they consider them to be int32 rather than uint32, so this somehow limits each dimension to go up to 2^{31}-1 = 2147483647 (\simeq 2GB).
• Clearly, this 2^{31} is still good enough to store a regular image.
• Where it gets tricky is when some commands have to internally use one-dimensional images to store an entire image buffer. This is done in some places in the code. The most obvious command is, e.g. unroll that produces a 1D image where only one of the axis is non-zero and has actually a dimension of whds. So, in that case, you’ll get some issues with big images.
• In the same way, the math evaluator uses an internal memory for variables that is stored as a one-dimensional image, so the math evaluator gets the same limitation : the size of all the variables required by a math expression cannot be larger than 2GB.

I’ve done a few tricks yesterday to circumvent these limitations for the serialize, unserialize and store commands, but this is a bit hacky.

[grosgood (Garry R. Osgood)]

@helour

Is there a memory limit for the “store” command?

Getting back to helour’s original question, the answer is “yes,” and you have furnished the particulars. For my lot, it means revisiting some of the hoary old tutorial pages, such as Images and reconsider some of the more bombastic phrasings there and provide practical limits as of a particular version level. All in a day’s work…

[KaRo (Karsten R)]

humm, with this command
gmic -debug input 0 name. Dummy input 3028,651,1,3,255\*u -name. foo vlen='{wh}' -d , store\[foo\] bar eval "{ref(get('bar',"\$vlen"),noise);print(size(noise))}" -d\[Dummy\] , input \$bar

I get still the new error message

<gmic>-1./ Item[16]: ‘eval’, selection [0].
[gmic]-1./ *** Error *** Item substitution ‘{ref(get(‘bar’,1971228),noise);print(size(noise))}’: Function ‘get()’: Specified variable size 1.97123e+06 is larger than 2^31.