And for the community files, another solution would be that each contributor puts his own G’MIC filter file somewhere (e.g. on his own site).
in practice, being totally independent of github’s future is a great thing!
That’s a whole different topic but how would you track all of this when it’s time to build a new version? (Note that i actually have no idea how the community files are included into a g’mic build. They’re just magically there).
I have a script gmic_buildpackage that gathers all community files and compile them into a single gmic file. Nothing fancy.
FYI: When ImageMagick writes floating-point tiff files, by default it sets SMINSAMPLEVALUE and SMAXSAMPLEVALUE to 0.0 and 1.0 respectively. Users can set these values to anything they want via “-define quantum:maximum=12.3” etc.
I’ve finally fixed my code to allow more large range without too much computation overhead.
I’m also trying right now some code to parallelize the search of min/max in an image to see if that can be even faster.
I am glad the issue was simpler than I thought, not to say easier to solve, but nothing related to layout of data in memory.
Compiling right now the 3.5.6 pre-release code. Thanks for sharing.
Let us know if that works for you !
The min and max values per plane might not be as useful as the min and max of the entire image for work in 3D, and even so, in my case, it doesn’t really matter. Unless it matters for gmic computations. Does it?
In the very latest version of the code I’ve pushed, the min/max are now indeed computed for the whole image and are the same for each saved plane.
I did the same test I reported above, timing considerably improved, see below (before: 2:40.98, after: 0:14.43),
time gmic stack-1-crop-xzyc_e3D_xy.tif -keep’[0-599]’ a z -o test.tif,uint8
[gmic]./ Start G’MIC interpreter (v.3.5.6).
[gmic]./ Input all frames of TIFF file ‘stack-1-crop-xzyc_e3D_xy.tif’ at position 0 (1268 images [0] = 820x612x1x1, (…),[1267] = 820x612x1x1).
[gmic]./ Keep images [0,1,2,(…),597,598,599] (600 images left).
[gmic]./ Append images [0,1,2,(…),597,598,599] along the ‘z’-axis, with alignment 0.
[gmic]./ Output image [0] as tif file ‘test.tif’, with pixel type ‘uint8’, no compression and bigtiff support (1 image 820x612x600x1).
[gmic]./ End G’MIC interpreter.
14.211u 0.900s 0:14.43 104.7% 0+0k 0+588928io 0pf+0w
Using tic toc for output timing like you did only took 4.544 s in an IBM Power8 server. Before was a brutal 8:50.60.
Many thanks for taking the time to work on this!
Would this imply that now gmic foot.tif a z max={iM} would be faster ?
Well, on my laptop:
$ gmic 820,612,600 noise 20 tic o large.tif toc
[gmic]./ Start G'MIC interpreter (v.3.5.6).
[gmic]./ Input black image at position 0 (1 image 820x612x600x1).
[gmic]./ Add gaussian noise to image [0], with standard deviation 20.
[gmic]./ Initialize timer.
[gmic]./ Output image [0] as tif file 'large.tif', with pixel type 'auto', no compression and bigtiff support (1 image 820x612x600x1).
[gmic]./ Elapsed time: 1.309 s.
[gmic]./ End G'MIC interpreter.
The file saving itself takes 1.309s.
By chance, could you try to run this command on your server to see if you have equivalent timings ?
No, that only means that when saving a tiff file, the search for the min/max values is faster because it’s done only once, for the whole image data.
Elapsed time: 3.547 s.
This is not SSD but old HDD. I modified mtune in the src/Makefile to have now mtune=power8 but I am not sure how optimization is doing in your code for this architecture - used gnu gcc compilers.
1 Like
The implication is then that reading mix and max from the Tiif Min/Max Tags you saved for each plane is not really taking into account to report ‘iM’ right after reading the file. Is that correct?
I see, thank you for all your tests.
Seems that the optimization is OK then ? comparable to what tiffcp does ?
Actually G’MIC does not use these tag values at all. It just sets it just in case other software may read it. In G’MIC, if you write max={iM} , it will always recompute the max value, by reading all image data.
Definitely okay, solved the really slow writing we had before.
The advantage of tiffcp is compression, I can get smaller files which I cannot get with gmic’s available compressions for tiffs, e.g. lzw. Here is an example (same data as above, gmic 820,612,600 noise 20 n 0,255):
gmic: 332M
tiffcp: 233M
That’s almost 1/3 reduction in size using tiffcp with aggressive zip:p9 compression. I generate many 3D tiff stacks so this difference matters, but it has not being a stopper to continue using gmic (sometimes I call tiffcp on the gmic generated tiffs with no compression).
Wouldn’t be possible to have other compression types when saving tiffs besides lzw and jpeg? libtiff provides a bunch, listed below via tiffcp --help
-c lzw[:opts] compress output with Lempel-Ziv & Welch encoding
-c zip[:opts] compress output with deflate encoding
-c lzma[:opts] compress output with LZMA2 encoding
-c zstd[:opts] compress output with ZSTD encoding
-c webp[:opts] compress output with WEBP encoding
-c jpeg[:opts] compress output with JPEG encoding
-c jbig compress output with ISO JBIG encoding
-c packbits compress output with packbits encoding
-c g3[:opts] compress output with CCITT Group 3 encoding
-c g4 compress output with CCITT Group 4 encoding
-c sgilog compress output with SGILOG encoding
-c none use no compression algorithm on output
I will extend the compression argument to be one of { none (default) | jbig | jpeg | lzma | lzw | webp | zstd }, in the next version.
Looks like it is working.
EDIT: works, but not for all images. It sometimes imposes constraints on the pixel datatype and/or number of channels.
Thanks for including those compression options on the next version, definitely welcome. ‘zip’ is algo great 
Zip is not proposed as it in the libtiff,or at least I haven’t found how to enable it.