No there is no size limit, except the available memory.
G’MIC stores the image with 32bits-float-valued values, so a MxN color image takes MxNx3x4 bytes in memory. Some filter requires 3 or 4 copies of the image.
If the problem happens only with the appimage, then maybe it’s an appimage-related issue ? 