collaborators for image tiling

gubach · November 25, 2018, 12:17pm

are there no looping constructs or decisions

until now the “language” of the CRMT (CloneRotateMirrorTranslate) approach has 6 elements. Perhaps this could some day grow to a “Pattern Description Language” that includes the usual control structures.

I guess that has to be periodic tiling only (i.e. penrose would require an increasingly long list)?

It makes no difference if a command list describes 60 proto-tiles of a periodic or non-periodic tile. The restriction is RAM because image proto-tiles should have an interesting internal texture. I am using as standard t_w = 1000 so if there is a complex tiling type like islamic patterns then moving 60 proto-tiles over a background would have serious RAM requirements.

The economic idea of periodic tiling is to make a rectangle tile from possibly non-rectangle proto-tiles and then make an arbitrary large pattern by making a n x m array of the tile (translation vector (1 1)). Or to make a non-rectangle basic cell from the proto-tiles like a hexagon in the p3m1 case and then use a more complex translation vector to distribute such hexagons. Such strategies are not working with non-periodic image tilings where any pattern detail must always be constructed by moving the proto-tiles.

this is possibly more interesting than you make it appear!

I do not know how this could possibly be make more interesting than by linking to the incredible aesthetic result pictures as motivation for developers and list writers. And the writers have also the motivation of sustainability because command lists can generate images even decades later independent of any actual interpreter implementation.

Or is the intent for that never to be used directly and only as a backend?

yes a user don’t have to get in contact with the content of the command list and the coordinate list. In my prototype this two files were selected by giving their paths and the program is doing the string and image processing.

garagecoder · November 25, 2018, 12:40pm

Since there are so few, could we have a clear definition for each? I propose something like:

Clone <element> | C<element> :
    Duplicate tile number <element> to current tile.

Rotate <angle> | R<angle> :
    Rotate  the current tile by <angle> degrees

… and so on. Obviously however you choose, but if there’s a nice clear definition of the “language” (even with a visual example for each) the task will perhaps be easier to follow.

gubach · November 25, 2018, 1:44pm

I don’t get along with the formatting of < and > you used in your example so I using "

Clone “element” | C"element":
Duplicate proto-tile number “element” from the prototile_list/cell and make it the current proto-tile for the following processing steps

Rotate “angle” | R"angle":
Rotate the current proto-tile by [degrees] (plus additional trim = auto-crop of borders that are possibly generated by rotation Cutting and Bordering -- IM v6 Examples)

Flop | Fo :
Flop the current proto-tile = left-right mirroring

Flip | Fi :
Flip the current proto-tile = top-bottom mirroring

x-translation “length” | x"length" :
Translation of current proto-tile in x direction by “length” [pixels]

y-translation “length” | x"length" :
Translation of current proto-tile in y direction by “length” [pixels]

garagecoder · November 25, 2018, 1:53pm

Thankyou, that’s quite clear enough
I’ll have a look at it now, but if I run out of time will be another week. That of course doesn’t preclude others from investigating

David_Tschumperle · November 25, 2018, 2:10pm

@garagecoder, maybe I can help saving you a little bit of time. I’ve finally managed to implement a beginning of a Style Transfer filter (https://twitter.com/gmic_ip/status/1066460833021157381).
It does not really use the idea I’ve talked with you a few days ago, so just forget about it

garagecoder · November 25, 2018, 2:18pm

Wow nice stuff, that’s good news! Work became… intense this past week so frankly I had to forget about it for now anyway
But now it’s is in my head, it’s staying on the (extremely long) list of ideas!

garagecoder · November 25, 2018, 3:27pm

I’ll use one of your tiles here (tell me if this is not acceptable and I’ll remove it!). Here’s how I interpret the method, please correct if I’m wrong:

You begin with one or more polgyons, defined using rectangular raster image and opacity mask - please excuse the rough cropping.
tile0

This is copied/mirrored/rotated/translated and rendered to a target output image. An example of “Fo, R-60” in G’MIC:

gmic tile0.png mirror x rotate -60 autocrop

tile1

The translations tend to be done in terms of a predefined constant (here triangle side length, height) and a factor.

Given that, I can think of 3 ways to handle it (without changing the “language”):

Raster based (the current method)
Vector field based (manipulate source pixel positions rather than the data itself)
Vertice based (not thought this through but I think it’s possible)

Option 1 is quite simple in G’MIC, but has downsides - aliasing/loss of quality. What do you think? I suppose if you only ever clone “tile zero” then it’s not an issue.

Edit:
Here’s a quick demo of how you can simply define some custom commands in G’MIC to do this:

_tile_Translate : skip ${1=0},${2=0} # this command actually renders as well
  split_opacity. n. 0,1 j[1] ..,$1,$2,0,0,1,. rm[-2,-1]

tiling :
  v -
  t_w={w} t_h={round(1/2*sqrt(3)*$t_w)} # define constants
  {3*$t_w},{2*$t_h},1,3 # create the background image
  m "_tile_Clone : [$""1]"
  m "_tile_Rotate : rotate. $""1 autocrop."
  m "_tile_Flop : mirror. x"
  m "_tile_Flip : mirror. y"
  _tile_Clone 0 _tile_Translate 0
  _tile_Clone 0 _tile_Flop _tile_Rotate -60 _tile_Translate {[-1/2*$t_w,0]}
  _tile_Clone 0 _tile_Flop _tile_Rotate 60 _tile_Translate {[1/2*$t_w,0]}
  _tile_Clone 0 _tile_Flop _tile_Rotate -60 _tile_Flip _tile_Translate {[$t_w,0]}
  _tile_Clone 0 _tile_Flip _tile_Translate {[3/2*$t_w,0]}
  _tile_Clone 0 _tile_Flop _tile_Rotate 60 _tile_Flip _tile_Translate {[2*$t_w,0]}
  _tile_Clone 0 _tile_Flop _tile_Rotate -60 _tile_Translate {[5/2*$t_w,0]}
  _tile_Clone 0 _tile_Flip _tile_Translate {[0,$t_h]}
  _tile_Clone 0 _tile_Flop _tile_Rotate -60 _tile_Flip _tile_Translate {[-1/2*$t_w,$t_h]}
  _tile_Clone 0 _tile_Flop _tile_Rotate 60 _tile_Flip _tile_Translate {[1/2*$t_w,$t_h]}
  _tile_Clone 0 _tile_Flop _tile_Rotate -60 _tile_Translate {[$t_w,$t_h]}
  _tile_Clone 0 _tile_Translate {[3/2*$t_w,$t_h]}
  _tile_Clone 0 _tile_Flop _tile_Rotate 60 _tile_Translate {[2*$t_w,$t_h]}
  _tile_Clone 0 _tile_Flop _tile_Rotate -60 _tile_Flip _tile_Translate {[5/2*$t_w,$t_h]}
  k[1] v +

The trick here is to define translation as actually drawing the tile. It shouldn’t be too hard to turn this into a filter which accepts a command string, but I find string handling in G’MIC tedious and I ran out of time for now! Seams are clearly visible (I didn’t try to get a better crop) but that can be dealt with in other ways later…

gmic tile0.png tiling

afre · November 25, 2018, 6:48pm

I need a floor with a pattern like this… a much duller version.

gubach · November 26, 2018, 9:58am

Hello garagecoder,

thank you very much for your commitment!

The cropped proto-tile is totally fine. Attention must be drawn to the edges; if there are artifacts here they are multiplied in the resulting tile.

The stripes you mentioned are the edge artifacts that exists if non-rectangle polygon shapes were composed which is mostly the case. They are the main aesthetic problem with image tiling. The are appearing in my Matlab prototype more than in my (unfinished) prototype with PerlMagick&ImageMagick and my main work was to find post-processing methods to correct them. I was not able to find image processing methods that can correct both the heavy black edge artifacts and the more subtile ones until I started using edge masks of such a tiling. In the last days I finished a version that is using a drawn edge mask given by a set of line coordinates and the Matlab function inpaint_nans. This version delivers finally!! a satisfying quality but is only acceptable fast if the simple averaging method is used instead of the PDE-inpaint methods. I will publish in the next days a flicker album with this method to show the subtile advancements compared to the album I linked in my first post.

I am very curious how this can be solved in GMIC; probably with a boundary method because this are gap-artifacts.

Option 1 is quite simple in G’MIC, but has downsides - aliasing/loss of quality… I suppose if you only ever clone “tile zero” then it’s not an issue.

I think it is not an issue if there are no longer processing pipelines with quality disturbing operations are used. If every proto-tile (independent of the number) is directly cloned from the original list instead of reusing an already processed (rotated) proto-tile then the effects are minimized.
Mirroring and translation are non-disturbing matrix operations but rotation can cause massive disturbance dependent on the used interpolation method. “nearest” is very bad but default is mostly bicubic but this causes edge problems (From the imrotate doc: “This interpolation method can produce pixel values outside the original range.”) and I suspect that this is the most part of the edge artifacts in the tiling; the rest might be inexact autotrim because every pixel counts here. Using bilinear is causing a poorer quality of the shape texture so this is not an option for me; it is already an annoyance that there is not the opportunity for user defined interpolation in imrotate as there is in imresize where I defined Mitchel as a better option to bicubic.

Vector field based (manipulate source pixel positions rather than the data itself)

I can only speculate that this option is an analogy to meshgrid processing in Matlab. I find this very interesting!! because I have speculated about the existence a meshgrid representation of a whole! tile: every element in a [2t_h 3t_w] meshgrid points to a pixel in the proto-tile polygon image which means several elements points to the same pixel. The tile image is then generated by an interpolation of the meshgrid and the proto-tile image instead of a sequence of image processing procedures.

In my batch processing context this would be a huge advantage because such a vector/meshgrid representation must be generated only once and it is then applied to all the input images that are processed. At least in Matlab the interpolation is much faster than a longer image processing pipeline which would lead to a huge efficiency boost. Additionally I believe that the interpolation could also be a solution for the edge artifacts. But I did not came far in an implementation of a tile meshgrid representation; perhaps some day a case for a freelancer-com project.

garagecoder · November 26, 2018, 6:35pm

I must keep my answer short for now, but fear not - I’m very aware the edge artifacts must be dealt with, preferably in a way easily replicated in other environments. With G’MIC we’re fortunate to have several inpainting algorithms ranging from very fast to very clever (hence not so fast…), but there are several other options. It needs thought and experiment, which I’ll have time for next week! Again, anyone else is completely free to chip in of course

Edit: almost ready to push the filter (it’s very rough)

garagecoder · December 7, 2018, 12:43am

Filter added, I think only for G’MIC version 2.4.2 (and probably an hour or two before it’s available to download with filter update). Some gotchas:

Only working with one input tile.
Inpaint is rudimentary, don’t try to use it unless the background is full!
I’ve stuck with the design of using “Translate” to actually draw, the format of which is T[x_coord,y_coord].

So the equivalent command list you can paste in for the above:

C0 T0
C0 Fo R-60 T[-1/2*$t_w,0]
C0 Fo R60 T[1/2*$t_w,0]
C0 Fo R-60 Fi T[$t_w,0]
C0 Fi T[3/2*$t_w,0]
C0 Fo R60 Fi T[2*$t_w,0]
C0 Fo R-60 T[5/2*$t_w,0]
C0 Fi T[0,$t_h]
C0 Fo R-60 Fi T[-1/2*$t_w,$t_h]
C0 Fo R60 Fi T[1/2*$t_w,$t_h]
C0 Fo R-60 T[$t_w,$t_h]
C0 T[3/2*$t_w,$t_h]
C0 Fo R60 T[2*$t_w,$t_h]
C0 Fo R-60 Fi T[5/2*$t_w,$t_h]

gubach · December 7, 2018, 10:49am

The text formatting garagecoder used in his last post is also a template for parallel processing: every line is independent and can be processed in parallel.

In the meantime I have worked on the collection of CRMT-lists (and connected mask coordinate lists) of the non-trivial wallpapergroups (= with non-rectangle proto-tiles), so available are now:

p4m, p3m1, p6m, p31m, p3, p6.

What might be a good way to publish this and future lists?

garagecoder · December 7, 2018, 1:07pm

Entirely your choice of course, but I’m wondering about adding some “presets” to the filter (where you choose one from a drop-down list).

@David_Tschumperle more questions I’m afraid!
What’s the best way to test a new version of a GUI filter? Overriding commands works fine at command line - just add to .gmic file. Doesn’t appear to work for GUI (it’s like it’s “locked”).

Is there a way to add multiline text to a GUI multiline text parameter? I can’t seem to figure that out (single line works no problem).

David_Tschumperle · December 7, 2018, 3:17pm

Ah yes, this is a bit tricky, as putting a “\n” doesn’t substitute it by the ASCII code 10.
But the following sample works for me:

#@gui Filter : filter,filter
#@gui : Text = text{1,"Really,that's\na multi-line\ntext !"}
filter :
  to "$1"
  LF={`10`}
  u "{And this can change"${LF}"with multi-lines"${LF}"too !!}"

That’s a good question, the GTK version of the plug-in was giving the priority to the commands defined in the user .gmic file, but it seems it’s not the case anymore. I’ll ask Sébastien why

gubach · December 7, 2018, 5:35pm

some “presets” to the filter

I think that the wallpapergroups are the first choice for such presets like in the old Terrazzo filter because they are basic for all periodic tilings and they are using only one proto-tile. I attach the other CRMT-lists and the masks for further usage (the commands relay on the specific mask shapes and orientations); be aware that they are much less tested than p3m1.

CRMT_p3.txt (174 Bytes)
CRMT_p4m.txt (164 Bytes)
CRMT_p6.txt (375 Bytes)
CRMT_p6m.txt (772 Bytes)
CRMT_p31m.txt (505 Bytes)
p3_mask.txt (122 Bytes)
p4m_mask.txt (59 Bytes)
p6_mask.txt (122 Bytes)
p6m_mask.txt (67 Bytes)
p6_mask.txt (122 Bytes)

garagecoder · December 7, 2018, 6:41pm

Should be able to include those, probably compress quite well.

I was thinking about vector maps, it turns out very simple for a single tile (edge aliasing issues aside). In G’MIC all you need is a vector valued tile as the input (each vector at x,y holds it’s own position) - very trivial to produce:

gmic tile0.png f ":i3>0?[x,y,z,i3]:[0,0,0,0]" o vectortile.tiff,float

Then when you create a completed tiled image as usual, what you have is a vector map. Now you can take any tile of the same dimensions and do:

gmic vectormap.tiff tile0.png +warp. ..,0

warptile

The top part is the “warp map”

For more than one input tile it becomes more tricky, you’d probably need to save an extra channel in the map specifying the source tile (amongst other things).

Edit: what I mean about using a warp map is you don’t even need the opacity mask, only that the whole tile dimensions match. Example:

tiledcats
tiledwall

afre · December 8, 2018, 4:23am

Been thinking about the aliasing issue. If you are going by the upside down triangle, only 2 edges are aliased. Couldn’t you simply make the edges fit pixel perfectly?

garagecoder · December 8, 2018, 9:57am

Hmm I suppose the problem is lack of information. If you know it’s a triangle, you can use that fact to rotate by vertices and match them exactly. The problem is the filter is only given a raster image, so there’s no concept of shape (without getting into transforms). When you begin using rotations less than 90 degrees, edge mismatch is probably inevitable with a raster image. I might look at testing worst cases to see what reduces that error though

Reptorian · December 8, 2018, 2:22pm

Hmm, I’m interested into the filter as I am a fan of tileable hex-art. That’s all I have to say here.

Ooh, I have a idea, each triangle can go into a separate layer, right? So that I can adjust these triangle individually?

gubach · December 8, 2018, 2:50pm

vector maps

VERY INTERESTING!!! That is probably near to the idea of a meshgrid representation of tilings in Matlab I wrote about on Nov. 25 and something that is useful in my batch processing context.

What about the expected run time of such a representation compared to an original CRMT-pipeline? The run time of a CRMT-pipeline is proportional to the number of proto-tiles that are distributed over the background (14 with p3m1, 28 with p6m, …) and the background area but a grid representation is independent of the number of proto-tiles and only dependent on the number of output pixels (and the interpolation method).
If I am comparing the interpolation of conformal maps to a p3m1-CRMT-pipeline with a similar image area I estimate a 2-3x faster image generation. But perhaps the expected problems in the majority of cases with more than one type of proto-tile (C1, C2,…) could make this efficiency advantage disappear.

Interesting to see that in your first image example some edge artifacts vanishes, some were reduced or transformed but some remain unchanged. So the vector method have active influence on the edge artifacts which could lead to the hypothesis that there exists parameters of the warp/interpolation function (like interpolation method) that could also solve the edge problems (two birds with one stone).