Decompose an image into a set of patches

[David_Tschumperle (G'MIC staff)]

Hello.
Today, I’ve implemented and added two new commands in the G’MIC standard library
(namely img2patches and patches2img), which I find very interesting.
These two commands work hand in hand. They are designed to manipulate image patches as separate entities.
With this post, I’d like to give some information about them. So, let’s start!

---

• First, the command img2patches :

When calling img2patches, you end up with a volumetric image with size patch\_size \times patch\_size \times nb\_patchs \times nb\_channels that contains all the (possibly overlapping) patches from your input image.
Like :

$ gmic sp colorful,256 +img2patches 32

gives this:

(default arguments make the patch non-overlapped).

You get the idea: it returns a volumetric image where each slice is one image patch. A kind of “patch representation” of your initial image.

---

• Command patches2img: it simply reconstructs the image from its set of patches.

So that:

$ gmic sp colorful,256 +img2patches 32 +patches2img. 256,256

will reconstruct the original image from its patch representation:

Think of it as a (reversible) patch-transform of an image (I think it’s also called patch embedding in the ML literature).

I won’t enter in all the option details, but basically you can extract overlapping patches from images, and reconstruct images from them, even with a choice for the local reconstruction weights in case of overlapping patches (weights can be uniform or gaussian, for the moment).
When possible, the extraction / reconstruction steps even use multi-threading to speed up the whole process.

---

• Now the interesting part: of course, if you manipulate the patch representation before the reconstruction of the full image, you can easily get funny/interesting effects.
  Let me show you some some examples:

---

1. Mirror the patches:

$ gmic sp colorful,256 w,h:=w,h img2patches 16 mirror xy patches2img $w,$h

(here you see that no overlapping is used).

---

2. Shuffling the patches, but with a smooth image reconstruction, using overlapping patches and gaussian weights:

$ gmic sp colorful,256 img2patches 32,16  s z sort_list +,u a z patches2img $w,$h,16,30%

I guess this could be interesting to do this on more regular textures.

---

3. Randomly rotate the patches: Here again, with a smooth reconstruction.

$ gmic sp colorful,256 w,h:=w,h img2patches 16,6 s z foreach { rotate {90*round(u(3))} } a z patches2img $w,$h,6,5

---

4. Normalize patch values: A very simple way to apply local normalization to lightness/colors to create vivid colors.

$ gmic sp colorful,256 w,h:=w,h +img2patches 16,5 l. { s z foreach { n 0,255 } a z patches2img $w,$h,5,1 }

---

5. Local patch quantization:

$ gmic sp colorful,256 w,h:=w,h img2patches 32,16 s z quantize 2 n 0,255 a z patches2img $w,$h,16,10

Here, I’ve quantized each patch with 2 colors only. But these two colors are of course chosen differently for each patch. As a result, the reconstructed image (done with patch overlapping) is quite smooth and locally adapts to reproduces the colors of the original image.

Now, if I try with more quantization levels (e.g. 5 instead of 2), I get this:

I think this kind of local quantization was something @Reptorian has been interested in, a few weeks ago. With these new commands, it becomes really simple to deal with it.

---

That’s all for now. As you see, these two commands already allow to quickly build different image filters. If you have more ideas to test, do not hesitate to share them with us.

These commands have been pushed to the G’MIC stdlib, so they are already available after a $ gmic update.
Happy patching!

[David_Tschumperle (G'MIC staff)]

5. Patch masking:

$ gmic sp dog w,h:=w,h img2patches 64 200%,200% circle. 50%,50%,33%,1,1 r2dx. 50% * patches2img $w,$h

renders this:

Easy mode

[David_Tschumperle (G'MIC staff)]

6. Same, with random rotations added:

$ gmic sp dog  w,h:=w,h img2patches 64,0,3 s z foreach { rotate {u(-120,120)},1,3,50%,50%  } a z 200%,200% circle. 50%,50%,33%,1,1 r2dx. 50% * patches2img $w,$h

renders this:

[Reptorian]

I would like to do M*N patches rather than M. Can this command allow for that?

[garagecoder]

This will simplify testing some things, thanks!

An interesting thing to try if you have time: assemble patches in gradient domain, then use ilaplacian. I did that not too long ago, maybe I’ll find an example and show here…

[garagecoder]

OK, very hastily hacked together but it will do. First define a couple commands:

gcd_fill_patches_all : skip ${1=8}
    T=$! [0],[0],[0],2 noise_poissondisk. $1
    f. "begin(const K=$1*2+1);i>0?[u(w#0-K),u(h#0-K)]"
    round. 1 => ref
    repeat $T {
    +f[$>] 0
    eval[ref] "begin(const K=$1*2+1; const S=$1);
      if(i>0,
        P=I; U=P[0]; V=P[1];
        ref(crop(#"$>",U,V,K,K),patch);
        draw(#-1,patch,x-S,y-S,0,0,K,K,1,s#0,1);
      );"
    }

gcd_addv :
  pass$1 0
  repeat $!-1 l[$>,-1]
    repeat s#0 sh[0] $> add. {i[#1,$>]} rm. done
  done done rm.

Then use the commands in gradient domain. For some reason I coded it to go gradient → integral → reconstructed, rather than gradient → laplacian → reconstructed. I can’t remember why, sorry!

gmic sp colorful +r 1,1,1,100%,2 l[0] g xy,1,2 gcd_fill_patches_all 64 ilaplacian[-2,-1] 0 g.. x,-1,2 g. y,-1,2 add[-2,-1] } gcd_addv.. . k.. c 0,255

Edit: if you want to go from laplacian, it’s a simple tweak anyway:

gmic sp colorful +r 1,1,1,100%,2 l[0] { g xy,1,2 gcd_fill_patches_all 64 g.. x,-1,2 g. y,-1,2 add[-2,-1] ilaplacian. 0 } gcd_addv.. . k.. c 0,255

[samj]

@David_Tschumperle

I really like these 2 new commands and the speed of execution.
Would it be possible to have a GUI interface for Gimp, etc.

Thanks :o)

[David_Tschumperle (G'MIC staff)]

@garagecoder , neat idea ! I’ve got some prototype working with Poisson reconstruction of shuffled patches.
@samj , as a result, I’ll probably add a new filter Shuffle Patches soon in the plug-in

[David_Tschumperle (G'MIC staff)]

Here it is, new filter Arrays & Tiles / Shuffle Patches :

[garagecoder]

Damn, right as I have to go out. Looking forward to trying later!

[David_Tschumperle (G'MIC staff)]

Filter applied on a video:

New G'MIC filter "Shuffle Patches" applied on a video

[samj]

@David_Tschumperle

Thank you for this new filter

---

With my previous message I was thinking of this sequence for GUI :
img2patches >> [Filters] >> patches2img >> [Filters for the recomposed image]

A little similar to this image :

[alosca]

This is a great addition, thank you! I have been using split_tiles to create patches and montage to reassemble them after processing using apply_parallel, but the overlap option was missing and this really comes handy!