Shedding light on all Inpaint (patch-based) parameters?

Hi,
I really like and often use the “Patch-based” inpainting filter on the Qt interface of G’mic.
I know this filter is becoming a bit dated and slow compared to others (like multiscale or Gimp’s resynthesizer) but to me it still has great value in quality of results and compatibility with applications.

However, knowing truly how its parameters work would prevent a lot of blind trial and error every time I use this filter on my photos.

I’ve looked everywhere I can think of (documentation pages, Pat David’s tutorial, this forum and others) but couldn’t get a complete understanding of what the sliders and their value really mean and why they are clamped at the values they are. Namely:

  • All “size” parameters: what is their unit? Pixels? Percent? Why do they clamp at their respective value?

  • Lookup factor: @afre said here it might be :

    how much the lookup area influences the structure of the inpainting.

    That is very unclear to me. Isn’t the inpainting using only parts of the lookup area anyway? And what does “structure” mean here?

  • Blend threshold: Very obscure to me. I guess based on the input value, it may stop applying a blend in some areas of the picture where… something is happening to a certain degree. Yay!

  • Blend decay: Wait. I thought the blend size would already define its decay. Are we thinking of the same kind of decay? Also why does it clamp at 0.5?

  • Blend Scales: Are we talking “scales” as in “sizes” or as in “structure of patches” (i.e like the scales on a chameleon :wink:)? And what does the value really mean anyway?

  • Mask dilation: What in this process contains a mask? Is it the patches? If so, I assume that if you dilate, each patch grows bigger or shrinks? So then which patch dominates a neighbor patch is up to the order in which they are pasted? I’m purely guessing from the name here.

Thanks in advance for your help.

Better ask @David_Tschumperle for clarification. @grosgood is also good at explaining.

@ChameleonScales
Yes indeed. I`splain a lot around here, in the tradition of The Celebrated Jumping Frog of Calaveras County. So if it is a ‘Leonidas Smiley’ you are asking around for, I’ll gladly tell you about ‘Jim.’ Now, what is it you want to know?

On a more serious note, I see you have @patdavid 2014 blog post on it; that’s the best going at the moment.

I do gmic command tutorials, not the plug-ins exactly, but there are pretty close parallels between the two. And since I do these tutorials in the order that pleases me, let’s put inpaint at the top of the tall, teetering pile. Stay tuned. Not tomorrow, but in the sweet fullness of time. Like a week, maybe. Not more than a month. Got a day job; it pays the rent.

1 Like

Sorry, I’m a bit busy at the moment, working to releasing v. 2.9.7, but maybe this could help.
Some general things to know:

  • What is called “the mask” is the set of pixels to fill-in.
  • A mathematical description of the algorithm is available in this research paper (click on the paper title to download the pdf):

Exemplar-Based Inpainting: Technical Review and New Heuristics for Better Geometric Reconstructions.
(P. Buyssens, M. Daisy, D. Tschumperlé , O. Lézoray).
IEEE Transactions on Image Processing, Vol. 24, No. 6, pp 1809–1824, June 2015.

It doesn’t necessarily use the same terms than the filter in the G’MIC-Qt plug-in, but this describes the inpainting algorithm in details.
Now, for the answers to your questions:

  • The patch size is specified in pixels. A value of 7 (default) means the region is reconstructed by copy/pasting 7x7 patches from the mask edges to the interior.
  • The Lookup size is specified in pixels as well. That’s the maximal size of the neighborhood used to search similar patches for the reconstruction.
  • Lookup factor is a multiplier applied to the lookup size that can shrink or grow the lookup neighborhood. It’s actually a bit strange not to have included this directly in the lookup size parameter. I can’t remember why I did it that way TBH.
  • Blend size is specified as a multiplier of the patch size. So, 1.2 by default means the blend size in pixels is actually (int)(7x1.2) = 8.

About the lookup parameters : it’s actually a bit hard to explain in a few lines how the lookup neighborhood (i.e. the image areas scanned by the algorithm to find patch similarity) works. If you have time, please read the research paper, where this is explained.

Actually, same problem with the blend parameters. The blending algorithm is really adaptive and tries to first analyze where the copy/pasting of the patches have succeeded or failed, and then apply blending, but only where there is a high risk of errors.
The threshold value set how this “error detection” must be achieved (i.e. detect potential errors everywhere and blend everything -> blend threshold = 0, or blend only on “high” errors detected -> blend_threshold = 1).

The decay is different from the size. It tells about the blending curve used (how much the blending decreases when going far from the detected reconstruction error).

We are talking about “quantization” of the blending. Theoretically, we should be able to locally blend patches with a size and profile that are given by continuous functions. But for the sake of computation speed, we enable an optimization of the blending by quantizing the blending map. The more blending scales you set, the more blending precision you get (but it’s slower).

The mask is the set of pixels to reconstruct. In practice, when a user defines a mask, it is often narrower than it should be. This parameter allows to dilate the mask a little bit (specified in px) before inpainting.

3 Likes

Thank you so very much, this changes everything!
I’ll try to find a moment to read your awesome paper.
Right off the bat though I notice something that seems a little strange to me:
So the lookup factor is a multiplier to the lookup size, but the default values in Qt are:
Lookup size : 16
Lookup factor : 0.1
so the actual lookup size would be 1.6 pixel, which means the algorithm would seek in an area that is almost directly connected to the mask?
I guess it’s not problematic but it seems counter-intuitive.

Anyway, this definitely shed light on this filter for me. Thanks again and thanks for developing the next version.

Not really because:

  • There is a minimal fixed size of 5 pixels any way :slight_smile:
  • The lookup area is not necessarily located near the mask. It is in fact located near the locations fo the latest ‘good matches’ that have been found to reconstruct the image data at previous iterations.

:thinking: interesting and good to know. Thanks again