Shedding light on all Inpaint (patch-based) parameters?

Hi,
I really like and often use the “Patch-based” inpainting filter on the Qt interface of G’mic.
I know this filter is becoming a bit dated and slow compared to others (like multiscale or Gimp’s resynthesizer) but to me it still has great value in quality of results and compatibility with applications.

However, knowing truly how its parameters work would prevent a lot of blind trial and error every time I use this filter on my photos.

I’ve looked everywhere I can think of (documentation pages, Pat David’s tutorial, this forum and others) but couldn’t get a complete understanding of what the sliders and their value really mean and why they are clamped at the values they are. Namely:

  • All “size” parameters: what is their unit? Pixels? Percent? Why do they clamp at their respective value?

  • Lookup factor: @afre said here it might be :

    how much the lookup area influences the structure of the inpainting.

    That is very unclear to me. Isn’t the inpainting using only parts of the lookup area anyway? And what does “structure” mean here?

  • Blend threshold: Very obscure to me. I guess based on the input value, it may stop applying a blend in some areas of the picture where… something is happening to a certain degree. Yay!

  • Blend decay: Wait. I thought the blend size would already define its decay. Are we thinking of the same kind of decay? Also why does it clamp at 0.5?

  • Blend Scales: Are we talking “scales” as in “sizes” or as in “structure of patches” (i.e like the scales on a chameleon :wink:)? And what does the value really mean anyway?

  • Mask dilation: What in this process contains a mask? Is it the patches? If so, I assume that if you dilate, each patch grows bigger or shrinks? So then which patch dominates a neighbor patch is up to the order in which they are pasted? I’m purely guessing from the name here.

Thanks in advance for your help.

Better ask @David_Tschumperle for clarification. @grosgood is also good at explaining.

@ChameleonScales
Yes indeed. I`splain a lot around here, in the tradition of The Celebrated Jumping Frog of Calaveras County. So if it is a ‘Leonidas Smiley’ you are asking around for, I’ll gladly tell you about ‘Jim.’ Now, what is it you want to know?

On a more serious note, I see you have @patdavid 2014 blog post on it; that’s the best going at the moment.

I do gmic command tutorials, not the plug-ins exactly, but there are pretty close parallels between the two. And since I do these tutorials in the order that pleases me, let’s put inpaint at the top of the tall, teetering pile. Stay tuned. Not tomorrow, but in the sweet fullness of time. Like a week, maybe. Not more than a month. Got a day job; it pays the rent.

1 Like

Sorry, I’m a bit busy at the moment, working to releasing v. 2.9.7, but maybe this could help.
Some general things to know:

  • What is called “the mask” is the set of pixels to fill-in.
  • A mathematical description of the algorithm is available in this research paper (click on the paper title to download the pdf):

Exemplar-Based Inpainting: Technical Review and New Heuristics for Better Geometric Reconstructions.
(P. Buyssens, M. Daisy, D. Tschumperlé , O. Lézoray).
IEEE Transactions on Image Processing, Vol. 24, No. 6, pp 1809–1824, June 2015.

It doesn’t necessarily use the same terms than the filter in the G’MIC-Qt plug-in, but this describes the inpainting algorithm in details.
Now, for the answers to your questions:

  • The patch size is specified in pixels. A value of 7 (default) means the region is reconstructed by copy/pasting 7x7 patches from the mask edges to the interior.
  • The Lookup size is specified in pixels as well. That’s the maximal size of the neighborhood used to search similar patches for the reconstruction.
  • Lookup factor is a multiplier applied to the lookup size that can shrink or grow the lookup neighborhood. It’s actually a bit strange not to have included this directly in the lookup size parameter. I can’t remember why I did it that way TBH.
  • Blend size is specified as a multiplier of the patch size. So, 1.2 by default means the blend size in pixels is actually (int)(7x1.2) = 8.

About the lookup parameters : it’s actually a bit hard to explain in a few lines how the lookup neighborhood (i.e. the image areas scanned by the algorithm to find patch similarity) works. If you have time, please read the research paper, where this is explained.

Actually, same problem with the blend parameters. The blending algorithm is really adaptive and tries to first analyze where the copy/pasting of the patches have succeeded or failed, and then apply blending, but only where there is a high risk of errors.
The threshold value set how this “error detection” must be achieved (i.e. detect potential errors everywhere and blend everything → blend threshold = 0, or blend only on “high” errors detected → blend_threshold = 1).

The decay is different from the size. It tells about the blending curve used (how much the blending decreases when going far from the detected reconstruction error).

We are talking about “quantization” of the blending. Theoretically, we should be able to locally blend patches with a size and profile that are given by continuous functions. But for the sake of computation speed, we enable an optimization of the blending by quantizing the blending map. The more blending scales you set, the more blending precision you get (but it’s slower).

The mask is the set of pixels to reconstruct. In practice, when a user defines a mask, it is often narrower than it should be. This parameter allows to dilate the mask a little bit (specified in px) before inpainting.

3 Likes

Thank you so very much, this changes everything!
I’ll try to find a moment to read your awesome paper.
Right off the bat though I notice something that seems a little strange to me:
So the lookup factor is a multiplier to the lookup size, but the default values in Qt are:
Lookup size : 16
Lookup factor : 0.1
so the actual lookup size would be 1.6 pixel, which means the algorithm would seek in an area that is almost directly connected to the mask?
I guess it’s not problematic but it seems counter-intuitive.

Anyway, this definitely shed light on this filter for me. Thanks again and thanks for developing the next version.

Not really because:

  • There is a minimal fixed size of 5 pixels any way :slight_smile:
  • The lookup area is not necessarily located near the mask. It is in fact located near the locations fo the latest ‘good matches’ that have been found to reconstruct the image data at previous iterations.

:thinking: interesting and good to know. Thanks again

I noticed some discrepancies between the behavior of some values in the Qt (GIMP) version and that of the command line version.
For example it seems that while the blend size is as you said a multiple of the patch size in Qt, it seems to be an independent number (in pixels) when used in the command line. So if the lookup size is 16, you’d have to set the blend size to 16 in the command line for it to correspond to a value of 1 in Qt. This seems to be confirmed by the fact that command-line gmic shows the blend size as a rounded integer when you input a float, whereas the Qt slider allows float numbers.

However, there definitely are other differences that I didn’t figure out because my results are still very different. Not to mention that there is a “lookup increment” parameter in the command line that doesn’t appear in Qt and I don’t understand what this one does. Is it a number of iterations?

Here’s a comparison example:

Original image:
original+mask

Taking into account that behavior difference for the blend size, I used these parameters in Qt:
Qt-screen

and this command in the terminal (I had the mask in a separate png file):
gmic "./original.JPG" "./mask.png" -inpaint[0] [1],12,16,1,1,24,0,0.05,10,1 remove[1] -output "out_command.jpg"

And here’s what I got:

From the Qt interface:
out_gimp

From the command line:
out_command

I’d love to have a better understanding of this so that I can batch process images.

The link in this screenshot is outdated and brings us to a 404 (G’MIC Inpainting tutorial on Patrick David’s blog)
I did found the new page, though
https://patdavid.net/2014/02/getting-around-in-gimp-gmic-inpainting-content-aware-fill/
if someone can update it :wink:

@David_Tschumperle can fix it.

1 Like

Let me clarify about the Qt plugin vs command line question. The plugin takes the preview data from the host app (e.g., GIMP), which may or may not be a 100% representation of the image(s) themselves. The preview itself is rendered differently than the command line display. Both aren’t 1 to 1 to the actual output; CLI is much closer, so I tend to go with it.

My advice is to view the output in the final medium; e.g., if you intend on publishing to the web, then view it in a browser like Firefox with colour management enabled. G’MIC doesn’t do colour management, so not only are its previews not managed but CLI won’t respect the ICC profiles or transforms. This can be a headache to reconcile. My solution is to learn about colour and what G’MIC can do for you with that in mind.

Lastly, specifically regarding the patched based inpaint filter, I recall there being a mislabelling/misnaming of terms between the CLI (see help) and GUI versions, which can lead to some confusion. I brought this up a long time ago but @David_Tschumperle hasn’t made the adjustment.

I think you should compare the qt gui file with the commands. Using gmic cmd, type e.g. for fx_inpaint_patch

gmic e $$fx_inpaint_patch

repeat $! l[$>] if $14 bs={max(16,min(w,h)*arg(1+$14,100,75,50,25,10,5,2,1)%)} at "_fx_inpaint_patch $*",$bs,$bs,1,25%,25%,0,2 else _fx_inpaint_patch $* fi endl done

gmic e $$_fx_inpaint_patch

repeat $! l[$>] R=$9 G=$10 B=$11 A=$12 if !$A" && "(s==2" || "s==4) split_opacity +!=. 0 *[0,-1] a c R=0 G=0 B=0 fi +round select_color. 0,{round([$R,$G,$B,$A])} if $13 dilate. {1+2*$13} fi inpaint.. [1],$1,{$1*$2},$3,1,{$4*$1},${5-8} rm. endl done

You might see there some intermediate steps and some special handling of parameters! If there are commands you don’t understand do “gmic h <command>”.

Perhaps even better to use the fx_… commands from gmic-qt directly instead of reinventing the wheel by using the gmic base commands. You might get them from the console with verbose output!

You may have Been Here, Done This, already, because in your Post #8 you are invoking -inpaint directly on a command line, so I hazard to guess that you are already aware that the parameters gathered by the pretty Inpaint [Patch-Based] GUI are not quite those that become arguments to -inpaint itself, that -fx_inpaint_patch is the first G’MIC command that runs to process the settings provided by the pretty GUI and some argument transformations take place.

That said, it may not be abundantly clear what happens to the GUI settings as they go on their merry way to -inpaint arguments, by way of -fx_inpaint_patch. For that, you need to know where the definition of -fx_inpaint_patch resides and what set-up it does for -inpaint. There are a few ways to do that:

  1. @KaRo has introduced you to the double-dollar ($$) substitution rule, quite new, which expands gmic custom commands so that -echo can print their implementation in the command shell.
    a. The $$ substitution typically references the *.gmic download file. You could peruse that source yourself. It arrives at your system when you click on the update button in the gmic-qt dialog box or execute gmic -update command in a shell. For Windows 10, G’MIC puts this file in %AppData%\Roaming\gmic and is named updatexxx.gmic, where xxx may be 297 - current production - or 298 current development. You can determine which is which by looking at the window banner of the gmic-qt dialog box; that cites which gmic version the plug-in runs with. %AppData% is a Windows system environmental variable. Type %AppData% in your Windows 10 search bar to find the specific folder location on your system.
    b. Those with Unix-like systems may find updatexxx.gmic at ${HOME}/.config/gmic/updatexxx.gmic
  2. This download file may be perused with any text editor but it is not an easy read, as the file is generated from a number of primary sources, white space and comments have been filtered out and there is little in the way of indentation. Still, use your favorite text editor to search for fx_inpaint_patch and mark well its location for later study.
  3. Those of us who build from sources or are otherwise terminally curious about gmic obscurities fork the gmic git repository. There, fx_inpaint_patch may be found in gmic/src/gmic_stdlib.gmic around line 45198, as of commit 727952fef08a, (Fri Jul 2 13:16:51 2021 +0200), master development branch. This is one of the primary source files that is comment- and white-space stripped, then incorporated into updatexxx.gmic.

I’ve gone on at length on locating G’MIC commands, and -fx_inpaint_patch in particular, because through Adding Custom Commands, any command can be studied in detail and modified to observe effects. The Bouncing Ball Tutorial walks through the process of extending G’MIC with a new command; you start from a blank slate, as it were. That technique broadly points the way toward taking any existing G’MIC custom command and rewriting it as if it was new. It becomes your own personal version of the command.

There are two ways to go about doing this.

  1. You may add commands to your local G’MIC extensions, These are contained in the file %AppData%\user.gmic (Windows) or ${HOME}/.gmic (Unix-like).
  2. Create any text file with a .gmic extension — myradioactivecommands.gmic perhaps — and extend your command set on-the-fly by way of -command (m for short):
gmic -m myradioactivecommands.gmic ...

In either environment, the local extensions file is a text file composed by the rules at Adding Custom Commands.

To illustrate, here is one way to make a local, personal version of -fx_inpaint_patch for purposes of understanding how it sets up -inpaint
I copied the current definitions of fx_inpaint_patch and its helper -_fx_inpaint_patch to another command file, gtutor_ipaintpatch.gmic, expanded
command names to their full spellings, re-indented to clarify program structure and added comments.

gtutor_inpaint_patch
# Implementation. Don't call this.
_gtutor_inpaint_patch :
   -repeat $!
     -local[$>]
        R=$9
        G=$10
        B=$11
        A=$12               -d
        -if !$A" && "(s==2" || "s==4)
           -split_opacity   -d
   	   +neq. 0          -d
   	   -mul[0,-1]       -d
   	   -append c        -d
   	   R=0
   	   G=0
   	   B=0
        -fi # Purely transparent color.
        +round              -d  
        -select_color. 0,{round([$R,$G,$B,$A])} -d
        -if $13                                 
           -dilate. {1+2*$13}                   -d 
        -fi  -d
        -inpaint.. [1],$1,{$1*$2},$3,1,{$4*$1},${5-8}
        -remove. -d
     -endl
   -done

# Work-alike of fx_inpaint_patch. Call this like:
#
# gtutor_inpaint_patch 12,16,1.0,2.0,0.0,0.05,10.0,1,255,0,0,255,0,0
#
# $1:  Patch Size = _int(7,1,64)
# $2:  Lookup Size = _float(16,1,32)
# $3:  Lookup Factor = _float(0.1,0,1)
# $4:  Blend Size = _float(1.2,0,5)
# $5:  Blend Threshold = _float(0,0,1)
# $6:  Blend Decay = _float(0.05,0,0.5)  
# $7:  Blend Scales = _int(10,1,20)
# $8:  Allow Outer Blending = _bool(1)
# $9:  Mask Color = R - 255
# $10: Mask Color = G - 0
# $11: Mask Color = B - 0
# $12: Mask Color = A - 255
# $13: Mask Dilation = _int(0,0,32)
# $14: Process by Blocs of Size 100%,75%,50%,25%,10%,5%,2%,1%
gtutor_inpaint_patch :
   -repeat $!
       -local[$>]
          -if $14
             bs={max(16,min(w,h)*arg(1+$14,100,75,50,25,10,5,2,1)%)}
             -apply_tiles "_fx_inpaint_patch $*",$bs,$bs,1,25%,25%,0,2
          -else
             _gtutor_inpaint_patch $* 
          -fi
       -endl
  -done

This suffices in your shell to run the modified command:

gmic -v 3 -m gtutor_ipaintpatch.gmic -input im_damaged.png -gtutor_inpaint_patch. 12,16,1.0,2.0,0.0,0.05,10.0,1,255,0,0,255,0,0

This pipeline up-ticks verbosity, (-v 3), introduces the modified commands (-m), puts a red-splotched image on the command line (-input im_damaged.png) and then runs the modified command (-gtutor_inpaint_patch. 12,16,1.0,2.0,0.0,0.05,10.0,1,255,0,0,255,0,0).

You may notice sprinked throughout this modified command: -d. This is the shorthand form of -display, which can be repurposed as a fair dinkum breakpoint for stepping through G’MIC scripts. When G’MIC hits -d, it dumps details about the images on the list to the shell standard output as well as displaying the image list, somewhat inaccurately, but to be as illustrative as possible. You may add and remove -d as you like, as well as modify parameters and what the script actually does. I think anyone here who spelunks through G’MIC commands uses one or another version of this technique and are welcome to chime in with their own tips.

You may observe at some point that, as @KaRo points out, the GUI parameters are altered on their way to -ipaint. No surprise there, but now you have the means to precisely see how those alterations take place.

  1. The Patch Size and Lookup Size from the GUI are multiplied together to become -inpaint’s lookup_size parameter.
  2. The GUI’s Patch Size, unmultiplied, serves directly as -inpaint's patch_size
  3. inpaint's lookup_factor receives unaltered the GUI’s Lookup Factor
  4. inpaint's lookup_increment is not available in the GUI: it has been fixed at 1.
  5. The GUI’s Blend Size and Patch Size multiply together to provide inpaint's blend_size.
  6. The next four parameters fall into a one-to-one correspondence:
    a. GUI: Blend Thresholdinpaint: blend_threshold
    b. GUI: Blend Decayinpaint: blend_decay
    c. GUI: Blend Scalesinpaint: blend_scales
    d. GUI: Allow Outer Blendinginpaint: is_blend_outer

Other GUI parameters are not meant for in_paint directly, but establish characteristics of the selection mask that it is given and the wrapper command uses the parameters instead. Thus, the GUI’s color swatch becomes an argument to -select_color, responsible for generating the block-out mask, ipaint's initial parameter, and GUI’s Mask Dilation parameter dilates that mask.

With these connections, you can project the GUI’s settings onto what ipaint expects, and, by extension, take @David_Tschumperle 's-ipaint-centric notes in Post 4 and project them back to the GUI.

For many reasons, there are plug-in authors whose G’MIC coding skills are orders of magnitude beyond their English documentation abilities — let it be. English is not their native language. My course in G’MIC documentation resolutely remains with the core, because upon that ground is where all gmic-qt plug-ins ultimately sit. Except for an ocasional one I might develop, I don’t do plug-in documentation. I fear that means for many gmic-qt plug-ins there will be no orchestrated documentation effort for some time — perhaps never.

Alas, @ChameleonScales, you and people like you will have to embark on these spelunking expeditions from time to time to see how gmic-qt plug-ins connect to the core, which is rarely in a 1:1 fashion. If this seems a dauntingly long post, I apologize; perhaps it will be the basis of a Cookbook article at some point and become rather less like raw notes. In the mean time, I hope it gives you enough pointers in where to look for things in the code base and how to recover the connections between gmic-qt GUI’s and the core G’MIC language.

3 Likes

You can also use command return. I sometimes use return to diagnose my own command, and see how it match my theory on how the command should go, and then correct my command.

Exclusive GUI command developers would have to stick with display. Personally, I use CLI and return.

1 Like

Thank you very much @grosgood for your detailed explanation/tutorial and everyone else as well for your help.
Indeed, following the rules @grosgood layed down, I could get a 100% identical result between Qt and the -inpaint command. I might make a custom command that behaves like Qt but takes 2 images as input (the one to inpaint and the mask) instead of selecting by color, but at least I understand how it works much better now.

I hope that future development of filters won’t obscure the meaning of sliders as much, as artists are used to work with clarified types of numbers, whether they are pixels, percents, multipliers or what have you. An informative interface helps tremendously in getting quickly to the desired result, even if we don’t necessarily have a complete knowledge of the inner workings of the algorithm that produces that result.

But hey, most important is to have great people who code powerful features and other great people who can help explain them if needed. And that’s a double check on that!

I still don’t know what the Lookup Increment does. Do you know?

My take, pondering gmic.cpp and the paper.
lookup increment: Speed, in pixels, by which to shift a search window to a new candidate patch. See the technical reference in @David_Tschumperle March 24 post 4 in this thread. In particular, consider Figure 2c. Searching takes place in a rather large lookup area (green box); the algorithm shifts a search window (small blue box) and tests the candidate patch for the likelihood of being a good match for the target area (small red box). If the likelihood is poor, the algorithm shifts the search area by an amount controlled by the lookup increment. One may set a higher lookup increment (default is one pixel) to speed up the search rate, but this comes at an increased likelihood of “skipping over” a good candidate.

1 Like