Scripted foreground extraction

johndoe46 · September 3, 2019, 1:04pm

Hello,

I’ve tried this tutorial and I think it gives great results:

Compared to grabcut, I think it’s easier to set up a few points than to draw lines and curves. I would like to integrate this with another software. Provided I know foreground and background points coordinates, is there a way to script the extraction (using gmic cli, not gimp) ?

Thanks for your kind help

David_Tschumperle · September 3, 2019, 1:36pm

Yes, that’s possible. Basically G’MIC uses a simple watershed technique with a specific potential map, nothing really fancy.
So the main difficulty here is to provide an image where non-zero pixels are your labeled points (e.g. 1=background and 2=foreground). Once you have this, do something as:

$ gmic input.jpg labeled_points.png +_x_segment[0] +watershed[1] [-1]

and you’ll get the full labeled result as the last image.
The command _x_segment is the one used by the G’MIC filter to estimate the potential map from the input image.

johndoe46 · September 3, 2019, 2:06pm

Neat ! Looks like it won’t be too much work to integrate. I’ll try this and get back with my results.

Very clear explanations, thank you.

johndoe46 · September 4, 2019, 4:58pm

Hello,

I’m getting great results with fewer interactions and very reasonable time given the image size.

I could probably squeeze a bit more time by calling the C lib ?

David_Tschumperle · September 4, 2019, 6:52pm

Yes, I would say the C++ lib is maybe easier to use : https://gmic.eu/libgmic.shtml
Just feed gmic() with a list of two input images, and use the same command as the command string.

johndoe46 · September 5, 2019, 6:37am

Our current stack is python/numpy-based, so calling C++ libs is more involving than calling C libs. It should be possible though, I’ll give it a try.

David_Tschumperle · September 5, 2019, 8:07am

There is also a C API for the G’MIC library, which works basically the same.
Look at these sources for more info and examples of use : gmic-community/libcgmic at master · dtschump/gmic-community · GitHub

afre · September 5, 2019, 1:50pm

Ho-hum: maybe I should grit my teeth and learn C(++) some day so that my scripts aren’t so darn slow.

johndoe46 · September 15, 2019, 7:06pm

Sorry I didn’t update this thread. Here’s what I did to try to interface gmic with python:

compile gmic_libc
generate bindings to gmic_libc.h with ctypesgen (GitHub - davidjamesca/ctypesgen: Pure-python wrapper generator for ctypes). This automatically converts the structs and the functions to a python module, linked to the .so file.
test calling gmic from python. Example, unrelated to my extraction problem:

import gmic
from ctypes import POINTER, c_bool, c_float, c_uint

E_FORMAT_FLOAT = 0
E_FORMAT_BYTE = 1

p_bool = POINTER(c_bool)
p_float = POINTER(c_float)

options = gmic.gmic_interface_options()
options.ignore_stdlib = False
options.p_is_abort = p_bool(c_bool(False))
options.p_progress = p_float(c_float(0.0))
options.interleave_output = False
options.no_inplace_processing = True
options.output_format = E_FORMAT_FLOAT

image = gmic.gmic_interface_image()
image.name = “test”.encode(“utf-8”)
image.width = 500
image.height = 500
image.spectrum = 4
image.depth = 1
image.is_interleaved = False
image.format = E_FORMAT_FLOAT

command = ‘v 0 apply_channels “div 2”,rgba_r polaroid 5,30 rotate 20 drop_shadow , drgba display’

gmic.gmic_call(command, c_uint(0), image, options)

So far, so good, it works fine. BUT… I decided not to go this way because in my case the small performance improvement comes at the cost of more boilerplate and above all a packaging problem (it’s just easier to rely on the already available CLI version of GMIC). For my use case it’s just not worth it, but should I need python bindings it would be a good solution to use ctypesgen and maybe a light bridge between gmic_interface_image and numpy.

Thanks again for your help.

Manuel_Quinones · April 26, 2022, 2:29pm

This was a great reference! @David_Tschumperle out of curiosity, where is the source code of fx_extract_foreground ?

I spent a while figuring out how to create the image with labeled points. So here is my code for reference. I’m using the Python bindings:

import gmic
import numpy
from enum import Enum

class Labels(Enum):
    ZERO = 0
    BACKGROUND = 1
    FOREGROUND = 2

g = gmic.Gmic()
image_results = []

g.run("input input.jpg", image_results)
input_image = image_results[0]

zero_image = gmic.GmicImage(width=input_image._width,
                            height=input_image._height, spectrum=1)

# Have to copy the array because the to_numpy() result is read-only
arr = zero_image.to_numpy().copy()

bg_points = [(967, 1651), (1656, 1588), (2068, 3636), (649, 3675),
             (1107, 2093)]

fg_points = [(1323, 1665), (1173, 1710), (1167, 1879), (1110, 2263),
             (933, 2331), (630, 2671), (757, 2506), (1230, 3600),
             (964, 3487), (1236, 1788), (1065, 2281), (1217, 2173),
             (1140, 1842), (991, 2231)]

for bg_x, bg_y in bg_points:
    arr[bg_x, bg_y] = Labels.BACKGROUND.value

for fg_x, fg_y in fg_points:
    arr[fg_x, fg_y] = Labels.FOREGROUND.value

labeled = gmic.GmicImage.from_numpy(arr)
image_results.append(labeled)

# Do permutation here because using GmicImage.from_numpy_helper(arr
# permute='yxzc') is not working.
g.run("permute[1] yxzc", image_results)
g.run("+_x_segment[0] +watershed[1] [-1]", image_results)
g.run("display", image_results)

Here is the result:

Reptorian · April 26, 2022, 4:26pm

It’s found on github/gmic/src/gmic_stdlib.gmic.

Here’s the extracted code:

#@gui Extract Foreground [Interactive] : fx_extract_foreground, gui_no_preview
#@gui : Feathering = _float(0,0,4)
#@gui : Dilation = int(0,-32,32)
#@gui : Output Mode = choice{3,"RGBA Image (Full-Transparency / 1 Layer)","RGBA Image (Updatable / 1 Layer)",
#@gui : "RGB Image + Binary Mask (2 Layers)","RGBA Foreground + Background (2 Layers)"}
#@gui : View Resolution = _choice{1,"Small (Faster)","Medium","High (Slower)","Very High (Even Slower)"}
#@gui : sep = separator()
#@gui : note = note{"<small><b>Description:</b>\n
#@gui : This filter allows to quickly extract foreground objects from background in opaque RGB images.
#@gui : Click on the <i>Apply</i> or <i>OK</i> buttons below to open the interactive window and start adding
#@gui : foreground and background control points. When you're done, exit the interactive window: your extracted
#@gui : foreground will be transferred back to the host software.\n\n
#@gui : If you are not satisfied with the result, click on <i>Apply</i> once again to modify your control points
#@gui : defined previously. To remove all control points, click on the <i>Reset</i> button above.
#@gui : </small>"}
#@gui : Last Image Size = value(0,0)
#@gui : Control Points = value(-1)
#@gui : sep = separator()
#@gui : note = note{"<small><b>Interactions:</b>\n
#@gui : Use the following actions in the interactive window to build your extraction mask :\n\n
#@gui : - <b>Left mouse button</b> or key <b>F</b> create a new <b>foreground</b> control point
#@gui : (or move an existing one).\n
#@gui : - <b>Right mouse button</b> or key <b>B</b> create a new <b>background</b> control point
#@gui : (or move an existing one).\n
#@gui : - <b>Mouse wheel</b>, or keys <b>CTRL+arrows UP/DOWN</b> zoom view in/out.\n
#@gui : - Key <b>SPACE</b> updates the extraction mask.\n
#@gui : - Key <b>TAB</b> toggles background view modes.\n
#@gui : - Key <b>M</b> toggles marker view modes.\n
#@gui : - Key <b>BACKSPACE</b> deletes the last control point added.\n
#@gui : - Key <b>PAGE UP</b> increases background opacity.\n
#@gui : - Key <b>PAGE DOWN</b> decreases background opacity.\n
#@gui : - Keys <b>CTRL+D</b> increase window size.\n
#@gui : - Keys <b>CTRL+C</b> decrease window size.\n
#@gui : - Keys <b>CTRL+R</b> reset window size.\n
#@gui : - Keys <b>ESC</b>, <b>Q</b> or <b>ENTER</b> exit the interactive window.
#@gui : </small>"}
#@gui : sep = separator()
#@gui : note = note("<small>Author: <i>David Tschumperlé</i>.      Latest Update: <i>2014/29/09</i>.</small>")
fx_extract_foreground :
  if !$! return fi
  resolution={arg(1+$3,512,1024,2048,0)}
  foreach {
    nm=${-gui_layer_name}
    => "[G"{`39`}"MIC] Interactive Foreground Extraction"
    if [$6][0]==-1" || "[$5]!=[w,h] _gui_control_points= else _gui_control_points=$6 fi
    status=${x_segment\ $resolution}
    sh 3 b. $1% if $2>0 dilate. {1+2*$2} elif $2<0 erode. {1-2*$2} fi
    rm.
    if $3==1 sh 3 max. 1 rm.
    elif $3==2 s c,-3 r. 100%,100%,1,4 rv =>[0] "name(Mask)" =>[1] "name("$nm")"
    elif $3==3
      .
      sh.. 0,2 +channels... 3,3 >=. 3 *[-2,-1] rm.
      sh. 0,2 +channels.. 3,3 <=. {255-3} *[-2,-1] rm.
      sh. 3 *. -1 +. 255 rm.
      gui_autocrop_layers[0]
      pos0=${gui_layer_pos[0]} pos1=${gui_layer_pos[1]}
      =>[0] "name("$nm" [foreground]),pos("$pos0")"
      =>[1] "name("$nm" [background]),pos("$pos1")"
    fi
  }
  if narg($status)>=4 u \{$1\}\{$2\}\{$3\}\{$4\}\{{w},{h}\}\{$status\} else u "" fi

Manuel_Quinones · April 26, 2022, 6:19pm

Thank you very much. This is useful! I was not finding it online because the file is too big for github to index it for search or direct display. My fault for not cloning the repo and check locally

afre · April 26, 2022, 8:18pm

I bookmark this for easy access:

https://raw.githubusercontent.com/dtschump/gmic/master/src/gmic_stdlib.gmic