Developing and Fiddling with the G'MIC Python Binding

Do you mean for bundling gmic-py in Gimel Studio ?

Yes. Though I wouldn’t just want to bundle it, but have core nodes for each of the effects, etc.

How far are you from a Windows version working with just pip install?

The related windows support issue is here on Github.

Compiling is done as a dedicated Github Workflow, the builds of which are here and the recipe file here.

I do not have Windows myself and had attempted to build in the blind through Github’s Windows containers and MSVC, creating many commits for this, to trigger builds. Once I had managed to get a fresh Windows VM packed with Microsoft development tools for my PC, another higher priority came in, so I paused the Windows support progress.

Adding Windows support entices:

  • tuning setup.py if needed
  • ensuring built .whl get the proper .dll files (ie. G’MIC’s dependency), given there is no standard wheels repair kit for windows. Even though there are libgcmic (for C API) static objects for Windows around provided by @Tobias_Fleischer , I will not use them as my Python binding is coded too tightly to libgmic’s native C++ API.
  • tuning the Windows Github Workflow (followed by a pypi.org .whl push, which is the build_tools.bash 11_send_to_pypi command).

For bundling gmic-py, the .whl format is useless, what is needed is a .dll or .so with its shared libraries dependencies around or bundled statically.

I may need 1 extra week maximum of work and cannot say when I may have managed it in 2021. Windows C/C++ Python developers should be much faster than me.

For core nodes support, you could use gmic-py + the latest GMIC filters description as JSON using native Python json parsing module.

This is what this gmic-blender example does by hand generating Python classes offline for each filter first (here the example just skip showing filter parameters and default to running the default example command), then evaluating those classes.

gmic-blender’s non-flipbook Github examples are demo’ed here on Youtube, as part of my 2020 Libre Grahpics Meeting talk.

Sorry, I’d like to help but I am not a C++ developer. :slightly_frowning_face:

1 Like

So that creates the filters so that they can be used offline?

Yes (use of filters offline)
well you can use them offline easily by predownloading the .json file first and parsing it, then generating your nodes from it.
This .json file is generated using the parse_gui command, as described in this Gist, but it takes 5-20 minutes depending on your computer, so the easiest way is to pre-download and embed it in your shippable software bundle, or embed some derived form (eg. generated Python code from parsing it).

:tiger:Just released gmic-py v2.9.4 alpha1.

Grab it using:
pip install gmic==2.9.4a1

Greening its tests suite in order to make a stable release will be done in a few weeks only.

The latest stable gmic-py release remains 2.9.2 (reachable through pip install gmic (or with --upgrade flag)).

So, I need to download the json file, parse it and the effects are callable from python? Or is this to keep updated with gmic filter updates, etc?
Sorry, I don’t quite understand.

:slight_smile:

wget https://gmic.eu/update294.json # or the python way, see stackoverflow

then in pseudo python

import json
filters_descr_dict = json.parse("update294.json")
for category in filters_descr_dict["categories"]:
    for filter in category["filters"]:
        # create a new type of node for filter["command"] which needs parameters filter["parameters"]

Beware, filter[“name”] are unicode and are sometimes in a non-latin language… so the filter[“command”] is maybe better as a key.

You may notice that filters’ categories have user names… this is because many of them are community-contributed… especially people can contribute a new category and filter very easily and have it merged at the whole community’s risk without @David_Tschumperle controlling stability much for them. So do not believe that these community filters will never change… Hence the possible need for a time-to-time nodes regeneration, if some update button is pushed (which corresponds to gmic.run("update") + looking at the downloaded file in $HOME/.cache/gmic/*.gmic or fetching the .json URL once again)

Just looking at https://gmic.eu/update294.json with Firefox or Chrome opens up a nice JSON browsing windows, otherwise I used to use http://jsonviewer.stack.hu/ in the old days for digging into a JSON’s structure easily without installing anything.

:tiger2:

1 Like

and for calling you basically run:

gmic.run(filter["command"] + " " + user_value_for_filter["parameters"][0]...[N])

You should also use the g.run instead of gmic.run() when possible, as described in this tutorial 2 part on the G’MIC interpreter instance. Which may imply your caching/attaching some G’MIC interpreter instance object within your program, as some singleton (bad), global variable, cache entry etc… But do not get stressed about that for the beginning.
I can review your gmic-py code if asked for.

1 Like

O.k. I will look into this. Thanks!

We can share screens if needed if I can help you code in any way using gmic-py.
Happy new year to you!

Hi @myselfhimself, thanks for the invite! :smiley: I probably will enventually need to do so, though at the moment I am working on a few other back-end related things with Gimel Studio before I integrate gmic-py.

(I am also sorta waiting for better windows support too…:wink:)

Also, is there good MacOs support for gmic-py?

I guess I am looking for wheels for gmic-py on all 3 platforms: win, linux, and macos. :grin:

Note: I understand the amount of work it is to do this, so I really appreciate your work on gmic-py. Keep up the good work!

2 Likes

gmic-py for MacOS is in 2.8.4 stable only right now, noone has ever tested it, only Github Actions Mac Os headless build and test bots for gmic-py or gmic-blender succeeded silently with me feeling some virtual gratefulness.
It can be installed with pip install gmic (or suffixed by ==2.8.4)

What is unsure of is whether the display command of G’MIC works under Mac OS, which should pop a display window. Most probably, if your wrap gmic-py for nodes-like images filtering, you will not care about the display command, but about the gmic.GmicImage list (2nd in-place parameter of gmic.run or gmic.Gmic.run()).
We have had no one who officially tested gmic-py 2.8.4…

If you or friends have a Mac OS, please test a simple command for me there:

pip install gmic
python3 -c "import gmic; gmic.run('sp blur 3 display')"

This will test both the Mac OS files (eg. image sample) download support and the display command.

When the Windows support gets out (I do not know when), the display command will very likely work and open up a native GDI window, as the gmic command line executable already does.

I am concentrating on my driver license theoretical exam for January 19th then will start a chocolate internship from February 15th to April 9th (but I will leave early enough every day to do cool stuff afterwards).

1 Like

Hi, I tried it with my virtual python environment. I am under MacOS Mojave 10.14.6

Unluckily your link path is a bit special: /usr/local/opt/libpng/lib

I am using MacPorts and my libraries are under /opt/local/lib/libpng16.16.dylib. Don’t know how to create links for your paths, assuming that this is only the first message.

(env) /Users/karo $ python3 -c “import gmic; gmic.run(‘sp blur 3 display’)”
Traceback (most recent call last):
File “”, line 1, in
ImportError: dlopen(/Users/karo/env/lib/python3.6/site-packages/gmic.cpython-36m-darwin.so, 2): Library not loaded: /usr/local/opt/libpng/lib/libpng16.16.dylib
Referenced from: /Users/karo/env/lib/python3.6/site-packages/gmic.cpython-36m-darwin.so
Reason: image not found

If I find some time I’ll try to change in your library gmic.cpython-36m-darwin.so.

By the way, I have llvm-6.0 still not libomp.dylib!

After some symbolic links I got at least

(env) /Users/karo $ python3 -c “import gmic; gmic.run(‘sp blur 3 display’)”
[gmic]-1./ Display image [0], from point (333,250,0) (console output only, no display support).
[0] = ‘cliff’:
size = (667,500,1,3) [3908 Kio of floats].
data = (91.9907,92.0012,92.0435,92.1084,92.1785,92.2411,92.2935,92.3397,92.3961,92.4777,92.5906,92.7337,(…),32.7627,34.537,34.3839,32.963,31.3481,30.1685,29.6017,29.5243,29.7324,30.0895,30.5156,30.9541).
min = 12.8431, max = 237, mean = 127.432, std = 68.9827, coords_min = (594,498,0,0), coords_max = (237,69,0,2).

gmic +v delivers

(env) /Users/karo $ python3 -c “import gmic; gmic.run(’+v’)”

gmic: GREYC’s Magic for Image Computing: command-line interface
(https://gmic.eu)
Version 2.9.0

    Copyright (c) 2008-2020, David Tschumperle / GREYC / CNRS. 
    (https://www.greyc.fr)

Thank you very much for testing and for your developer-oriented insight!!! You are possibly the first MacOS not-unknown tester thanks!!

The 2.8.4 release version embedding a 2.9.0 libgmic runtime instead is not a big issue. I had not a strict release process at that time and 2.9.0 was basically the version following 2.8.4.

 [gmic]-1./ Display image [0], from point (333,250,0) (console output only, no display support).

It seems like libgmic estimates that you have no display, and I guess that you have not seen a window popping up. I myself cannot help much at the gmic-py binding level. I remember @David_Tschumperle telling me that he wished some Mac OS user / developer would come in, give feedback and why not suggest code changes.

MacOS install instructions are written here on the documentation website for gmic-py
The G’MIC Python version which you have tested was compiled in a Github CI script with llvm/clang 6 because I could not manage to make OpenMP pytest test cases work with newer LLVM versions (see this test checking the gmic.__build__ variable’s contents and this one just after, leveraging an magic formula provided by @dtschump).
I apologize for forgetting to mention the libomp requirement.

Be it for MacOS, Linux, but possibly not Windows which handles .dll nicely, I do not know how to repair the gmic*.so file by hand (ie. ELF patching if that is the proper naming), though you seem to do. In the binary Python world, Linux (respectively MacOS) Python modules builds are usually followed by an auditwheel repair (respectively delocate). So… delocate was used to try to pack libpng, libfftw3, libz etc into the Mac OS final wheel. I can happily accept new build script repair step commands, so that the final .whl file has no shared libraries issues anymore… Or should we try to skip linking against a wheel-copied libpng library and let Mac OS find it on the system?

It seems the network works though. Which is a good news.

1 Like

So without display showing a preview, you can use output somefilename for previewing… or you can use Python matplotlib or other Python previewing system which works on your OS for now.

For example:

Though I do not remember when numpy/scikitimage/PIL support was added precisely… maybe 2.9.1… this is what my CHANGELOG suggests

Hmm, I am not a developer…

However, building gmiclib seemingly does not find X11. I, using MacPorts have to add for building gmic “SUBLIBS=-lX11”!

Your gmic has the following libraries, listed with
otool -L env/lib/python3.6/site-packages/gmic.cpython-36m-darwin.so
env/lib/python3.6/site-packages/gmic.cpython-36m-darwin.so:
/usr/local/opt/libpng/lib/libpng16.16.dylib (compatibility version 54.0.0, current version 54.0.0)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
/usr/local/opt/fftw/lib/libfftw3.3.dylib (compatibility version 9.0.0, current version 9.8.0)
/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.100.1)
/usr/local/opt/fftw/lib/libfftw3_threads.3.dylib (compatibility version 9.0.0, current version 9.8.0)
/usr/local/opt/llvm@6/lib/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
/usr/local/opt/llvm@6/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libc++abi.dylib (compatibility version 1.0.0, current version 902.0.0)

I can only show you my gmic listing:

(env) /Users/karo $ otool -L /usr/local/bin/gmic
/usr/local/bin/gmic:
/opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
/opt/local/lib/libfftw3.3.dylib (compatibility version 9.0.0, current version 9.8.0)
/opt/local/lib/libfftw3_threads.3.dylib (compatibility version 9.0.0, current version 9.8.0)
/opt/local/lib/libcurl.4.dylib (compatibility version 12.0.0, current version 12.0.0)
/opt/local/lib/libpng16.16.dylib (compatibility version 54.0.0, current version 54.0.0)
/opt/local/lib/libjpeg.9.dylib (compatibility version 14.0.0, current version 14.0.0)
/opt/local/lib/libtiff.5.dylib (compatibility version 12.0.0, current version 12.0.0)
/opt/local/lib/libX11.6.dylib (compatibility version 11.0.0, current version 11.0.0)
/opt/local/lib/libIlmImf-2_3.24.dylib (compatibility version 25.0.0, current version 25.0.0)
/opt/local/lib/libHalf.24.dylib (compatibility version 25.0.0, current version 25.0.0)
/opt/local/lib/libopencv_dnn.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/opt/local/lib/libopencv_highgui.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/opt/local/lib/libopencv_ml.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/opt/local/lib/libopencv_objdetect.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/opt/local/lib/libopencv_shape.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/opt/local/lib/libopencv_stitching.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/opt/local/lib/libopencv_superres.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/opt/local/lib/libopencv_videostab.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/opt/local/lib/libopencv_calib3d.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/opt/local/lib/libopencv_videoio.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/opt/local/lib/libopencv_imgcodecs.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/opt/local/lib/libopencv_features2d.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/opt/local/lib/libopencv_video.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/opt/local/lib/libopencv_photo.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/opt/local/lib/libopencv_imgproc.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/opt/local/lib/libopencv_flann.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/opt/local/lib/libopencv_core.3.4.dylib (compatibility version 3.4.0, current version 3.4.13)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)

I have tried to build gmic-py, but I don’t know how to switch compiler.

Actually I have clang and clang-mp-11 and -9.0 and in MacPorts llvm-6.0 … llvm-11, not knowing what that means.

By the way a quick trial with ‘sp karo_close 5 o /tmp/test.pgn’ worked1

AND … I am not a python user!

A ja, on Mac “install_name_tool” allows to change library paths!

1 Like