Using 'crossroad' to compile Windows versions of G'MIC on Linux

I’m trying to use @Jehan’s tool crossroad to cross-compile Windows binaries of G’MIC and G’MIC-Qt on Linux. I follow the tutorial written here : crossroad · PyPI

I’ve installed the required packages, and at this point, I get:

$ crossroad --list-targets
crossroad, version 0.9.0
Available targets:
- android-mips64       Generic Android/Bionic on MIPS64
- w32                  Windows 32-bit
- android-x86-64       Generic Android/Bionic on x86-64
- android-arm          Generic Android/Bionic on ARM
- android-x86          Generic Android/Bionic on x86
- w64                  Windows 64-bit
- android-arm64        Generic Android/Bionic on ARM64
- android-mips         Generic Android/Bionic on MIPS
- native               Native platform (x86_64 GNU/Linux)

See details about any target with `crossroad --help <TARGET>`.

which looks good.

But now, when I try to set up a w64 project for G’MIC, I don’t have any change in my terminal (the prompt should be modified, as I’m using bash):

dtschump@elxxxra:~$ crossroad w64 gmic_w64
You are now at the crossroads...

dtschump@elxxxra:~$ 

And of course, when I try to compile then, it generates a binary that is an ELF linux executable (I’ve tried compiling the CLI version gmic using my own Makefile).

$ file gmic
gmic: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=d05d750091d89b9cd3d57de80126958f5c5631fa, for GNU/Linux 3.2.0, stripped

Does the project have to be compiled with cmake or autotools to make it work ?

Another try, this time with cmake:

$ crossroad w64 gmic_w64
You are now at the crossroads...

dtschump@exxxa:~/work/src/gmic-qt/build$ crossroad cmake ..
"cmake" is not a platform known by `crossroad`. Do not hesitate to contribute: <jehan at girinstud.io>
dtschump@exxxa:~/work/src/gmic-qt/build$

That does not work as in the tutorial :frowning:
Any idea ?

Hi @David_Tschumperle!

I was going to send you an email with a small tutorial about this as I did a successful cross-build yesterday. I was doing some other stuff first (one of them is some new command in crossroad to help setting up binfmt_misc as I needed it in your case. Until now I only had a small comment about this in the man, but I realize not everybody might know how this works).

Anyway this part does not look very good:

But now, when I try to set up a w64 project for G’MIC, I don’t have any change in my terminal (the prompt should be modified, as I’m using bash):

Which distribution are you using?

Maybe can you run this:

crossroad w64 gmic --verbose > crossroad.log 2>&1

Then it will hang (as crossroad is run interactively) so just ctrl-c after a few secs and email me the log. I’ll have a look.

P.S.: actually in your case, it might not even hang as it looks you immediately exit from crossroad (which is the problem we need to elucidate). Anyway send the logs and tell me it it hanged or not. :wink:

Yeah that’s because you fail to enter the crossroad environment. I need to figure out why, hopefully by looking at the verbose logs. :confused:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 19.10
Release:	19.10
Codename:	eoan

It hangs yes. And I’ve send you the e-mail with the log.

Ok so it’s a problem with the pip install. It tries to load files from what is obviously a temporary directory which was likely used during pip install. My setup scripts do some text replace during the build depending on the exact install prefix. I assume pip install in some temp directory and only move the files in the end, which messes up with my script.

I will need to figure this out and find a solution, but in the meantime, just install from the source:

git clone git://git.tuxfamily.org/gitroot/crossroad/crossroad.git
cd crossroad
./setup.py build && ./setup.py install

(install step accepts a --prefix argument of course)

I’ll have lunch now then I will write up the tutorial on how to build gmic-qt in like 15 minutes with crossroad. :slightly_smiling_face:

For the record, I could reproduce the issue on a Debian stable with pip3 version 18.1 (python 3.7), but the install works fine on pip3 version 19.1.1 (Fedora 31, also python 3.7). So it might be a pip bug which has been fixed in recent versions.

@David_Tschumperle Which version of pip3 are you using?

$ pip3 --version
pip 18.1 from /usr/lib/python3/dist-packages/pip (python 3.7)

That corresponds to my test then. I assume this issue has been fixed from pip3 version 19 and later (though I could the bug report about this).

OK, so bypassing the pip3 install and downloading the archive helps.
But I’m still stuck at some point.

Trying to compile the gmic CLI tool, with cmake (for a w64 architecture):

w64✘gmic dtschump@exxxxa:~/work/src/gmic/build$ crossroad cmake ..
crossroad info: running "cmake -DCMAKE_INSTALL_PREFIX:PATH=$CROSSROAD_PREFIX -DCMAKE_TOOLCHAIN_FILE=$CROSSROAD_CMAKE_TOOLCHAIN_FILE .."
-- The CXX compiler identification is unknown
-- The C compiler identification is GNU 9.2.0
-- Check for working CXX compiler: /usr/local/share/crossroad/bin/x86_64-w64-mingw32-g++
-- Check for working CXX compiler: /usr/local/share/crossroad/bin/x86_64-w64-mingw32-g++ -- broken
CMake Error at /usr/share/cmake-3.13/Modules/CMakeTestCXXCompiler.cmake:45 (message):
  The C++ compiler

    "/usr/local/share/crossroad/bin/x86_64-w64-mingw32-g++"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /home/dtschump/work/src/gmic/build/CMakeFiles/CMakeTmp
    
    Run Build Command:"/usr/bin/make" "cmTC_980fd/fast"
    /usr/bin/make -f CMakeFiles/cmTC_980fd.dir/build.make CMakeFiles/cmTC_980fd.dir/build
    make[1]: Entering directory '/home/dtschump/work/src/gmic/build/CMakeFiles/CMakeTmp'
    Building CXX object CMakeFiles/cmTC_980fd.dir/testCXXCompiler.cxx.obj
    /usr/local/share/crossroad/bin/x86_64-w64-mingw32-g++     -o CMakeFiles/cmTC_980fd.dir/testCXXCompiler.cxx.obj -c /home/dtschump/work/src/gmic/build/CMakeFiles/CMakeTmp/testCXXCompiler.cxx
    /usr/local/share/crossroad/bin/x86_64-w64-mingw32-g++: 135: -o: not found
    make[1]: *** [CMakeFiles/cmTC_980fd.dir/build.make:66: CMakeFiles/cmTC_980fd.dir/testCXXCompiler.cxx.obj] Error 127
    make[1]: Leaving directory '/home/dtschump/work/src/gmic/build/CMakeFiles/CMakeTmp'
    make: *** [Makefile:121: cmTC_980fd/fast] Error 2
    

  

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:71 (project)


-- Configuring incomplete, errors occurred!
See also "/home/dtschump/work/src/gmic/build/CMakeFiles/CMakeOutput.log".
See also "/home/dtschump/work/src/gmic/build/CMakeFiles/CMakeError.log".

I’m back!

You probably don’t have the cross-compiler x86_64-w64-mingw32-g++ installed. The one which CMake finds (in /usr/local/share/crossroad/bin/) is a wrapper installed by Crossroad (editing some env variables and some flags). But it still needs to have the real compiler behind. I am guessing that if you run crossroad -h w64, C++ is in the list of uninstalled language.

On Debian stable, I think that the package to install is g++-mingw-w64-x86-64. This may be the same on Ubuntu.

P.S.: I just pushed a fix to our wrapper so that next time, the error will be more explicit.

P.P.S.: I will now redo a build from scratch and write down each step in a new message.

So here is the full procedure to build G’MIC! Based on building on a Debian stable (except I target it to Ubuntu users, since @David_Tschumperle is using Ubuntu, so I use sudo for few root-needing commands) .

  1. Install crossroad!
$ git clone git://git.tuxfamily.org/gitroot/crossroad/crossroad.git
$ cd crossroad
$ pip3 install -r requirements.txt
$ ./setup.py build && ./setup.py install

(obviously a --prefix option is available to the install subcommand if preferred)

  1. Just as for a native build, following README.md:
$ git clone https://github.com/dtschump/gmic.git
$ git clone https://github.com/c-koi/gmic-qt.git
$ make -C gmic/src CImg.h gmic_stdlib.h
  1. Let’s make sure we have C and C++ cross-compilers:
$ crossroad -h w64
w64: Setups a cross-compilation environment for Microsoft Windows operating systems (64-bit).

Not available. Some requirements are missing:
- x86_64-w64-mingw32-gcc [package "gcc-mingw-w64-x86-64"] (missing)
- x86_64-w64-mingw32-ld [package "binutils-mingw-w64-x86-64"] (missing)
$ sudo apt install binutils-mingw-w64-x86-64 gcc-mingw-w64-x86-64
[…]
$ crossroad -h w64
w64: Setups a cross-compilation environment for Microsoft Windows operating systems (64-bit).

Installed language list:
- C
Uninstalled language list: 
- Ada                 Common package name providing the feature: gnat-mingw-w64-x86-64
- C++                 Common package name providing the feature: g++-mingw-w64-x86-64
- OCaml               Common package name providing the feature: mingw-ocaml
- Objective C         Common package name providing the feature: gobjc++-mingw-w64-x86-64
- fortran             Common package name providing the feature: gfortran-mingw-w64-x86-64
$ sudo apt install g++-mingw-w64-x86-64
$ crossroad -h w64
[…]
Installed language list:
- C
- C++
[…]
  1. Ok now we have the basics to build for Windows 64-bit. Let’s enter the crossroad environment. I will call it gmic:
$ crossroad w64 gmic
⤫ ccd -y gmic-qt
⤫ crossroad source msys2

Note: you will notice I use a different prompt in my tutorial to show when I am inside crossroad () or not ($).

  1. Finally let’s configure. We will use CMake (Crossroad supports autotools, meson and CMake only, but not qmake, patches accepted!), so it obviously has to be installed as well, if not already (sudo apt install cmake), same for pkg-config (sudo apt install pkg-config). I could just do a one-liner with all crossroad dependencies, but I will take the opportunity to show how the package manager works by discovering the first dependency issue together.
⤫ crossroad cmake /path/to/src/gmic-qt/
[…]
CMake Error at CMakeLists.txt:129 (find_package):
  By not providing "FindQt5.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Qt5", but
  CMake did not find one.

  Could not find a package configuration file provided by "Qt5" (requested
  version 5.2.0) with any of the following names:

    Qt5Config.cmake
    qt5-config.cmake

  Add the installation prefix of "Qt5" to CMAKE_PREFIX_PATH or set "Qt5_DIR"
  to a directory containing one of the above files.  If "Qt5" provides a
  separate development package or SDK, be sure it has been installed.


-- Configuring incomplete, errors occurred!
See also "/home/jehan/.local/share/crossroad/artifacts/w64/gmic/gmic-qt/CMakeFiles/CMakeOutput.log".
⤫ crossroad search --search-files Qt5Config.cmake
[…]
The following packages have files matching the search "Qt5Config.cmake":
	- phonon-qt5
	- qca-qt5
	- qt5
	- qt5-static
	- snorenotify
⤫ crossroad install qt5

Now re-run crossroad cmake /path/to/src/gmic-qt/, check the next dependency error, search the package with crossroad search --search-files, install and so on. Loop on. I will spare you each run. Here is the command to install remaining missing dependencies:

⤫ crossroad install fftw curl gimp

Note: the gimp dependency is for building with GMIC_QT_HOST=gimp (the default, which is the only tested so far) as it requires to link against libgimp. That pulls quite a lot of dependencies, but obviously they won’t be necessary after the build.

  1. Special case: the build needs to call some Qt binaries (moc and rcc) and I have not figured out yet if we can use native ones. But also even if the format is the same, native tools may not be the same version as the used Windows Qt libraries (apparently it’s important as I see that there is a wrapper tool called qtchooser to select the right Qt binary depending on your version/target, etc.). I didn’t want to spend too long figuring out what was what so I just rely on Wine for this and it works fine (no build issue on a Fedora 31 and a Debian stable at least). Therefore I temporarily install Wine as a binfmt_misc interpreter for Windows binaries.
$ sudo apt install wine
$ sudo sh -c "echo ':DOSWin:M::MZ::/usr/bin/wine:' > /proc/sys/fs/binfmt_misc/register"

Note: I use $ as it doesn’t need to be run from inside the crossroad environment, but whatever. It would work inside as well.

  1. Configure again and now it should be a success!
⤫ crossroad cmake /path/to/src/gmic-qt/
crossroad info: running "cmake -DCMAKE_INSTALL_PREFIX:PATH=$CROSSROAD_PREFIX -DCMAKE_TOOLCHAIN_FILE=$CROSSROAD_CMAKE_TOOLCHAIN_FILE /path/to/src/gmic-qt/"
-- Using CMake version: 3.13.4
Build type is Release
Building for target host application: gimp
G'MIC path: ../gmic/src
LTO is disabled (windows platform)
Found G'MIC repository
Found ../gmic/src/CImg.h
Found ../gmic/src/gmic_stdlib.h
CImg version is [293]
G'MIC version is [293]
-- Correctly found FFTW3
G'Mic: using OpenMP
-- The following OPTIONAL packages have been found:

 * CURL
 * OpenMP, A low-level parallel execution library, <http://openmp.org/wp/>
   Optionally used by gmic-qt

-- The following REQUIRED packages have been found:

 * Qt5Core
 * Qt5Gui
 * Qt5Widgets
 * Qt5Network
 * Qt5 (required version >= 5.2.0)
 * Qt5LinguistTools
 * PNG
 * ZLIB
 * FFTW3

-- Configuring done
-- Generating done
-- Build files have been written to: /home/jehan/.local/share/crossroad/artifacts/w64/gmic/gmic-qt
  1. Now build:
⤫ make
[…]
[100%] Linking CXX executable gmic_gimp_qt.exe
[100%] Built target gmic_gimp_qt
⤫ make install
[  1%] Automatic MOC for target gmic_gimp_qt
[  1%] Built target gmic_gimp_qt_autogen
[100%] Built target gmic_gimp_qt
Install the project...
-- Install configuration: "Release"
-- Installing: /home/jehan/.local/share/crossroad/roads/w64/gmic/lib/gimp/2.0/plug-ins/gmic_gimp_qt.exe

You may take a rest while watching the build, surf the web, watch webcomics or movies, etc. Unlike a native build in a VM, a cross-build won’t take more time that a native Linux build, and won’t kill your resource usage (not more than a native Linux build again; technically it’s exactly the same except that the output format is different, that’s all). On a very non-powerful micro machine, full build took about 20 minutes.

  1. The files will be found under $CROSSROAD_PREFIX. In particular, the GIMP plug-in for instance will be in $CROSSROAD_PREFIX/lib/gimp/2.0/plug-ins/gmic_gimp_qt.exe:
⤫ file $CROSSROAD_PREFIX/lib/gimp/2.0/plug-ins/gmic_gimp_qt.exe
/home/jehan/.local/share/crossroad/roads/w64/gmic/lib/gimp/2.0/plug-ins/gmic_gimp_qt.exe: PE32+ executable (GUI) x86-64 (stripped to external PDB), for MS Windows

Nice we have a Windows binary! Small bonus for packaging: you may look for the library dependencies automatically with x86_64-w64-mingw32-objdump (which come with the binutils-mingw-w64-x86-64 package which we already installed).

⤫ x86_64-w64-mingw32-objdump -p $CROSSROAD_PREFIX/lib/gimp/2.0/plug-ins/gmic_gimp_qt.exe |grep dll
	DLL Name: Qt5Core.dll
	DLL Name: Qt5Gui.dll
	DLL Name: Qt5Network.dll
	DLL Name: Qt5Widgets.dll
	DLL Name: libbabl-0.1-0.dll
	DLL Name: libcurl-4.dll
	DLL Name: libfftw3-3.dll
	DLL Name: libgegl-0.4-0.dll
	DLL Name: libgimp-2.0-0.dll
	DLL Name: libgobject-2.0-0.dll
	DLL Name: libpng16-16.dll
	DLL Name: libgcc_s_seh-1.dll
	DLL Name: libgomp-1.dll
	DLL Name: libstdc++-6.dll
	DLL Name: GDI32.dll
	DLL Name: KERNEL32.dll
	DLL Name: msvcrt.dll
	DLL Name: SHELL32.dll
	DLL Name: USER32.dll
	DLL Name: zlib1.dll

Then recursively do the same thing for each DLL, so that you get the full list of dependencies you need to package. Siril for instance automatized this process with a Python script calling this tool recursively (which is pretty cool so I am planning on using the same thing for GIMP).

  1. You may reset the binfmt_misc configuration to not run Windows binaries by mistake on your pretty Linux box (note that this was not a permanent change anyway and would have been resetted after reboot):
sudo sh -c "echo -1 > /proc/sys/fs/binfmt_misc/DOSWin"
  1. Enjoy G’MIC with a beer, a glass of wine, kir, whisky, schnaps, soda, water, juice… whatever suits you!

Note: a lot of these steps are only to be done once or automatized a bit more. So on next builds, it will be a lot faster.

2 Likes

This doesn’t work.

$ crossroad -h w64
Unknown option for the "help" command: w64
$ crossroad --help w64
Unknown option for the "help" command: w64

The package you mention was indeed not installed. So I’ve installed it.

Trying to compile again…

This is to be run from outside the crossroad environment. There are commands for inside, and others for outside. Actually -h also works inside the crossroad environment, but then it is to get help on target subcommand (for instance crossroad -h install), instead of help on target platforms.

Trying to compile again…

Did you see the tutorial I just posted above your comment? :slightly_smiling_face:

Just seen your comment above, trying to deal with it :slight_smile: Thanks!

Wow, that installs tons of stuffs, and I’m getting out of space on my device.
Will try later then.
Thanks again.

Wait what? This is an echo inside a kernel file (see more about binfmt_misc). It is not supposed to install anything. This is simply a string to tell your kernel how it can execute Windows files (i.e. by running them through Wine).

I was talking about this, it just exploded all my disk space:

  • the qt5 libs.

Ahah yeah. It does install quite a bunch of dependencies, most of them are obviously not even needed (that’s the problem of making big keep-all packages; especially for GIMP, a libgimp package would have been useful for build-only use cases).

Though you were probably already close to a full disk, I would guess, no?

✘du -sh $CROSSROAD_PREFIX
2.3G	/home/jehan/.local/share/crossroad/roads/w64/gmic
✘du -sh $CROSSROAD_HOME
61M	/home/jehan/.local/share/crossroad/artifacts/w64/gmic
✘du -sh ~/.cache/crossroad/
521M	/home/jehan/.cache/crossroad/

The whole build (artifacts, prefix with dependencies and cache included, so basically everything) only takes me ~3GiB. :slightly_smiling_face:

O_o