rawproc and CMake...

I’ve never been happy with the hackish build system for rawproc. I think it’s what has kept a lot of you from trying it, especially on Linux. The Windows installer releases have been pretty simple and reliable to get a full-up rawproc, but the AppImage is borked for most uses other than Ubuntu…

I’m writing this in part to illustrate the challenges in integrating all the supporting libraries to make a raw processor useful; sudo apt install (a bunch of packages) just doesn’t cut it. i enumerated most in a previous post somewhere recently but here’s the key concerns:

  • Distro packages don’t keep up with library development. This is particularly problematic with libraw, and probably any other raw-ingesting library. New cameras get introduced all the time, and their unique-ness has to be accommodated per-camera.

  • Distro packages don’t sometimes include useful features. LittleCMS’ fast-float speeds up color transforms greatly, not every build environment has enabled it due to the different licensing. Lensfun 0.3.95 contains useful features to allow an application developer to flag which corrections are available from the database, but no build system I’ve inspected (Ubuntu, mxe, msys2) goes better than 0.3.3.

So, for rawproc to be really functional one needs to build at least libraw, lcms2, and lensfun by-hand, and then incorporate their include headers and libraries into the build system. I’ve been using an autotools/configure and GNU make build system, which just let me force-feed paths and files at the right places, I’d be curious if there’s anyone out there who’s built rawproc with one or more hand-fed libraries…

Anyway, a cascade of little things led me to write a CMake build system. Now, that’s been a painful experience, because I really want to learn just enough CMake to do my particular task. But CMake isn’t like that, it has its own personality: syntax conventions, data organization, that just defy my notions of dealing with a programming language. My hobby here is photography, trying hard not to make CMake a ‘hobby’…

Anyway, rawproc as of commit #7161f42 pushed a minute ago now lets one build rawproc with cmake, either with distro packages, downloaded and compiled packages, or a mix of the two. If you go the distro route the package dependencies are in the README, install those and do this:

mkdir build
cd build
cmake ..
make
make doc 

and you’ll get a ready-to-run rawproc in that build directory.

Alternatively, if you want to tie up your machine for a few minutes, do this:

mkdir build
cd build
cmake -DBUILD_LCMS2=ON -DBUILD_LIBRAW=ON -DBUILD_LENSFUN=ON -DBUILD_EXIV2=ON -DLENSFUN_DBUPDATE=ON -DBUILD_LIBRTPROCESS=ON  ..
make
make doc

and in a bit you’ll have a working rawproc with all the most recent libraries. This bit of CMake business will download or clone each library, build it and make the headers and libs available to rawproc for its build.

For you developers, I used CMake’s ExternalProject facility to do this. It handles all manner of retrieval, and give one pretty decent control over the configure/build process, handling such things as requiring an install step for exiv2 because it does a header build that doesn’t show up until install. It took a bit of stackoverflow-searching to get the library and dependency thing sorted out, but it works well with the alternative FindPackage option for using distro packages.

One downfall of it is doing subsequent makes for rawproc code development. It’s a mishmash of cmake and autotools configuration, and ExternalProject wants to run the autotools configures every time. LittleCMS and Libraw have this situation. The CMake literature talks about using a ‘superproject’ organization to get around this, where you make your software just another ExternalProject, sorry, my head cmake-hurts too much right now to consider this.

I simply hate these things; my mill is sitting with a model steam locomotive part halfway-cut while I muddle through all this. That’s another rabbit hole; my other hobby is model railroading, but I’m also wading through learning machine shop to support my aims there. Machining is like cmake in that regard, but it makes more sense… :stuck_out_tongue_closed_eyes:

If any of you try the new build system, let me know if/how it works for you…

Edit: Cripes forgot wxWidgets! For Ubuntu, do this:

sudo apt install libwxgtk3.0-gtk3-dev

for msys2:

pacman - S mingw-w64-x86_64-wxwidgets3.2-msw

I’m going to work on the ExternalProject for it in the next couple of days…

4 Likes

It probably doesn’t help that LensFun does not seem to keep their releases neat and tidy:

BTW, Arch is on 0.3.4.

Anyway, I just wanted to say that yes, this is a major pain point, and several large projects have resorted to building and maintaining their own dependencies. A good example is Julia, my favorite language. They maintain a huge infrastructure just to be able to control every library and binary they depend on: GitHub - JuliaPackaging/BinaryBuilder.jl: Binary Dependency Builder for Julia

Maybe there’s a way to leverage that infrastructure for your own project?

1 Like

It’s a tag in the github repo. And, they just tagged 0.3.99 after removing the deprecated functions. rawproc cmake builds lensfun with the last code commit as of yesterday, which is tagged 0.3.99…

They haven’t released since 0.3.0, IMHO really need to do a 4.0.0 with all the recent restructure. I’d also like to see them carve out the code and database, but who am I to talk… :laughing:

1 Like

One more post here, and I’m done with this little project…

The cmake build system now has another option, to link in externally-built libraries without CMake shenanigans. Basically, make a build-support directory somewhere, clone or download/unpack whatever projects there, and build each with an install prefix defined in the configure/cmake command, e.g., for cmake, -DCMAKE_INSTALL_PREFIX=…/…/usr. You’re basically building your own little /usr/local… And then, building rawproc, export the variable PKG_CONFIG_PATH with the path of your little /usr/local prepended to the original contents:

export PKG_CONFIG_PATH="/home/glenn/ImageStuff/rawproc10/build-support/usr/lib/pkgconfig:${PKG_CONFIG_PATH}"

and rawproc cmake will find the self-built libraries just like it would find system libraries.

All three options are described in the rawproc README:

https://github.com/butcherg/rawproc

If I were just wanting to try rawproc, I’d do a combination: distro-install all the required libraries, then build rawproc like this:

$ cmake -DBUILD_LIBRTPROCESS ..

to get the librtprocess demosaics.

Anyway, have fun…