preliminary libraw support -- call for testing

Hi,
I just pushed a libraw branch which uses – well… :slight_smile: – libraw for the raw decoding. The support is alpha quality, and I expect bugs and glitches, so I’d appreciate some testing from people who can build from source. (You need to set ENABLE_LIBRAW=1 in CMakeCache.txt to enable it)

Thanks in advance!

3 Likes

Hi Alberto,

i just tried to compile on Arch with libraw 0.20.2 installed, but the build fails:

[ 25%] Building CXX object rtengine/CMakeFiles/rtengine.dir/rawimagesource.cc.o
/home/rene/source/aur/art-rawconverter-git/src/art-rawconverter-git_src/rtengine/rawimage.cc: In member function ‘int rtengine::RawImage::loadRaw(bool, unsigned int, bool, rtengine::ProgressListener*, double, bool)’:
/home/rene/source/aur/art-rawconverter-git/src/art-rawconverter-git_src/rtengine/rawimage.cc:693:28: error: ‘struct libraw_data_t’ has no member named ‘rawparams’; did you mean ‘params’?
  693 |             libraw.imgdata.rawparams.shot_select = shot_select;
      |                            ^~~~~~~~~
      |                            params
make[2]: *** [rtengine/CMakeFiles/rtengine.dir/build.make:1112: rtengine/CMakeFiles/rtengine.dir/rawimage.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:164: rtengine/CMakeFiles/rtengine.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
==> ERROR: A failure occurred in build().
    Aborting...

Hi @guzzisti , thanks for the feedback! Indeed, I used the latest libraw snapshot from github, and I relized just now that they added a backwards-incompatible change… Can you try this patch (against the current head of the libraw branch) and see if it fixes it?

diff --git a/rtengine/rawimage.cc b/rtengine/rawimage.cc
--- a/rtengine/rawimage.cc
+++ b/rtengine/rawimage.cc
@@ -690,7 +690,11 @@
 #else // ART_USE_LIBRAW
         {
             //libraw.recycle();
+#if LIBRAW_COMPILE_CHECK_VERSION_NOTLESS(0, 21)
             libraw.imgdata.rawparams.shot_select = shot_select;
+#else
+            libraw.imgdata.params.shot_select = shot_select;
+#endif
             
             int err = libraw.open_buffer(ifp->data, ifp->size);
             if (err) {
1 Like
Version: 1.8.2-208-g09460789f
Branch: libraw
Commit: 09460789f
Commit date: 2021-11-02
Compiler: cc 11.2.0
Processor: Skylake
System: Windows
Bit depth: 64 bits
Gtkmm: V3.24.5
Lensfun: V0.3.95.0
Exiv2: V0.27.5.2
LCMS2: V2.12
LibRaw: V0.20.2
Build type: release
Build flags:  -m64 -mwin32 -msse2 -mfpmath=sse -mthreads -Wno-attributes  -Wno-aggressive-loop-optimizations -Wno-parentheses -std=c++11 -fno-tree-loop-vectorize -march=native -Werror=unused-label -fno-math-errno -Wall -Wuninitialized -Wno-deprecated-declarations -Wno-unused-result -fopenmp -Werror=unknown-pragmas  -DNDEBUG  -O3 -ftree-vectorize
Link flags: -m64  -mthreads  -static-libgcc   -march=native  -s  -O3  -fno-use-linker-plugin
OpenMP support: ON
Build OS: Windows
Build date: 2021-11-02T17:23:43Z

It crashes on generating thumbnails with segfaults.

I am going to build a debug version to get more info.

Thanks! should be fixed now

the versioning made me think for a moment if i built the correct branch :wink:

Version: 1.8.2-209-gc495ff1ae
Branch: makepkg
Commit: c495ff1ae
Commit date: 2021-11-02
Compiler: cc 11.1.0
Processor: x86_64
System: Linux
Bit depth: 64 bits
Gtkmm: V3.24.4
Lensfun: V0.3.95.0
Exiv2: V0.27.3
LCMS2: V2.12
LibRaw: V0.20.2
Build type: Release
Build flags: -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2,-D_GLIBCXX_ASSERTIONS         -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -std=c++11 -fno-tree-loop-vectorize -march=native -Werror=unused-label -fno-math-errno -flto -Wall -Wuninitialized -Wno-deprecated-declarations -Wno-unused-result -fopenmp -Werror=unknown-pragmas -O3 -DNDEBUG -ftree-vectorize
Link flags: -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -march=native -flto
OpenMP support: ON
Build OS: Linux
Build date: 2021-11-02T18:18:22Z

Did some quick test, no obvious problems with the raw images of my Fuji. Will do some more thorough testing the next days.

Seems to be a bit faster then the “old” version - besides speed, what are the main reasons for switching to libraw?

I don’t know if it’s faster actually. The reason for switching is not having to update the internal dcraw code to support new formats and cameras, which is getting increasingly difficult… For example, libraw already supports Sony lossless compressed raws (e.g. produced by the A1 and probably A7 IV) which are not supported by art master.

5 Likes

Yay libraw! I’ve used it from the start in rawproc, and I’ve been able to concentrate on the image operations, color, and optimizing my workflow. And not on .CR3…

Hello, I guess I’m doing something wrong. I switched to the libraw branch with the command git checkout libraw. In CMakeCache.txt I see the following lines:
//Use libraw for decoding
ENABLE_LIBRAW:BOOL=1

Then I compile, make and install, works as usual. But when I start Art, I see in the splash/welcome screen:
LibRaw: N/A

Synaptic tells me that libraw-bin, libraw-dev and libraw19 (all versions 0.19.5) are installed on my system (Xubuntu 20.04.3).

Any ideas?

Hi @paulmatth, what is the output of grep LIBRAW_FOUND CMakeCache.txt?

LIBRAW_FOUND:INTERNAL=

  • I put in the cmake command the tag -DENABLE_LIBRAW=ON

and in CMakeCache. txt, a line ENABLE_LIBRAW:BOOL=ON is generated. That seems to work.

  • W10 libraw builds are available at https://keybase.pub/gaaned92/ART-W64NightlyBuilds/

  • problem with CR3 files
    I use those files that come from playraw. Sorry to not give credit.
    A64I5539.CR3 (17.2 MB)
    IMG_1562.CR3 (13.3 MB)
    IMG_3022.CR3 (9.4 MB)
    ART-libraw generates a black thumbnail. When opening, it crashes

    Preprocessing: 493926 usec
    s<=0!
    Assertion failed: s > 0, file D:/ART/ARTSOURCE/ART/rtengine/LUT.h, line 152


    [Thread 9168.0x293c exited with code 3]
    [Thread 9168.0x3ec exited with code 3]
    [Thread 9168.0x1f38 exited with code 3]
    [Thread 9168.0x3c84 exited with code 3]
    –Type for more, q to quit, c to continue without paging–
    [Inferior 1 (process 9168) exited with code 03]
    (gdb) bt
    No stack.
    (gdb)

    Version: 1.8.2-214-gd5c13a6f3
    Branch: libraw
    Commit: d5c13a6f3
    Commit date: 2021-11-03
    Compiler: cc 11.2.0
    Processor: undefined
    System: Windows
    Bit depth: 64 bits
    Gtkmm: V3.24.5
    Lensfun: V0.3.95.0
    Exiv2: V0.27.5.2
    LCMS2: V2.12
    LibRaw: V0.20.2
    Build type: debug
    Build flags: -m64 -mwin32 -msse2 -mfpmath=sse -mthreads -Wno-attributes -Wno-aggressive-loop-optimizations -Wno-parentheses -std=c++11 -fno-tree-loop-vectorize -Werror=unused-label -fno-math-errno -Wall -Wuninitialized -Wno-deprecated-declarations -Wno-unused-result -fopenmp -Werror=unknown-pragmas -O1 -g2
    Link flags: -m64 -mthreads -static-libgcc -O1
    OpenMP support: ON
    Build OS: Windows
    Build date: 2021-11-03T11:42:59Z

@paulmatth I just realized I put a constraint that libraw has to be version 0.20 or higher, so your problem is that your version is too old. There’s a chance that it might work also with 0.19, but I don’t know (see also below). If you want to test, you can try changing

    pkg_check_modules(LIBRAW libraw_r>=0.20)

to

    pkg_check_modules(LIBRAW libraw_r>=0.19)

on line 547 of CMakeLists.txt in the root dir of ART.

These all work fine for me using LibRaw 0.21.0-Snapshot202101 (the latest available from github). So I guess your libraw version is simply too old…

I use the last version available on MINGW64.
So I have to build libraw for W10?
Or perhaps @ggbutcher has already a recent build for W10?

Hello Alberto, this doesn’t work, compiling stops after 25%, see terminal output below. Seems my libraw version is too old. I’ll try to get an newer version (but not today).

@gaaned92, tried your suggestion as well (-DENABLE_LIBRAW=ON). Compiling went fine but the splash screen of Art still says
LibRaw: N/A

[ 25%] Building CXX object rtengine/CMakeFiles/rtengine.dir/rawimage.cc.o
[ 25%] Building CXX object rtengine/CMakeFiles/rtengine.dir/rawimagesource.cc.o
/home/paul/art/rtengine/rawimage.cc: In member function ‘int rtengine::RawImage::loadRaw(bool, unsigned int, bool, rtengine::ProgressListener*, double, bool)’:
/home/paul/art/rtengine/rawimage.cc:531:47: error: ‘struct libraw_iparams_t’ has no member named ‘normalized_make’
  531 |             if (strcmp(libraw_->imgdata.idata.normalized_make, "Fujifilm") == 0 && strncmp(libraw_->imgdata.idata.normalized_model, "GFX", 3) == 0) {
      |                                               ^~~~~~~~~~~~~~~
/home/paul/art/rtengine/rawimage.cc:531:115: error: ‘struct libraw_iparams_t’ has no member named ‘normalized_model’
  531 |             if (strcmp(libraw_->imgdata.idata.normalized_make, "Fujifilm") == 0 && strncmp(libraw_->imgdata.idata.normalized_model, "GFX", 3) == 0) {
      |                                                                                                                   ^~~~~~~~~~~~~~~~
/home/paul/art/rtengine/rawimage.cc:542:29: error: ‘struct libraw_iparams_t’ has no member named ‘normalized_make’
  542 |             strncpy(make, d.normalized_make, sizeof(make)-1);
      |                             ^~~~~~~~~~~~~~~
/home/paul/art/rtengine/rawimage.cc:544:30: error: ‘struct libraw_iparams_t’ has no member named ‘normalized_model’
  544 |             strncpy(model, d.normalized_model, sizeof(model)-1);
      |                              ^~~~~~~~~~~~~~~~
/home/paul/art/rtengine/rawimage.cc:588:27: error: ‘struct libraw_colordata_t’ has no member named ‘raw_bps’
  588 |             tiff_bps = cd.raw_bps;
      |                           ^~~~~~~
make[2]: *** [rtengine/CMakeFiles/rtengine.dir/build.make:1025: rtengine/CMakeFiles/rtengine.dir/rawimage.cc.o] Fout 1
make[2]: *** Wachten op onvoltooide taken...
make[1]: *** [CMakeFiles/Makefile2:167: rtengine/CMakeFiles/rtengine.dir/all] Fout 2
make: *** [Makefile:130: all] Fout 2
paul@graveyron:~/art$ 

I build it statically from their master branch.

I just pushed a fix that should make also 0.20.2 work

Thanks Alberto.

I tried to build libraw. It’s painfully slow and at the end, during linking it says some files ar missing and thus cannot generate dll!

Hi Alberto,
I compiled and installed libraw 0.20.2 on my system (mint 20.2) in usr/local/lib.
I compiled ART successful, but when I start it, I got the message:

./programs2/art-test/ART: error while loading shared libraries: libraw_r.so.20: cannot open shared object file: No such file or directory

What can I do ? Where I have to put libraw_r.so.20?

This is AboutThisBuild.txt:

Version: 1.8.2-215-g2b761b8ea
Branch: libraw
Commit: 2b761b8ea
Commit date: 2021-11-03
Compiler: cc 9.3.0
Processor: x86_64
System: Linux
Bit depth: 64 bits
Gtkmm: V3.24.2
Lensfun: V0.3.2.0
Exiv2: V0.27.2
LCMS2: V2.9
LibRaw: V0.20.2
Build type: release
Build flags:  -std=c++11 -march=native -Werror=unused-label -fno-math-errno -Wall -Wuninitialized -Wno-deprecated-declarations -Wno-unused-result -fopenmp -Werror=unknown-pragmas -O3 -DNDEBUG -ftree-vectorize
Link flags:  -march=native
OpenMP support: ON
Build OS: Linux
Build date: 2021-11-03T19:32:30Z

Try a sudo ldconfig. If that’s not enough, search for something like this

HTH

1 Like