3D LUT module in darktable 2.7 (dev)

[David_Tschumperle (G'MIC staff)]

@David_Tschumperle

It would be a huge work to modify the code to avoid shadowing (I don’t even know if it would be possible).

I’ve done some cleaning this morning, and I can say CImg now compiles with -Wshadow, at least for g++ version 5.5.0 (and later).
The modified CImg.h file can be downloaded from the repo : Sign in · GitLab

[phweyland]

@David_Tschumperle

I’ve done some cleaning this morning, and I can say CImg now compiles with -Wshadow

Works for dt build too ! Thanks a lot for this huge (at least for me ) work.

@darix

gmic caches the extracted cluts. you could use the same cache dir

Sounds interesting. Could you be more precise about the sources I can look at ?

[David_Tschumperle (G'MIC staff)]

@phweyland

Sounds interesting. Could you be more precise about the sources I can look at ?

When G’MIC decompress one CLUT, it stores a version of the decompressed CLUT in ~/.cache/gmic/clut_name.cimgz .
The cimgz format is a CImg-specific format (using lossy compression this time), which is able to represent a 3d volumetric image of colors (which is what a CLUT is).
With the CImg library, you can load it with CImg<T>::load_cimg().

The ‘decompressed’ CLUT is not stored as an HaldCLUT in .png because this is limited to a few resolutions (siz^2 = SIZ^3).

[phweyland]

@David_Tschumperle

The ‘decompressed’ CLUT is not stored as an HaldCLUT in .png because this is limited to a few resolutions (siz^2 = SIZ^3)

I realize that decompress_lut accepts whatever dimension. Using dimension (SIZ) = 64, as currently coded, produced luts png compatible. However, not to be as specific, may preserve some future enhancement.

@darix

use the same cache dir

dt and G’MIC can share the same cache as long as they share the same set of lut names (same clut library gmic_cluts.png or at least same naming convention). But that can be confusing is the user has built his own collection of luts (different from gmic_cluts.png), right ?

[David_Tschumperle (G'MIC staff)]

@phweyland

I realize that decompress_lut accepts whatever dimension. Using dimension (SIZ) = 64, as currently coded, produced luts png compatible. However, not to be as specific, may preserve some future enhancement.

I’d add that G’MIC uses a resolution of 48x48x48 for decompressing the CLUTs. This is IMHO a good compromise between speed/memory usage/storage size and precision of the color data.
At the moment, I don’t really see any interest for switching to storing the decompressed CLUTs as .png files (with resolution 64^3 = 512^2) which would be larger and slower to decompress.

[darix]

see this discussion:

Packaging questions

currently the build system installs the gmic_cluts.gmz into the gimp plugin directory … this seems wrong. wouldnt something like /usr/share/gmic/ make more sense? also how big would be the “uncompressed” files be? would it be possible to add something like /usr/share/gmic/cluts/ to the search path for those uncompressed files? or are they arch dependent? also how can I unpack those files in my RPM build? last point for now: a lot of files you currently store in ~/.config/gmic/ should probably…

[phweyland]

@darix
For dt under Windows g_get_user_cache_dir() returns:
C:\Users\philippe\AppData\Local\Microsoft\Windows\INetCache
I don’t know what is returned under Linux and if that could correspond to .cache.
Any thought ?

@David_Tschumperle
With the filename C:\Users\philippe\AppData\Local\Microsoft\Windows\INetCache\gmic\edgyember.cimgz
The code:

cimg::exception_mode(0);
CImg img(1,1,1,3,0);
try { img.load_cimg(filename); }
catch (…) { return 0; } // or catch(CImgIOException&) as well

catches the exception when the file doesn’t exist (that’s ok) but not when the folder just below doesn’t exist (gmic for example). And dt is killed (that isn’t ok). Any advice ?

EDIT. cimgz files for 48x48x48 weigh 1297Kb. Is that correct ?

[Tobias]

Arggg, why does dt use the browser cache for luts?

C:\Users\philippe\AppData\Local\Microsoft\Windows\INetCache
This looks so wrong.

[matze (Matthias)]

@Tobias

Arggg, why does dt use the browser cache for luts?

Because it’s the only sane and portable solution to get a location for cache data?

[darix]

it is not DT or GMIC that picked this location but glib/gtk. so your complains would need to go there. and IIRC you can override it with an environment variable if you want. (XDG_CACHE_DIR IIRC)

[David_Tschumperle (G'MIC staff)]

@phweyland

catches the exception when the file doesn’t exist (that’s ok) but not when the folder just below doesn’t exist ( gmic for example). And dt is killed (that isn’t ok). Any advice ?

That is strange, I’ll check later.

@phweyland

EDIT. cimgz files for 48x48x48 weigh 1297Kb. Is that correct ?

If I take the CLUT ‘summer’, it’s more like:

• 116Kb in .png (64^3 = 512^2).
• 50Kb in .cimgz (48^3).
• 94Kb in .cimgz (64^3).

[phweyland]

Exception. Setting (instead using exception_mode()=0;):
#define cimg_verbosity 0
let me catch all errors.

File size, setting:
#define cimg_use_zlib 1 (not sure that’s the right syntax…)
plus saving the image as unsigned char instead of float gives me numbers aligned with yours (48^3):

Lossy compression + integer rounding … to be checked visually I guess.

[David_Tschumperle (G'MIC staff)]

@phweyland

The code:

cimg::exception_mode(0);
CImg img(1,1,1,3,0);
try { img.load_cimg(filename); }
catch (…) { return 0; } // or catch(CImgIOException&) as well

catches the exception when the file doesn’t exist (that’s ok) but not when the folder just below doesn’t exist ( gmic for example). And dt is killed (that isn’t ok). Any advice ?

I don’t have this behavior here, on my Ubuntu Linux.
Could you please give me one example of code where the issue happens?
Does it happen on Windows, or Linux ?

Also, cimg::exception_mode(0) is roughly equivalent to #define cimg_verbose 0, for the exceptions thrown by CImg, so we shouldn’t see any difference with the sample code you provided.

@phweyland

Lossy compression + integer rounding … to be checked visually I guess.

Integer rounding won’t change anything I guess, as it’s already almost impossible to distinguish between two colors with a +/-1 step on R,G or B.

[phweyland]

@David_Tschumperle

Does it happen on Windows, or Linux ?

On Windows, with that code. But I’m happy with#define cimg_verbose 0 which is probably better in the dt context.

[phweyland]

I’ve got a travis shadow error:

/usr/bin/clang+±7 -DGDK_DISABLE_DEPRECATED -DGDK_VERSION_MIN_REQUIRED=GDK_VERSION_3_22 -DGLIB_VERSION_MAX_ALLOWED=GLIB_VERSION_MIN_REQUIRED -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_40 -DGTK_DISABLE_DEPRECATED -DGTK_DISABLE_SINGLE_INCLUDES -DHAVE_BUILTIN_CPU_SUPPORTS -DHAVE_CONFIG_H -DHAVE_GPHOTO2 -DHAVE_GPHOTO_25_OR_NEWER -DHAVE_GRAPHICSMAGICK -DHAVE_HTTP_SERVER -DHAVE_KWALLET -DHAVE_LENSFUN -DHAVE_LIBSECRET -DHAVE_MAP -DHAVE_OPENCL -DHAVE_OPENEXR -DHAVE_OPENJPEG -DHAVE_OSMGPSMAP_110_OR_NEWER -DHAVE_PRINT -DUSE_COLORDGTK -DUSE_LUA -D_XOPEN_SOURCE=700 -D__GDK_KEYSYMS_COMPAT_H__ -Dlut3d_EXPORTS -I/build/darktable/src -Isrc -I/build/darktable/src/external/LuaAutoC -Isrc/iop/… -I/build/darktable/src/iop -isystem /build/darktable/src/external -isystem /usr/include/glib-2.0 -isystem /usr/lib/x86_64-linux-gnu/glib-2.0/include -isystem /usr/include/gtk-3.0 -isystem /usr/include/at-spi2-atk/2.0 -isystem /usr/include/at-spi-2.0 -isystem /usr/include/dbus-1.0 -isystem /usr/lib/x86_64-linux-gnu/dbus-1.0/include -isystem /usr/include/gio-unix-2.0 -isystem /usr/include/cairo -isystem /usr/include/libdrm -isystem /usr/include/pango-1.0 -isystem /usr/include/harfbuzz -isystem /usr/include/fribidi -isystem /usr/include/atk-1.0 -isystem /usr/include/pixman-1 -isystem /usr/include/freetype2 -isystem /usr/include/libpng16 -isystem /usr/include/gdk-pixbuf-2.0 -isystem /usr/include/libmount -isystem /usr/include/blkid -isystem /usr/include/uuid -isystem /usr/include/libxml2 -isystem /usr/include/libsoup-2.4 -isystem /usr/include/OpenEXR -isystem /usr/include/lensfun -isystem /usr/include/librsvg-2.0 -isystem /usr/include/x86_64-linux-gnu -isystem /usr/include/json-glib-1.0 -isystem /usr/include/openjpeg-2.3 -isystem /usr/include/libsecret-1 -isystem /usr/include/GraphicsMagick -isystem /usr/include/lua5.3 -isystem /usr/include/osmgpsmap-1.0 -isystem /usr/include/colord-1 -pipe -Wall -Wformat -Wformat-security -Wshadow -Wtype-limits -Wvla -Wthread-safety -Wno-error=varargs -Wno-error=address-of-packed-member -Wframe-larger-than=32768 -Wlarger-than=524288 -std=c++14 -std=c++11 -fopenmp=libomp -march=native -msse2 -g -O2 -g -DNDEBUG -O2 -fPIC -fvisibility=hidden -fvisibility-inlines-hidden -Werror -include common/module_api.h -include iop/iop_api.h -std=c++14 -MD -MT src/iop/CMakeFiles/lut3d.dir/decompress_clut.cpp.o -MF src/iop/CMakeFiles/lut3d.dir/decompress_clut.cpp.o.d -o src/iop/CMakeFiles/lut3d.dir/decompress_clut.cpp.o -c /build/darktable/src/iop/decompress_clut.cpp
In file included from /build/darktable/src/iop/decompress_clut.cpp:66:
/build/darktable/src/iop/./CImg.h:23543:25: error: declaration shadows a field of ‘cimg_library::CImg::_cimg_math_parser’ [-Werror,-Wshadow]
CImg expr(mp.opcode[2] - 4);
^
/build/darktable/src/iop/./CImg.h:16087:19: note: previous declaration is here
CImg expr, pexpr;
^
/build/darktable/src/iop/./CImg.h:24230:23: error: declaration shadows a field of ‘cimg_library::CImg::_cimg_math_parser’ [-Werror,-Wshadow]
CImg expr(mp.opcode[2] - 5);
^
/build/darktable/src/iop/./CImg.h:16087:19: note: previous declaration is here
CImg expr, pexpr;
^
2 errors generated.

[phweyland]

Some GMIC commands to manipulate lut files.

compress a (color to color transform) haldclut.png file
gmic -i dark_green_1.png -compress_clut 8,2,2048 split c,2 append x permute yxzc -o dark_green_1-c.png

compress a (color to b&w transform) haldclut.png file
gmic -i black_and_white.png -compress_clut 8,2,2048 split c -r[3] 100%,100%,100%,3,1 append c split c,2 append x permute yxzc -o black_and_white-c.png

decompress compressed clut to haldclut png file
gmic -i summer-c.png -split y -append c -permute cxyz -decompress_clut 64,64,64 -r 512,512,1,3,-1 -o summer.png

transform cube to haldclut
gmic -input_cube K_TONE.cube -r 64,64,64,3,3 -r 512,512,1,3,-1 -o K_TONE.png

compress (color to color transform) cube file
gmic -input_cube identity.cube -compress_clut 8,2,2048 split c,2 append x permute yxzc -o identity-c.png

extract one clut from gmic_cluts compressed library compatible with dt, but I haven’t succeeded yet removing the useless black points.
gmic -i gmic_cluts.png rows 128,129 -o edgyember-c.png

[Pascal_Obry]

If not already done those commands should probably be in comment in the 3dlut.c file. And maybe also in the RELEASE_NOTES?

[David_Tschumperle (G'MIC staff)]

https://framagit.org/dtschump/CImg/commit/0c333883a9d16d007e2db6a1792c97aeb2021b51 should fix this.

[phweyland]

@David_Tschumperle
Great ! that did it ! thank you very much.

@Pascal_Obry
I’ll do it. Meanwhile I hope I’ll have a complete set.

[phweyland]

@David_Tschumperle

Integer rounding won’t change anything I guess, as it’s already almost impossible to distinguish between two colors with a +/-1 step on R,G or B

I know that identity is the most useless lut but it helped me a lot to find my way … Here below is the identity compressed lut work (on the right). I’ve tried 48^3 and 64^3 with the same results: a slight fading of orange and blue. I don’t get that with the identity.cube file (no change when applied)

cube file (remove txt extension) identity size 2.cube.txt (259 Bytes)

identity compressed lut: (compression_lut did a perfect work here)

cimgz file: (replace txt extension by cimgz) identity-from cube.txt (52.1 KB)

EDIT: the above quote is misleading, sorry. The cache rounding/compression is not involved here. The effect is already there on the first decompression of the identity compressed clut.