Using digiKam to view and organize GIMP-2.9 XCF files

[Elle (Elle Stone)]

I’ve been using GIMP-2.9 since 2012 and over the years accumulated a lot of XCF files scattered hither and yon. Fortunately I kept these XCF files sorted into “project folders” - one folder per finished image. The project folders contain a lot of different file types in addition to the XCF files, such as raw file “pp3/pfp/xml/etc”, output tiffs and exr files from the various raw processors, intermediate files exported from GIMP for processing in other software, jpegs for the web, and etc.

I spent some time pulling the project folders together into one top-level folder. The next step was to be able to view, tag, sort, and rate the accumulated final images and associated XCF files.

digiKam would be perfect for the task, except that digiKam (as of 5.7.1) crashes upon encountering GIMP-2.9 XCF files. This likely will continue to be the case until GIMP-2.10 has been released and kimageformats has been updated to support GIMP-2.10 XCF files (see Bug 782244 – Make XCF format more accessible to file managers and similar software).

In the meantime, the way to keep digiKam from crashing when it hits a GIMP-2.9 XCF file is to not install kimageformats, but of course you won’t see a thumbnail or a preview image. I worked around this problem by creating “png sidecar files” for all my GIMP-2.9 XCF files (details below).

Not installing kimageformats brings other problems:

• Some Linux distributions likely won’t allow you to install digiKam without also installing kimageformats. Or even if digiKam can be installed without kimageformats, other KDE programs (for example Gwenview) might pull in kimageformats.

• Kimageformats is required for digiKam to be able to recognize and make preview images not just for GIMP XCF files, but also for Krita kra files, OpenEXR files, PSD files, and several other image file formats.

So the workaround is to build and install kimageformats from source, after modifying the source code to make kimageformats not build whatever it builds to allow digiKam to recognize XCF files. And then XCF files can be added to digiKam as an unsupported file format, for which digiKam will provide a generic thumbnail and no preview image.

Here is how to modify kimageformats to keep digiKam from crashing on GIMP-2.9 XCF files:

Download the source code:
Stable:
KDE - Experience Freedom! and scroll down to kimageformats
git - surely there is a kde anongit link, but I can’t find it, I used the github version:
GitHub - KDE/kimageformats: KImageFormats

Modify the source code:

1. Open “src/imageformats/CMakeLists.txt” with a code or text editor (I use geany)

2. Find the following lines for XCF files:
   ##################################

   add_library(kimg_xcf MODULE xcf.cpp)
   target_link_libraries(kimg_xcf Qt5::Gui)

   install(TARGETS kimg_xcf DESTINATION {KDE_INSTALL_QTPLUGINDIR}/imageformats/) install(FILES xcf.desktop DESTINATION {KDE_INSTALL_KSERVICES5DIR}/qimageioplugins/)

3. Comment out the lines for XCF files as follows, and save the modified file:
   ##################################

   #add_library(kimg_xcf MODULE xcf.cpp)
   #target_link_libraries(kimg_xcf Qt5::Gui)

   install(TARGETS kimg_xcf DESTINATION {KDE_INSTALL_QTPLUGINDIR}/imageformats/) #install(FILES xcf.desktop DESTINATION {KDE_INSTALL_KSERVICES5DIR}/qimageioplugins/)

4. Compile and install the source code. You are on your own for this step. It worked for me, but if your computer quits working after compiling and installing kimageformats, or if all your image files disappear, well, that’s a risk you take when you install software from source.

Unfortunately, at least on my Gentoo Linux system digiKam still can’t properly display Krita kra or OpenEXR files. That’s the bad news. The good news is that it no longer crashes when it reaches GIMP-2.9 XCF files.

5. Now start digiKam. You might have to add “xcf” to the list of supported file types (“Settings/Configure digiKam/Views/Mime Types”). Now digiKam will import the XCF files without crashing, and simply display a generic thumbnail with no preview.

Which of course still doesn’t solve the problem of actually being able to view the contents of GIMP-2.9 XCF files. But an obvious workaround is to make “GIMP sidecar files” such as sidecar pngs or sidecar tiffs. This way one can view the contents via the sidecar files, write tags and such to the sidecar files, and also right-click the GIMP XCF generic thumbnail to directly open any given XCF file from within digiKam, or simply drag the XCF file from digikam to an already running GIMP-2.9 and drop it on the tool box.

Making GIMP XCF sidecar files:

To export the sidecar files I’m using a very slightly modified version of the last script from this post:

A script to open a GIMP 2.9 XCF file and export a png

Anyone here familiar with writing scripts? My ability to write scripts is practically non-existent, but I want to write a script to run at the command line, that will Open all XCF files in a folder and its subfolders, one xcf file at a time Export a 16-bit png with the double extension “xcf.png” Close the XCF file The script needs to work with GIMP-2.9 files. There are scripting engines supplied with GIMP, but my efforts to make sense of “what to do” have completely failed. But maybe it’s no…

For the directory specified in the script, put in the path to the top level folder that you want the script to iterate over. And don’t forget to set the desired format for the exported sidecar file (png, tiff, etc). Also be sure to open GIMP and set whatever defaults you want GIMP to use for the output file type used in the script.

I debated back and forth between tiff and png sidecar files. The advantage of tiffs is that the tiffs will preserve the bit depth and precision of the XCF file. The advantage of pngs is that the bit depth will be at most 16-bit integer (smaller files sizes than 32-bit tiffs - almost all of my XCF files are at 32f precision). Also GIMP allows to set defaults for exporting pngs such as writing or not writing metadata to the exported pngs. This is not the case for exporting tiffs.

If you export pngs, I’d suggest setting the compression to 0 as the script takes much less time to execute if the pngs aren’t compressed by GIMP during the export. Then if you really want compressed pngs, use ImageMagick. But digiKam doesn’t have to decompress the png for display, if it’s not compressed in the first place. So I just left the pngs uncompressed.

Jpeg sidecar files also would work, but only if all of your GIMP XCF files are in perceptually uniform RGB color spaces such as regular sRGB. Otherwise the jpegs will have posterized shadows.

I decided to use uncompressed png sidecar files, with no metadata written to the files. Otherwise GIMP will export a lot of metadata taken from the original raw file (most of my images started as raw files), and all of that information is already in my digiKam database for the actual original camera files. Repeating all those tags for the files in the project folder would not be useful for my particular workflow.

Over in the following thread @Carmelo_DrRaw very kindly provided two very useful scripts if you want to take the sidecar files exported from GIMP, resize them to a fixed size, and convert them to a fixed color space:

Keep imagemagick from making grayscale images from tiffs

I’m trying to use ImageMagick to convert some tiffs from the embedded ICC profile to another ICC profile from disk. I’d prefer to use just about any command line program there is other than ImageMagick. But some of the tiffs are 32f which aren’t supported by cctiff. And as far as I can tell LCMS tificc requires that the input profile be included as part of the command (otherwise it defaults to sRGB), and the input tiffs don’t all have the same embedded ICC profile. This imagemagick command work…

For some workflows the resized and converted sidecar files from @Carmelo_DrRaw 's scripts would work very well. But shortly after getting all the resized sidecar files in place I realized that now I had no way to know the original dimensions of the corresponding XCF file, and also no way to know the RGB color space for XCF file, unless I actually open the XCF file. So I went back to the unmodified png sidecar files as directly output by GIMP. Sigh.

Displaying correct shadow tonality in digiKam for linear gamma RGB images:

This has been a lot of work to “work around” the fact that digiKam crashes upon encountering GIMP 2.9 XCF files. And I’m not done yet, because most of my XCF files - and also most of the other image files in the project folders - are all in linear gamma RGB working spaces, and digiKam as of 5.7.1 can’t correctly display the shadows of the previews for linear gamma images. It also trashes the shadows for the thumbnails, but that’s to be expected because at least to date thumbnails are always 8-bit images. See the very last posts to this thread for details: http://digikam.1695700.n4.nabble.com/Thumbnails-not-color-managed-severely-distorted-shadow-tonality-for-thumbs-previews-td4704410.html

Well, hopefully eventually the upstream digiKam code will provide a user switch to disable the LCMS optimizations (these optimizations are the reason digiKam incorrectly displays the shadow tonality for linear gamma images). But right now the only way to fix the problem of trashed shadows for linear gamma RGB images is to download the digiKam source code, modify one file, and then compile and install the modified code.

Stable:

Installing a tarball

git:

Download from git

I used the code from git. After downloading the source code:

1. Open “core/libs/dimg/filters/icc/icctransform.cpp” and locate the line which reads:
   transformFlags = 0;

2. Change it to read:
   transformFlags = cmsFLAGS_NOOPTIMIZE;

3. Save the file and compile and install as usual.

You might want to take a look at bootstrap.linux to see if there are any compile options you might want to disable. You can also modify the lines that read as follows:
if [ “$DIGIKAM_INSTALL_PREFIX” == “” ]; then
DIGIKAM_INSTALL_PREFIX=“/usr/local”

and change the install prefix to whatever you want. I think it might default to “/usr”, but I changed it to “/usr/local”.

Concluding thoughts:

Well, while working out the details of getting digiKam and GIMP-2.9 to play nice together, I also managed to whittle down my total number of XCF files in the “projects” folders from around 1000 to around 450 - that’s progress! Plus it’s really nice to be able to use digiKam to navigate to a folder, pick an XCF file via the associated sidecar file, and open the XCF file from within digiKam.

As a bonus, I discovered that the same script that I used to output GIMP-2.9 XCF “sidecar” files, can be modified to open PhotoShop PSD files using GIMP-2.9 and output sidecar files for all the PSD files. I don’t have many PSD files left on my computer (well, it’s been a long time since I ran Windows, and most of the old PSD files got lost along the way). Most of the PSD files that are left weren’t saved with maximum compatibility, so digiKam didn’t show any thumbnail. So it was nice to be able to use GIMP to get an idea of what was actually in those old PSD files.

Anyway, my apologies for the long post. But there might be other people trying to organize a bunch of GIMP-2.9 XCF files, who would like to be able to use digiKam to organize, locate and open these XCF files. So hopefully this “how to” might be useful.

[paperdigits (Mica)]

This is quite wonderful, thank you.

Perhaps a distracting question, but one I’ve been thinking about: Do you plan to keep all your intermediate files? I’ve been using a lot of hugin so I have intermediate files from there, plus the “final” tiff output from hugin. That tiff goes into gimp, where I’ll preserve it by immediately locking the layer. Storage is cheap, but I’m thinking I’d rather have good notes on how to reproduce rather have all the intermediate files. What do you think?

[Elle (Elle Stone)]

@paperdigits

Do you plan to keep all your intermediate files?

Hi @paperdigits - I’ve been thinking about this very question, and wanted to ask for other people’s input. So thank you! for raising the question.

Storage may be cheap these days, but even with 3TB storage attached to my system, some of the partitions were running out of space from huge layer stacks from blended images and lots of intermediate files for exposure-bracketed images. Cameras make big files these days! So before I could even get started re-organizing I had to do some preliminary culling, combined with stashing images to one drive to be able to reformat another drive and vice versa, in the process eliminating some partitions that in retrospect were not well thought out.

And again, storage may be cheap, but time is not. Sorting through endless piles of intermediate image files is not an especially good use of time, or at least that’s the conclusion I’ve reached. So I’ve tentatively concluded that probably the XCF file is the best place to store the intermediate files.

I’m thinking about including processing notes in my project folders, perhaps using something like “.notes” as the file extension, and telling digiKam to index these files. I tried using Zim to keep track of processing notes, but that didn’t work out so well.

I desperately need to work out a consistent naming scheme for my XCF, intermediate, and final image files. Often (especially for painting) there are many XCF files for the same project. At some point these 32f XCF files just get too large to handle comfortably, in terms of opening/saving time and also for screen-refreshing. So I tend to break the XCF files up into one file per stage in the processing, if the image requires a lot of processing. Plus I change my mind a lot about “what to do”, and so try different approaches.

So I would very much like to hear how other people handle such issues, from camera raw to final image including all the intermediate files as output for and by various softwares, and the various “try this approach” files, and for things like “one version for paper prints, another for the web”.

[paperdigits (Mica)]

My workflow is as follows:

If I can get by with only darktable or RawTherapee, that’s the best case. Produce a tiff file for print or a jpg for web, and be done with it. I always start in a raw developers, no matter which application it is. Never with dcraw as you color pros seems to do

If I need retouching in gimp, I dump a tiff file, open it gimp, immediately duplicate the layer, lock the original, then start working. I use layer names to tell myself what I’ve done. I can usually just toggle visibility to see any retouching, since I’ve preserved the output from darktable/rawthreapee as the base layer. I delete the tiff output from dt/rt, since, again, it’s in the xcf.

For hugin, I’ve been deleting all the intermediate files, since with the PTO file and the XMP/PP3 from dt/rt, I should be able to recreate.

I’m leaving more towards deleting intermediate files, especially if I can easily recreate them. You are right about disk space, I’m finding hugin intermediate files to be 1 - 3 gigs. Yikes!