User provided custom DCP Profiles are not loaded by the AppImage build

Hello!
I am trying to make RT use automatically a custom DCP Profile for a particular camera, Nikon Z5 in this case. I obtained the DCP profile - which can be explicitly loaded, but I would like RT to behave as the documentation says, meaning that the user can come with own file, name it as the camera is named and RT will match that and apply it automatically.
Surce:
https://rawpedia.rawtherapee.com/Color_Management
https://rawpedia.rawtherapee.com/How_to_create_DCP_color_profiles

It seems that I am not the only one having the issue: Rawtherapee Appimage and the use and installation of .dcp profiles
The solution suggested there may work, but that seems a bit more convoluted and not what the docs actually say, that there is a directory on the system where the user can place custom DCP files which will be automatically looked up and used if the camera matches the name of the file.

So far I tried adding the custom DCP file in :
/usr/share/rawtherapee/dcpprofiles/
~/.local/share/rawtherapee/dcpprofiles
I also tried to make the camera name uppercase, lowercase etc to no avail. The custom file is just not picked up.

I also remember that when I was using older versions, that I was compiling from scratch, I was able to add these custom profiles without issue. What changed? Is RT now looking in another location? Which one would that be?

Screenshot from 2021-03-20 03-00-40 Screenshot from 2021-03-20 03-00-17

Screenshot from 2021-03-20 03-08-21 Screenshot from 2021-03-20 03-07-48

I find that auto select works only if the profile I want to be selected is just the plain camera name with nothing else in it. Try renaming the Fujifilm X-T30.dcp to something like Fujifilm X-T30 Standard.dcp and the adobe one to just Fujifilm X-T30.dcp.

The XT-30 works fine, as it already comes embedded within the appimage build. It is the Z5 one that it does not work.
The camera name as seen in exif by RT is “NIKON Z 5”.
I grabbed flat and standard DCP profiles from Adobe DNG converter and put copies of them using all variations of upper case or mixed case with or without spaces between Z and 5 in both the said directories, none of them is picked. The one that should be picked is “NIKON Z 5.dcp” as this matches the exif character by character. Yet, it is not.
For the AppImage build, it is likely that /usr/share/rawtherapee maps within the appimage itself as appimage is kind of a self contained root fs in itself so the directory and files one on my system would likely be totally useless.
However, well behaving apps also use a ~/.local/share directory for user specific overrides or addons. This one I think is not (or no longer?) used by RT. If some of the core devs could confirm my hypothesis, I can also go on and file a bug report so that it gets fixed in future releases.

Hi. Sorry, not a developer here, but anyway…

May I ask where is that said in the documentation, please? I would like to update the Spanish page if I missed that part.

To my understanding the profiles located in the user space (those folders starting with ~/) are the processing profiles (those used to save custom developing settings) and not the camera profiles. It would be nice if somebody proves me wrong, as it would be a huge mistake on my part…

And if I’m not wrong, the camera profiles must be located inside /usr/share/rawtherapee/dcpprofiles or inside /usr/share/rawtherapee/iccprofiles. Given that an appimage has those folders self contained, as you correctly guessed, I think you won’t be able to add your custom profiles without modifying the appimage itself, so maybe the solution given in the other thread still applies, as you only have to do it once per profile and be good to go.

Not ideal, but it works.

Hello,
The role of the .local/share directory comes from the GNOME/freedesktop guidelines:
https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html
These are the guidelines on how to write well behaved desktop apps on linux (and there are quite similar conventions on MacOS).
Relevant sections:

$XDG_DATA_DIRS defines the preference-ordered set of base directories to search for data files in addition to the $XDG_DATA_HOME base directory. The directories in $XDG_DATA_DIRS should be seperated with a colon ‘:’.

If $XDG_DATA_DIRS is either not set or empty, a value equal to /usr/local/share/:/usr/share/ should be used.
$XDG_DATA_HOME defines the base directory relative to which user specific data files should be stored. If $XDG_DATA_HOME is either not set or empty, a default equal to $HOME/.local/share should be used.

Now, according to RT documentation, the DCP profiles should exist under
/usr/share/rawtherapee/dcpprofiles/

Basically, these dcp profiles are considered application data, and RT would look for them in a correct location, if only partially. The guidelines says the app should look in both /usr/share/… and in ~/.local/share/… however RT does not honor the second possible location.

For a traditional build, with a binary compiled and deployed somewhere in the host filesystem, this limitation would not be a major issue since the significance AND content of /usr/share/$something would be the same for all binaries on the system, so if one put a file under usr/share/rawtherapee/dcpprofiles/ as the RT documentation says here https://rawpedia.rawtherapee.com/How_to_create_DCP_color_profiles, any other app on the system would potentially be able to find it.
UNLESS the app is an AppImage, which rather resembles a container that has its own private root filesystem with its own private /usr/share/$something, so now the contents of that folder is only whatever was bundled there in the AppImage.

For these cases the guidelines referred initially says that the app data can reside in ~/.local/share too, not only on /usr/share - to allow the user to add application specific data in a location that is both accessible to the user, but is also within the reach of the application.

More to the point, RT DOES honor another section of the guideline, the one sayin where the config files reside:

$XDG_CONFIG_HOME defines the base directory relative to which user specific configuration files should be stored. If $XDG_CONFIG_HOME is either not set or empty, a default equal to $HOME/.config should be used.

Wherever relative to the paths in XDG_DATA_DIRS I would place a rawtherapee/dcpprofiles subdir with the correct dcp profiles within, the profiles are still not loaded, it may be possible that the path RT’s documentation tells about is just a hardcoded default, and not a derivation from a proper implementation of the freedesktop guidelines.

If you mean this couple RT documentation sentences:

To have RawTherapee automatically use your new DCP, place the DCP file in the dcpprofiles folder. This folder will typically be inside the RawTherapee installation folder in Windows, or in /usr/share/rawtherapee/dcpprofiles/ in Linux and macOS.

then I’m afraid it is hardcoded, indeed.

And even when I would also be happy to have some easy place to put my camera profiles so they are automatically recognized and applied by RT, I don’t think splitting my custom files into several folders is a great idea. At least not when you try to build a reasonable backup strategy. At the very least they would have to be inside the config folder, but it would be better that all custom files where located in the directory set in Preferences>Color management>Directory containing color profiles.

But I guess this is not the place to ask for this. If you wish the developers to consider making such modifications to RT, using these guidelines (they talk about bugs, but feature requests work the same way), you may ask for it at github.

I strongly disagree with this - not when it prevents a user with no system write (aka sudo) privileges from being able to install their own profiles.

Depending on this weekend’s weather I’m going to dig through the code, I was positive RT did have a user-writable profile location, although I do recall seeing some “odd” behavior where the user-writable location didn’t seem to actually work properly for an ICC output profile, I was too busy and lazy enough to just sudo around it but it merits me spending more time if this weekend’s weather isn’t particularly good.

(If it’s good, I’m heading outside!)

1 Like

I think I haven’t explain it clearly. Sorry.

As @VDPhoto points out, the GNOME/freedesktop guidelines suggests placing some user selectable files in certain user space folder ($HOME/.local/share). That would mean placing our custom camera profiles into $HOME/.local/share/rawtherapee/dcpprofiles/ and $HOME/.local/share/rawtherapee/iccprofiles/.

From a backup strategy point of view, I would just have to create copies from $HOME/.local/share/rawtherapee/ and $HOME/.config/RawTherapee. And in case I have set a «Directory containing color profiles» (Preferences>Color management>Directory containing color profiles), also all files from within that folder.

That means I have to backup from 3 different places, no problem with that. But theoretically a backup is meant to be restored sometime in a future, and then I would have to remember where those files must be placed (if I haven’t recreated the whole path in my backup), and if the destination folders change in a future version, most probably I’ll have to read 3 different documentation pages to know where the backup files belong to.

At first I thought it would be better placing the custom camera profiles inside the $HOME/.config/RawTherapee folder, but I don’t know if that makes much sense. That’s why I suggested placing them in the same custom folder as the processing profiles folder (Preferences>Color management>Directory containing color profiles).

Hope it’s understandable now.

1 Like

It does but not as a SINGLE source.
It is perfectly normal for an app to come with some predefined data files that are used for processing other data (in this case various processing presets, just like the DCP files or just like the raw processing files)
As well as it is perfectly normal for the users to have their own overrides/replacements or new files supporting new hardware - especially if the app encourages the user to contribute such files for extra processing accuracy.
Now with the DCP profiles it can be argued that they are almost config files and we can consider having the USER contributed DCP files also looked up in the ~/.config/rawtherape directory. They are not QUITE config files, they are data files describing hardware as opposed to processing profiles that just contain a sequence of operations to apply on raw files.

Nevertheless, at least for RT’s case, the distinction is somewhat blurry and having the system files in /usr/share/rawtherapee and user contributed files in ~/.config/rawtherapee is not going to hurt anybody as well as we can have both system provided files AND user provided files for the same purpose.