Summary -
- if like me you’ve previously thought building software was too tricky, well it’s not, have a go!
- devs can help by posting up the details to build their changes. Then more people can test them!
- below shows how to build sigmoid in particular, and dt “master” generally, under Ubuntu 20.04. I expect this will work under a number of linux flavours.
I am no expert in this so bear that in mind.
There are various ways to build, some slicker than what I will describe. Also a wiki is available here - Building darktable · darktable-org/darktable Wiki · GitHub
Please note, this process is a “one shot” process where you start with a clean slate each time. It’s not efficient if you want to build daily, because you’d be downloading nearly 2Gb unnecessarily. Generally I only build every month or so.
1. Building darktable Master
Master is the current dt, dt under development, and includes all the work in progress which has passed certain checks. It seems logical to cover Master first.
1.1 In your existing dt, export any Styles you wish to continue using (see the manual).
1.2 Install any pending system updates (seems a wise move)
1.3 Install dependencies
dt relies on lots of software produced by others. This must be formally installed. Therefore -
Open up a Terminal window.
Terminal allows “commands” to be executed. We will type in or paste commands one by one. You can automate the build commands in future.
If Terminal is new to you, find out how to paste commands into it, and how to copy sections of the window.
Paste the following into Terminal
N.B. This step only needs doing the first time you build. Here I’m quoting the wiki.
sudo apt-get install gcc g++ cmake intltool xsltproc libgtk-3-dev libxml2-utils libxml2-dev liblensfun-dev librsvg2-dev libsqlite3-dev libcurl4-gnutls-dev libjpeg-dev libtiff5-dev liblcms2-dev libjson-glib-dev libexiv2-dev libpugixml-dev
The above is all one command. Then do
sudo apt-get install libgphoto2-dev libsoup2.4-dev libopenexr-dev libwebp-dev libopenjp2-7-dev libsecret-1-dev libgraphicsmagick1-dev libcolord-dev libcolord-gtk-dev libcups2-dev libosmgpsmap-1.0-dev libopenjp2-7-dev python3-jsonschema llvm-10-dev clang libimage-exiftool-perl liblua5.3-dev libsaxon-java imagemagick fop po4a
all one command
The wiki includes a package openjdk-14-jre but this was not found when I installed dependencies in February, and it doesn’t seem to have caused a problem.
From here on, the commands are needed each time you build.
1.4 Type, or paste in
mkdir -p ~/dt-Master-6Apr22
i.e. today’s date
This will create a folder in Home for all the code and build files. An advantage of having a new folder each time you build is that you can generally run your old version of dt if you want to e.g. compare behaviour.
1.5 type/paste…
git clone https://github.com/darktable-org/darktable.git ~/dt-Master-6Apr22/darktable-code
This will download code and other files from darktable’s repository / “database”. This stuff will be used to build dt. The files will be put in folder “darktable-code”.
About 2Gb of stuff is needed so it can take a while.
It should end saying something like “Resolving deltas: 100% (191893/191893), done.”
1.6 type/paste
cd ~/dt-Master-6Apr22/darktable-code
cd = “change directory”. This makes darktable-code your “working” folder, needed for what follows.
1.7 type/paste
git checkout master
The dt repository contains various developments/projects/code sets. You need “master”.
Often this command will say you are already using master.
1.8 type/paste
git status
This command is not strictly necessary but it is likely to say “Your branch is up-to-date with ‘origin/master’. nothing to commit, working tree clean”
which is good news.
If it says something else, you may have a problem.
1.9 type/paste
git config submodule.src/tests/integration.update none
This prevents a large amount of dt test data being downloaded.
1.10 type/paste
git submodule init
then
git submodule update
These are needed to prepare certain dt components ready for building.
1.11 type/paste
./build.sh --prefix $HOME/dt-Master-6Apr22 --build-type Release --jobs 5
This performs the first part of building. It will create a folder “build” under the prefix, for its output. I can’t remember what the build types are for but I think Release is recommended. The jobs parameter, 5, is telling it to use 5 CPU cores out of the 8 I have, for this step. Choose what you want.
This step produces a lot of output in the Terminal window. There are some warnings that look as if something may be wrong, however these are normal.
1.12 type/paste
cmake --build "/home/XXX/dt-Master-6Apr22/darktable-code/build" --target install -- -j5
This is the second and final part of the building.
XXX is your linux username.
-j5 is the same as – jobs 5.
Step 1.11 actually tells you to type this when it finishes. So you can copy it from the Terminal window and then paste it as the next command.
You are now ready to run dt -
1.13 type/paste
$HOME/dt-Master-6Apr22/bin/darktable --cachedir $HOME/.cache/darktable-6Apr --configdir $HOME/.config/darktable-6Apr
That’s all one command.
This runs the file “darktable” which has just been created.
dt needs a cache folder and a config folder – these are specified in the command. You can choose these two folders, although I suspect it would be wise to keep them under the standard system folders .cache and .config.
I use brand new folders. I like this because there’s no chance of any problems due to using files for a new version of dt which were created by an old version. Also, by having to set your preferences, you are alerted to new ones. However it means you are starting from scratch and will have to set up your preferences, custom module groups, and I imagine your custom presets, though I don’t use these. Often I think you could use the cache and config folders from your current dt without any problems. It’s your choice. If you rely on dt’s image database, my method will be no use because it starts with an empty database. In this case, use your previous folders. If a new version of dt needs a modified database structure, I think the processing for this is automatic when you run the new dt.
Once in dt, you will start editing photos. This means that normally-named XMP files in your photo folders will get updated. Your database will also get updated. Sometimes this means it becomes impossible for your XMPs/database to be subsequently used with any previous version of dt. Also you are now running a work-in-progress dt (Master) which might contain bugs which could corrupt your XMPs/database. Therefore it’s wise to back up your files before running the new version.
Import any Styles you exported.
Each build takes up nearly 2Gb. When you’ve completely finished with a build, you can simply delete e.g. dt-Master-6Apr22. You can delete .git under darktable-code (1.2Gb) at the end of the build process.
Tip – sometimes you mess up the process (or it messes itself up) and you want to start again. To avoid the relatively lengthy download in step 1.5 I copy e.g. dt-Master-6Apr22 into a separate location immediately after 1.5, and then I can quickly restore it if necessary, then continue with 1.6.
To run dt subsequently, open a Terminal window and paste in the line at the top of this section. (I have this in a text file on the desktop, for convenience) You might very reasonably want to put an entry in the dock / sidebar however this seems very much to depend on which desktop you use and may not be straightforward.
2 Building Sigmoid
In summary, the process here is to download Master and @jandren 's new code, fit the latter into the former, then build it in the same way as above.
2.1 If this is your very first build then install the dependencies as per 1.3.
2.2 Establish folders
mkdir -p ~/dt-Sigmoid-7Apr22/darktable-code
cd ~/dt-Sigmoid-7Apr22/darktable-code
2.3 Download Master
git clone https://github.com/darktable-org/darktable.git ~/dt-Sigmoid-7Apr22/darktable-code
2.4
git checkout master
2.5
git remote add jandren https://github.com/jandren/darktable.git
This points you to jandren’s code
2.5
git fetch jandren
This downloads the sigmoid program code.
2.6
git checkout sigmoid_tone_mapping
I think this is saying we want to build sigmoid and not anything else jandren might be working on. The command should reply with
Branch ‘sigmoid_tone_mapping’ set up to track remote branch ‘sigmoid_tone_mapping’ from ‘jandren’.
Switched to a new branch ‘sigmoid_tone_mapping’
2.7
git config --global user.email "EEE"
git config --global user.name "XXX"
I think this must be admin helping to track who’s got what.
EEE is your email address. It should be a valid address. It’s important if you were submitting code or changes, which you are not here. See @Entropy512 's post below for more info.
XXX is a user name, I don’t know for what.
I think you only need to do this the first time you merge code, based on my experience.
2.8 Now the two sets of code can be merged
git rebase master
This will produce a chunk of informatory messages.
2.9 Deal with submodules
git config submodule.src/tests/integration.update none
git submodule init
git submodule update
2.10 Build - part one
./build.sh --prefix $HOME/dt-Sigmoid-7Apr22 --build-type Release --jobs 5
2.11 Build - part two
cmake --build “/home/XXX/dt-Sigmoid-7Apr22/darktable-code/build” --target install – -j5
where XXX is your linux username.
2.12 Run it
$HOME/dt-Sigmoid-7Apr22/bin/darktable --cachedir $HOME/.cache/sigmoid-7apr --configdir $HOME/.config/sigmoid-7apr
with same comments as 1.13 re. cache and config.
If you do this about now, you should find you have not only Sigmoid, but also Filmic V6. The possibilities for obsessing about highlights are unbounded!
3. Building Master and Sigmoid in one session
Roughly, you could do up to 1.12, follow the Tip at the end of section 1, do 2.2, copy what you saved (the tip) into the new folder, then continue at 2.4.
4. @phweyland 's way
phweyland helped me with this building, I would have been stuck otherwise, thanks again. He has a slightly different method, as shown below. I think this puts the darktable code under HOME. Also you might want to suppress the download of the test data. This makes Master and Sigmoid in one session. XXX is the name of a folder where you want the built dt to be placed, under /OPT. I created it first.
git clone https://github.com/darktable-org/darktable.git
cd darktable
mkdir build
cd build
git submodule init
git submodule update
cmake -DCMAKE_INSTALL_PREFIX=/opt/XXX ../.
sudo make install -j5
git remote add jandren https://github.com/jandren/darktable.git
git fetch jandren
git checkout -b sigmoid_tone_mapping --track jandren/sigmoid_tone_mapping
git rebase master
git submodule update
sudo make install -j5
5. Good luck!
6. Edits since first posting
- say not a daily process
- further note on cache/config folders
- mention having multiple versions available
- use of email address
- comment on subsequent running
