master build help

Soupy · December 26, 2023, 7:13am

I want to build master from source - not because I’m a developer, but to have access to the latest updates without relying on the OBS repos.

I tried to git clone darktable for the first time with this command:

git clone --recurse-submodules --depth 1 git@github.com:darktable-org/darktable.git /home/usr/src/darktable

But kept running into this:

Cloning into '/home/usr/src/darktable/src/tests/integration'...
remote: Enumerating objects: 4965, done.        
remote: Counting objects: 100% (900/900), done.        
remote: Compressing objects: 100% (322/322), done.        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
error: 7659 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
fatal: clone of 'https://github.com/darktable-org/darktable-tests.git' into submodule path '/home/usr/src/darktable/src/tests/integration' failed
Failed to clone 'src/tests/integration' a second time, aborting

I tried many different solutions, from HTTP/2 to HTTP/1.1, increasing http.postBuffer, turning off core.compression, adding various lines to .ssh/config and .gitconfig, and nothing worked. It always failed at /tests/integration. Speed was ~500 KiB/s, over wifi, but the network wasn’t cutting out. It completed anywhere from 15-50% before the error.

Giving up on that, I instead downloaded the ZIP, giving me the location /home/usr/src/darktable/darktable-master, and ran:

cd /home/usr/src/darktable
git init
git remote add origin git@github.com:darktable-org/darktable.git
git add .
git pull origin master

That completed, but has given me a folder structure that doesn’t look quite right (I guess I should have done cd /home/usr/src/darktable/darktable-master but didn’t realise until too late):

As far as I can tell, the only difference between /src/darktable and /src/darktable/darktable-master is the folder .git. Therefore, is it safe to move .git into darktable-master and delete the rest? Is it also safe to then move everything inside darktable-master into darktable and delete darktable-master?

And after that, do I carry on as per usual, with:

git submodule update --init
./build.sh --prefix /opt/darktable --build-type Release --install --sudo

OS is Linux Mint 21.2

My goal is to be able to run git pull in the future without having to download the ZIP every time.
Many thanks.

priort · December 26, 2023, 9:33pm

syntax???

##git clone --recurse-submodules --depth 1 https://github.com/darktable-org/darktable.git##

Ignore the number signs… I forgot how to quote but this seemed to work…

tankist02 · December 26, 2023, 11:55pm

I use HTTPS without any further adjustments

git clone https://…

and don’t have any problems

Soupy · December 27, 2023, 12:54am

I tried HTTPS initially and it didn’t work, 1.1 and 2, which is why I went to SSH instead.

Is anyone who builds master able to take a screenshot of their directory, so I can see what type of folder structure I should be aiming for?

priort · December 27, 2023, 2:22am

In addition to git pull you need to initialize and update the submodules…

Also initially you need to be sure you have all the dependencies…

I think Kofa has a nice little sequence he follows.

I do it in WIndows so it might be best to search the forum or let him reply here… I will search too…

@kofa

Ah found it…

Soupy · December 27, 2023, 3:26am

Yes I will update the submodules but I need to make sure I am changing to the right directory first.

My first quick question to @kofa looking at his set up would be, do you have .git folder in /darktable or /darktable-master?

kofa · December 27, 2023, 6:10am

The simplest way to get the source (and keep it updated) is by first doing a git clone.
You use the command git clone <URL of the remote repository>, where the URL is on the github page, you can copy it into your clipboard (you can use either the SSH or the HTTPS version):

If you don’t specify a directory, the default is the name of the repo:

git clone git@github.com:darktable-org/darktable.git

will check it out to a directory called darktable, a new subdirectory of whatever directory you issued the command in. So, if you issue it in your home, your source will end up in ~/darktable; if you run it in ~/my-git-projects, it will be in ~/my-git-projects/darktable, and if you specify a directory name:

git clone git@github.com:darktable-org/darktable.git my-darktable-source-directory

then it will be in ~/my-darktable-source-directory or ~/my-git-projects/my-darktable-source-directory, depending on where you run it.

You only need to clone once; then you just update it.

To answer your question, the .git subdirectory is in the directory where you cloned the repo. In my case, the path to .git is ~/darktable/.git. It’s not a directory you’ll ever need to enter, it’s where git keeps its admin stuff.

My current build script, based on suggestions from kmilos (you can save this in a build-dt file and make that executable (chmod +x build-dt), and either put it on you path, or reference it by name.

The important part is the rm command and whatever comes after it; the first two (renice, ionice) are only for my convenience to reduce the load on the machine while building.

First, the compiled OpenCL kernels are removed to avoid version problems: rm -rf ~/.cache/darktable/*kernel*.

Then we switch to the repo’s directory (see above): cd ~/darktable.

Since I sometimes build not from the master branch, but some PR’s branch, this script always returns to master: git checkout master.

Then, all local modifications and the output files from the previous build are discarded: git clean -d -f -x.

The next two git commands refresh the sources: git pull --rebase the main ones, git submodule update --init --recursive the submodules.

As the last step before the actual build, the target directory, in my case ~/darktable-master, is removed: rm -rf /home/kofa/darktable-master.

And then we run the build, putting the result in: ./build.sh --prefix /home/kofa/darktable-master --build-type Release --install.

To run darktable: ~/darktable-master/bin/darktable -d common (I put that in a shell script called dt). Change your logging settings (replace -d common) with whatever you prefer.

#!/bin/bash
renice -n 19 -p $$
ionice -c 3 -p $$

rm -rf ~/.cache/darktable/*kernel*

cd ~/darktable
git checkout master
git clean -d -f -x
git pull --rebase
git submodule update --init --recursive
rm -rf /home/kofa/darktable-master
./build.sh --prefix /home/kofa/darktable-master --build-type Release --install

Hope this helps. I’ll be off for a week.

Soupy · December 27, 2023, 7:50am

Thanks for your reply, it is helpful, though I guess you didn’t read the initial post, where I clearly state git clone failed, hence why I am in this trouble, and trying to find a workaround using the downloaded ZIP.

So I have /home/tim/src/darktable which is the downloaded ZIP plus the .git folder obtained by running the commands mentioned in my initial post. I try running each line individually in your script and get errors:

git checkout master
gives
error: Your local changes to the following files would be overwritten by checkout ... Please commit your changes or stash them before you switch branches. Aborting

git pull --rebase
gives
fatal: Updating an unborn branch with changes added to the index.

./build.sh --prefix /opt/darktable --build-type Release --install --sudo
gives

Traceback (most recent call last):
  File "/home/tim/.local/bin/cmake", line 5, in <module>
    from cmake import cmake
ModuleNotFoundError: No module named 'cmake'

Although Synaptic tells me I do have cmake.

What am I missing?

kofa · December 27, 2023, 8:47am

Can’t help you with cmake.

You asked about the .git directory, so I gave you git instructions. I’ve never used the ZIP and have no idea what is packed inside it. I’m not sure you can mix git with the ZIP like that; git will see you have modified files locally (Your local changes...).
You could maybe try HTTPS instead of SSH when you copy the URL and try the clone with that.

Sorry, I’m leaving for the airport. Back in a week, until then only on the phone, and even that sporadically.

Soupy · December 27, 2023, 9:24am

I tried https first. It failed.
Appreciate your time, enjoy your trip.

LaurenceLumi · December 27, 2023, 10:02am

I am having a similar if not same problem, poor internet while travelling.

This approach worked:

initial cloning of repo:

git clone --depth 25 url-here

fetch commits with increasing twice per try depth:

git fetch --depth 50

git fetch --depth 100

git fetch --depth 200

…and so on

eventually (when I think enough is fetched) I run git fetch --unshallow - and it’s done.

Still can’t compile out of the box, but that’s another problem

kofa · December 27, 2023, 10:21am

You can clone with depth 1 for the current state. You don’t need the complete history.

LaurenceLumi · December 27, 2023, 11:49am

I realised that later, but I wanted to build 4.4.2 but I think depth 500 gets your there.

MStraeten · December 27, 2023, 1:15pm

If you just want 4.4.2 you can simply download the source package for it: Release release 4.4.2 · darktable-org/darktable · GitHub

kofa · December 27, 2023, 6:21pm

For a specific branch or tag:

git clone --depth 1 --branch <tag_name> <repo_url>

paperdigits · December 27, 2023, 6:31pm

You have a dirty git tree. Run git status to see what’s changed and if you don’t want to keep any of it, run git reset --hard.

Soupy · December 28, 2023, 12:38am

Thanks, that solved those issues.

Unfortunately, when running git submodule update --init --recursive I still hit the same original error when cloning into tests/integration

Cloning into '/home/tim/src/darktable/src/tests/integration'...
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
error: 2357 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output
fatal: clone of 'https://github.com/darktable-org/darktable-tests.git' into submodule path '/home/tim/src/darktable/src/tests/integration' failed
Failed to clone 'src/tests/integration' a second time, aborting

Do I need tests/integration? Is there a way to build without that submodule and still have it function correctly?

paperdigits · December 28, 2023, 2:04am

@Soupy how about

export GIT_TRACE_PACKET=1
export GIT_TRACE=1
export GIT_CURL_VERBOSE=1

Then try again.

Soupy · December 28, 2023, 2:58am

I tried that to no avail.
But miraculously, after fooling around with the ssh and git configs for the hundredth time, it worked!

Still getting the cmake error when trying to build.

Soupy · December 28, 2023, 11:38am

Darktable successfully built!
Deleting ~/.local/bin solved the cmake error. I may or may not have bricked stable diffusion in the process of doing that, but that’s a problem for another day.

Under the logo it says I have version dd604c164
Does that look right?
git status is happy

Is there a command to check which dependencies might be missing?

The only weird thing is that all the font in lighttable looks bold.