Compiling Natron yourself using the docker SDK

devernay · July 29, 2019, 5:47am

After installing docker, it’s as easy as:

mkdir builds
docker run -it --rm --mount src="$(pwd)/builds",target=/home/builds_archive,type=bind natrongithub/natron-sdk:latest

and the results are in a directory called builds

The docker image is available from the docker hub Docker Hub

The builds can be easily customized, for example to use you own git repo for Natron:

mkdir builds
docker run -it --rm --mount src="$(pwd)/builds",target=/home/builds_archive,type=bind --env GIT_URL=https://github.com/MyGitLogin/Natron.git --env GIT_BRANCH=mybranch natrongithub/natron-sdk:latest

If you want to make a custom docker image based on anything else than centos6, just modify the base image in the Dockerfile generated by this build script:

github.com

NatronGitHub/Natron/blob/RB-2.3/tools/docker/natron-sdk/build.sh

#!/bin/sh
cd `dirname "$0"`
cp  ../../jenkins/*.sh .
(cd ../../jenkins/; tar cf - include) | tar xf -
export CENTOS=7
export DTS=8
LABEL="natrongithub/natron-sdk${UBUNTU+-ubuntu}${UBUNTU:-}${CENTOS+-centos}${CENTOS:-}${DTS+-dts}${DTS:-}"
env GEN_DOCKERFILE=1 ../../jenkins/include/scripts/build-Linux-sdk.sh > Dockerfile
docker build -t "${LABEL}:latest" . --progress=plain
#docker build --no-cache -t "${LABEL}:latest" .
echo "please execute:"
#echo "docker-squash ${LABEL}:latest"
echo "docker login"
echo "docker tag ${LABEL}:latest ${LABEL}:$(date -u +%Y%m%d)"
echo "docker push ${LABEL}:latest"
echo "docker push ${LABEL}:$(date -u +%Y%m%d)"

Ivanohe73 · August 1, 2019, 7:12pm

Thanks! I was able to make it run and produce a valid tgz installation yesterday.

I am just trying to re-run it and it got stuck forever in a test :

Loading project: /home/tmp/Natron-Tests64/GNUVolador/gnu.ntp
(‘encoder=’, u’fr.inria.openfx.WriteOIIO’)
(‘ocioInputSpace=’, u’linear/Linear’)
(‘ocioOutputSpace=’, u’sRGB’)
script_write_node ==> Rendering started
script_write_node ==> Frame: 1, Progress: 11.1%, 0.7 Fps, Time Remaining: 11 seconds

I will retry running the docker build again but I would like to know if some test are known for being unstable?

Regards,

devernay · August 1, 2019, 8:18pm

Yes, this test gets stuck when I run it too (but the test script kills it after 30mn).
This may be due to the limited number of chores available in docker.
The Baymax test also fails but I did not have time to look into that.
Once you have Natron installed on a real machine you can also rerun the tests from the repo Natron-Tests. Does the gnuvolador pass? Does Baymax pass?

Ivanohe73 · August 1, 2019, 8:57pm

I started 4 build of different commit version since 3 days ago to see if this was a recent problem.

1 out of those 4 build didn’t get stuck and I got :
2019-08-01 20:30:50 *** END unit tests → 0
Unit tests: OK

The 3 other build are stuck at Baymax : two in Natron2_BayMaxProject_20_fixed_switched.ntp and one in /home/tmp/Natron-Tests64/BayMax/Natron2_BayMaxProject_20_fixed.ntp

Since one of those stuck build is the same git commit I was able to build 2 day ago I guess there is something unstable. I will try to run the test individually if I get some time to see if I can reproduce the problem outside of docker

Ivanohe73 · August 1, 2019, 9:22pm

Running the Natron2_BayMaxProject_20_fixed.ntp in GUI with my docker build or the 2.3.15 rc6 I got 2 issues:

There is a lots of warning when loading the project :

[17:12:19.864] Project: WARNING: The node LensDistortion (net.sf.openfx.LensDistortion) version 2.0 was found in the script but was loaded with version 4.0 instead.
[17:12:20.416] Premult1: Could not find a parameter named Source_channels
[17:12:20.417] Shufflecopy1: Could not find a parameter named outputChannelsChoice
…

The video is not right (I saw only the top of the attic and it is warped).

I am able to get the playback to get stuck. It play some frame and at some point it just stop and I can’t stop start the playback again. It is like if the play button didn’t work.

If I use the official Natron 2.3.14 I didn’t get any of those issue.

I make sure to delete all Natron caches before trying different version
rm ~/.cache/INRIA/Natron/* -rf

NOTE : I was sometime able to get the playback to stop responding on my own test project when using other build than the 2.3.14 but was not able to find a good way to reproduce it reliably. Seems this project is big enough to cause the problem to show more

Ivanohe73 · August 3, 2019, 5:42am

I tried building older version of Natron using the docker but was not able to go very far back in time because of build issue.

Is there a place where some portable intermediary build between version 2.3.14 and version 2.3.15rc6 could be found? I could try to see between which version the problem with BayMax appear.

rodlie · August 3, 2019, 10:14am

The project file from Natron_2.3.12_BayMax.zip works fine on latest snapshot available on GitHub. Must be something wrong with the ‘ntp’ in the Tests?

devernay · August 3, 2019, 5:28pm

Natron (and or plugins) should be fixed to load Natron2_BayMaxProject_20_fixed.ntp

This reveals several issues which have to be fixed without changing the ntp file

devernay · August 3, 2019, 5:39pm

Yves,

These warnings also appear in 2.3.14:

Summary

[10:35:35.450] Premult2: Could not find a parameter named Source_channels

[10:35:35.451] Premult1: Could not find a parameter named Source_channels

[10:35:35.451] Shuffle1: Could not find a parameter named outputChannelsChoice

[10:35:35.451] Shuffle1: Could not find a parameter named outputRChoice

[10:35:35.451] Shuffle1: Could not find a parameter named outputGChoice

[10:35:35.451] Shuffle1: Could not find a parameter named outputBChoice

[10:35:35.451] Shuffle1: Could not find a parameter named outputAChoice

[10:35:35.453] Shufflecopy1: Could not find a parameter named outputChannelsChoice

[10:35:35.453] Shufflecopy1: Could not find a parameter named outputRChoice

[10:35:35.453] Shufflecopy1: Could not find a parameter named outputGChoice

[10:35:35.453] Shufflecopy1: Could not find a parameter named outputBChoice

[10:35:35.453] Shufflecopy1: Could not find a parameter named outputAChoice

[10:35:35.892] Reset_Alpha_to_1: Could not find a parameter named outputChannelsChoice

[10:35:35.892] Reset_Alpha_to_1: Could not find a parameter named outputRChoice

[10:35:35.892] Reset_Alpha_to_1: Could not find a parameter named outputGChoice

[10:35:35.892] Reset_Alpha_to_1: Could not find a parameter named outputBChoice

[10:35:35.892] Reset_Alpha_to_1: Could not find a parameter named outputAChoice

[10:35:35.892] WriteOIIO1: Could not find a parameter named clipToProject

[10:35:35.892] WriteOIIO1: Could not find a parameter named outputChannelsChoice

[10:35:35.892] Shuffle1: Could not find a parameter named outputChannelsChoice

[10:35:35.892] Shuffle1: Could not find a parameter named outputRChoice

[10:35:35.892] Shuffle1: Could not find a parameter named outputGChoice

[10:35:35.892] Shuffle1: Could not find a parameter named outputBChoice

[10:35:35.892] Shuffle1: Could not find a parameter named outputAChoice

[10:35:35.893] LensDistortion: Could not find a parameter named k3

[10:35:35.893] LensDistortion: Could not find a parameter named p1

[10:35:35.893] LensDistortion: Could not find a parameter named p2

[10:35:35.893] Shuffle_CopyAlpha: Could not find a parameter named outputChannelsChoice

[10:35:35.893] Shuffle_CopyAlpha: Could not find a parameter named outputRChoice

[10:35:35.893] Shuffle_CopyAlpha: Could not find a parameter named outputGChoice

[10:35:35.893] Shuffle_CopyAlpha: Could not find a parameter named outputBChoice

[10:35:35.893] Shuffle_CopyAlpha: Could not find a parameter named outputAChoice

Here’s how to debug the issue:

load the ntp using 2.3.14 and 2.3.15 side-by-side, preferably not on the same system (use tow PCSs or VMs)
inspect each node output using the viewer, until you find where the two diverge.

Ivanohe73 · August 5, 2019, 10:13pm

Frederic,
I don’t see the warning in 2.3.14 on a centos 7 pc.
Thanks for the tips. I will post an update once I can confirm some of the things. So far I think there is 2 separate issues. If so, I will open bug report for each.

devernay · August 5, 2019, 11:56pm

You can see the warnings in ing the error log (Display->Show Project Error Log)
You can also use the environment variable NATRON_DISK_CACHE_PATH to get two versions of Natron running simultaneously but using different disk caches (it was not documented, I’ll add a section in the doc on environment variables)

Ivanohe73 · August 6, 2019, 3:05am

Ok thanks. I was able to compare the warning and found there is a new one. I open a bug for this issue : LensDistortion issue in Natron 2.3.15rc6 · Issue #416 · NatronGitHub/Natron · GitHub
I open issue Random freeze when rendering · Issue #417 · NatronGitHub/Natron · GitHub for the freeze rendering issue.

I will try to build intermediate version of Natron (from 2.3.14 to 2.3.15rc6) with a normal build and see if I can find the problematic commit. (assuming I can build easily older version of Natron)

devernay · August 6, 2019, 6:14am

The major change is when I fixed the the Python Gil handling

Ivanohe73 · August 22, 2019, 2:21am

While trying to fix an issue I am trying to build a complete tgz from my own repo of Natron and branch and got some error :

docker run -it --rm --mount src="$(pwd)/builds",target=/home/builds_archive,type=bind --env GIT_URL=https://github.com/Ivanohe73/Natron.git --env GIT_BRANCH=rotopaint natrongithub/natron-sdk:latest

./launchBuildMain.sh: line 306: GIT_NATRON: unbound variable

I tried adding a
--env GIT_NATRON=https://github.com/Ivanohe73/Natron.git

but got error:
Unknown plug-in repository URL https://github.com/Ivanohe73/Natron.git

I also forked all the plug-in in case and tried adding other options like GIT_URL_OPENFX_IO , etc but I still get the Unknown plug-in repository error…

rodlie · August 22, 2019, 1:24pm

github.com

NatronGitHub/Natron/blob/RB-2.3/tools/jenkins/launchBuildMain.sh#L203

    
      
          # By default breakpad is enabled
          if [ "${DISABLE_BREAKPAD:-}" != "1" ]; then
              DISABLE_BREAKPAD=0
          else
              echo "DISABLE_BREAKPAD specified: crash reporting support disabled"
          fi
          
          
# Remove any previous jenkins artifacts from previous builds
          if [ -n "${WORKSPACE:-}" ]; then
              if [ -d "${WORKSPACE}/jenkins_artifacts" ]; then
                  rm -rf "${WORKSPACE}/jenkins_artifacts" || true
              fi
          fi
          
          
# Do we need to build Natron ? No if the script was not invoked from a Natron CI project
          if [ -z "${GIT_URL_IS_NATRON:-}" ]; then
              GIT_URL_IS_NATRON=0
              isNatronRepo "$GIT_URL" && GIT_URL_IS_NATRON=1
          fi

github.com

NatronGitHub/Natron/blob/RB-2.3/tools/jenkins/gitRepositories.sh#L21

    
      
          # Natron is distributed in the hope that it will be useful,
          # but WITHOUT ANY WARRANTY; without even the implied warranty of
          # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
          # GNU General Public License for more details.
          #
          # You should have received a copy of the GNU General Public License
          # along with Natron.  If not, see <http://www.gnu.org/licenses/gpl-2.0.html>
          # ***** END LICENSE BLOCK *****
          
          

          
# List here the repositories
          GIT_URL_OPENFX_OPENCV_GITHUB=${GIT_URL_OPENFX_OPENCV_GITHUB:-https://github.com/NatronGitHub/openfx-opencv.git}
          GIT_URL_NATRON_GITHUB=${GIT_URL_NATRON_GITHUB:-https://github.com/NatronGitHub/Natron.git}
          GIT_URL_NATRON_GFORGE=${GIT_URL_NATRON_GFORGE:-git+ssh://natron-ci@scm.gforge.inria.fr/gitroot/natron/natron.git}
          GIT_URL_OPENFX_IO_GITHUB=${GIT_URL_OPENFX_IO_GITHUB:-https://github.com/NatronGitHub/openfx-io.git}
          GIT_URL_OPENFX_MISC_GITHUB=${GIT_URL_OPENFX_MISC_GITHUB:-https://github.com/NatronGitHub/openfx-misc.git}
          GIT_URL_OPENFX_ARENA_GITHUB=${GIT_URL_OPENFX_ARENA_GITHUB:-https://github.com/NatronGitHub/openfx-arena.git}
          GIT_URL_OPENFX_GMIC_GITHUB=${GIT_URL_OPENFX_GMIC_GITHUB:-https://github.com/NatronGitHub/openfx-gmic.git}
          GIT_URL_NATRON_TESTS_GITHUB=${GIT_URL_NATRON_TESTS_GITHUB:-https://github.com/NatronGitHub/Natron-Tests.git}
          
          
# List of known Natron repositories

Ivanohe73 · August 22, 2019, 5:36pm

Thanks @rodlie !

So here is the updated way with docker

to use you own git repo for Natron:

mkdir builds
docker run -it --rm --mount src="$(pwd)/builds",target=/home/builds_archive,type=bind --env GIT_NATRON=https://github.com/MyGitLogin/Natron.git --env GIT_URL=https://github.com/MyGitLogin/Natron.git --env GIT_BRANCH=mybranch natrongithub/natron-sdk:latest /bin/bash

sed -c -i "s/^GIT_URL_NATRON_GITHUB=.*/GIT_URL_NATRON_GITHUB=\$GIT_NATRON/" gitRepositories.sh
./launchBuildMain.sh

devernay · August 27, 2019, 11:33pm

Fixed in tools/jenkins: GIT_URL_IS_NATRON=1 can be used to launch a Natron bui… · NatronGitHub/Natron@a8cc6be · GitHub

If GIT_URL is an unknown Natron repository, GIT_URL_IS_NATRON=1 can be used to force a Natron build.

magdesign · January 2, 2020, 11:31am

Could you please give a short example on how to solve the: Unknown plug-in repository URL

Trying to compile a custom build with:

sudo docker run -it --rm --mount src="$(pwd)/builds",target=/home/builds_archive,type=bind --env GIT_NATRON=https://github.com/NatronGitHub/Natron.git --env GIT_URL=https://github.com/splidje/Natron.git --env GIT_BRANCH=#439 natrongithub/natron-sdk:latest /bin/bash
sed -c -i "s/^GIT_URL_NATRON_GITHUB=.*/GIT_URL_NATRON_GITHUB=\$GIT_NATRON/" gitRepositories.sh
./launchBuildMain.sh

Where would I add the GIT_URL_IS_NATRON=1 command?

rodlie · January 2, 2020, 11:45am

Add GIT_URL_IS_NATRON=1 together with the other env’s on the first line and remove the sed … line. That should work(?), note that I’m not a big docker user.

magdesign · January 2, 2020, 12:02pm

Thanks, using this command started the compiler (splidje build, will not compile):

sudo docker run -it --rm --mount src="$(pwd)/builds",target=/home/builds_archive,type=bind --env GIT_URL_IS_NATRON=1 --env GIT_URL=https://github.com/splidje/Natron.git --env GIT_BRANCH=#439 natrongithub/natron-sdk:latest

or rodlie build:

sudo docker run -it --rm --mount src=“$(pwd)/builds”,target=/home/builds_archive,type=bind --env GIT_URL_IS_NATRON=1 --env GIT_URL=GitHub - rodlie/Natron: Experimental Natron Repository --env GIT_BRANCH=SetDefaultProjectFormat natrongithub/natron-sdk:latest