Error attempting to create custom noise profile [SOLVED]

Per the tutorial at https://pixls.us/articles/how-to-create-camera-noise-profiles-for-darktable/, I’m trying to create a noise profile for my Canon T8i / 850D. I’m running Windows but have a Linux Mint 20.03 VM.* I’ve installed darktable 4 from darktable_4.0.0-1.1_amd64.deb, downloaded at https://software.opensuse.org/download.html?project=graphics:darktable&package=darktable per the tutorial. I’ve installed (or already have) the dependencies, as far as I can tell: git, gcc, make, gnuplot, convert, darktable-cli, exiv2 and gphoto.

* If there’s a way to create the profile natively on Windows, that would be logistically simpler. However, I don’t have a build environment on Windows. Prior to this computer I ran Linux for years and occasionally built packages but I’m no expert.

Anyway, since /usr/lib/darktable/tools was empty (no darktable-gen-noiseprofile), I thought I needed to build the noise tools, so using git I retrieved the source into /var/src/darktable. However, in /var/src/darktable/tools/noise I found a darktable-gen-noiseprofile binary.

I shot the gradient images, copied them to the VM, cd-ed to them, appended /var/src/darktable/tools/noise to my path (for convenience) and ran darktable-gen-noiseprofile -d $(pwd). That went OK until the end:

===> Prepare profiling job
--> Remove previous presets
--> Ready to profile images

NOTE: This process takes some time and a lot of memory and disc space
up-to several gigabytes, depending on the number of ISO settings and
the size of the RAW files.
ERROR: Could not find darktable-noiseprofile tool in /var/src/darktable/tools/noise

And it’s right, it’s not there. But that was supposedly the full darktable deb I installed. Do I need to build something additional? Or install a different DT package?

Actually just before I found darktable-gen-noiseprofile, I tried building what I got via git per the tutorial. But it complained about not finding the compiler and a few other things (that looked possibly like an environment issue). While looking around I found the working darktable-gen-noiseprofile and didn’t further chase building anything.

Any advice?

Thanks.

P.S. - I assume since the T8i / 850D isn’t in the list of supported cameras, there’s no noise profile for it yet… If I’m wrong that would really nice. :wink:

Len,
If you build dt from source, check option

BUILD_NOISE_TOOLS “Build tools for generating noise profiles”

in DefineOptions.cmake. Is it ON or OFF?

Have fun!
Claes in Lund, Sweden

I may bite the bullet, uninstall the binary DT package and see if I can build it from scratch. But of course I’m now tied up in something else for a little bit so it’ll have to wait…

Thanks for the tip.

Have you looked for additional darktable packages? The profile tool might be packaged separately. They are for my distribution, opensuse, in a package “darktable-tools-noise”, also from the graphics:darktable repository.

Well… you’re right – the noise package is available as an extra! I didn’t stop to consider that:

$ apt-cache search darktable | cut -d' ' -f1 | while read pkg
> do
> apt-cache show $pkg | egrep '(Package|Version):'
> done
Package: darktable
Version: 1:4.0.0-0build1~focal
Package: darktable
Version: 3.0.1-0ubuntu1
Package: darktable-dbg
Version: 1:3.8.1-0build2~ubuntu20.04
Package: darktable-tools-basecurve
Version: 1:4.0.0-0build1~focal
Package: darktable-tools-noise
Version: 1:4.0.0-0build1~focal

However the repo DT packages for Mint 20.03 are very old. Do you know if the noise tools have changed? Do I need to search out a more recent PPA or will I be OK with this to get the profile generated? I searched github for darktable-tools-noise but didn’t find anything. My git*-fu is pretty flabby…

I installed DT4 manually just to be on the newest one, but since this VM isn’t my “production” darktable environment I have no problem downgrading it a tad to get the noise tools for this task. Gotta say though at first blush a full version number drop pushes “a tad”. :slight_smile:

I won’t be doing anything else with this VM DT installation but since I’ll be submitting the profile for inclusion I don’t want to provide anything potentially dodgy other 850D owners might use.

Thanks.

Len, it is included in the normal darktable git — you just have to switch an option from OFF to ON. See post #2, above…

Right, gotcha. I’ll definitely do that if I have to build. All things being equal I’d like to find a usable binary installation, but if I have to build I will.

Thanks.

Did you check the repository where you got your dt main package from? For mine, the tools package have the exact same version number as the main package

Yes - They match after all, dpkg confirms that. I obviously misread the apt-cache show darktable output.

After installing the repo packages, they’re all version 4, so that’s good and the noise tools are there (no need to build):

$ dpkg -l | grep darktable
ii  darktable                                  1:4.0.0-0build1~focal               amd64        virtual lighttable and darkroom for photographers
ii  darktable-tools-basecurve                  1:4.0.0-0build1~focal               amd64        The basecurve tool from tools/basecurve/
ii  darktable-tools-noise                      1:4.0.0-0build1~focal               amd64        Noise profiling tools to support new camerasNoise profiling tools to support new cameras

Apparently I need to review apt-cache before further use. :slight_smile:

So I ran /usr/lib/darktable/tools/darktable-gen-noiseprofile -d $(pwd) against my raw images and it apparently did nothing. It appears to possibly be incorrectly calling darktable-cli.

Here’s the full output.

$ /usr/lib/darktable/tools/darktable-gen-noiseprofile -d $(pwd)
===> Check for required tools
--> Check for images handling tools availability
--> Check for images export tools availability
--> Check for profiling tools availability
--> Check for tethering tools availability
--> Check for pdf tools availability

===> Check profiling directory

===> List profiling input RAW images
--> Found ISO 100 image: /home/xxxxx/Pictures/850D_noise_profile/IMG_4490.CR3
(redundant raw file list output clipped for clarity)
--> Found ISO 25600 image: /home/xxxxx/Pictures/850D_noise_profile/IMG_4514.CR3

The script will use existing input RAW images for the profiling. No more
shot will be taken.

===> Checking profiling RAW images correctness + Jpeg export
--> ISO 100:
    /home/xxxxx/Pictures/850D_noise_profile/IMG_4490.CR3
usage: darktable-cli [options] [IMG_1234.{RAW,..}|image_folder/]

options:

  --cachedir <user cache directory>
  --conf <key>=<value>
  --configdir <user config directory>
  -d {all,cache,camctl,camsupport,control,dev,fswatch,imageio,input,
      ioporder,lighttable,lua,masks,memory,nan,opencl,params,perf,demosaic
      pwstorage,print,signal,sql,undo,act_on,tiling,verbose}
  --d-signal <signal>
  --d-signal-act <all,raise,connect,disconnect,print-trace>
  --datadir <data directory>
  --disable-opencl
  -h, --help
  --library <library file>
  --localedir <locale directory>
  --luacmd <lua command>
  --moduledir <module directory>
  --noiseprofiles <noiseprofiles json file>
  -t <num openmp threads>
  --tmpdir <tmp directory>
  --version

All the executables in /usr/lib/darktools/tools are binaries except subr.sh, but I’ve not found anything in it that correlates with this error (although I’ve not been utterly exhaustive). The tools directory is not in my path, but I invoked darktable-gen-noiseprofile with an absolute path.

Any ideas?

Thanks.

See the same problem reported (and fixed) recently. Searching the forum is highly recommended :wink:

Actually I usually (obviously not always!) search but I fired off that question in a rush earlier …and of course, the answer is already there. :upside_down_face: (note to self: slow down)

Thanks for the pointer.

UPDATE: After revising those two scripts I was able to create a profile and have opened a github issue to have it included. Topic subject line here has been edited to include [SOLVED].

1 Like