Create lens calibration data for lensfun

diopter · April 5, 2024, 4:22pm

Camera: Nikon Z8, firmware 2.0.
Lens: Nikkor 1000mm F11 Reflex (Reflector) attached with an FTZ II adapter.

I have a similar directory structure:

~/lens_calibrate/vignetting$ ls
10 100 12 15 20 30 50 6 8 exported inf

~/lens_calibrate$ find vignetting/
vignetting/
vignetting/inf
vignetting/inf/_DSC8099.NEF
vignetting/6
vignetting/6/_DSC8108.NEF
vignetting/30
vignetting/30/_DSC8102.NEF
vignetting/20
vignetting/20/_DSC8103.NEF
vignetting/15
vignetting/15/_DSC8104.NEF
vignetting/10
vignetting/10/_DSC8106.NEF
vignetting/exported
vignetting/exported/vignetting.xmp
vignetting/8
vignetting/8/_DSC8107.NEF
vignetting/12
vignetting/12/_DSC8105.NEF
vignetting/100
vignetting/100/_DSC8100.NEF
vignetting/50
vignetting/50/_DSC8101.NEF

Peter · April 5, 2024, 4:35pm

Can you zip the whole directory structure and send me via a file host?

diopter · April 5, 2024, 4:49pm

Directory structure;

with the NEF files := 443 MB zip file.
without the NEF files := 134 KB zip file.
with a single (inf) NEF file := 45 MB zip file.

Not sure what ‘via a file host’ entails. Is that a transfer mechanism via the forum?

Peter · April 5, 2024, 4:52pm

Nr 1 please.
Google drive, Dropbox, WeTransfer.

diopter · April 5, 2024, 5:12pm

Uploaded to dropbox.

Forgive my ignorance, but can I send you the link (as a private message; not sure if that option is available on this forum) or are you expecting the link to be posted in this thread?

Peter · April 5, 2024, 5:16pm

Doesn’t matter. You choose.

diopter · April 5, 2024, 5:20pm

Seems like I haven’t spent enough time on the forum to reach the level to send private messages.

Dropbox download: Dropbox - lens_calibrate.zip - Simplify your life

Fair warning: I have to wait for the moon to come out to find infinity on this lens. As such, NEF file in vignetting/inf folder is not quite infinity (it’s currently a 10 mile focus distance; the farthest visible object I was able to focus on).

Peter · April 5, 2024, 6:06pm

I got some errors too, but it seems like I could generate a file.
lensfun.zip (591 Bytes)

I converted to dng first, because I built my Lensfun system years ago before Z8 was supported in darktable. Shouldn’t matter.

You will have to edit the focal length, the model and mount yourself.

diopter · April 5, 2024, 6:32pm

Thanks for helping out with this.

Odd that it would work on your end. Could be you’re on a different distro (or even windows)? I’ll try with DNG shortly.

The darktable package on debian 12 (or fedora 39 for that matter) has no Z8 profile in /usr/share/darktable/rawspeed/cameras.xml. I had to add that manually, and I don’t even recall where it came from (perhaps I copied it from darktable’s github HEAD repo):

    <Camera make="NIKON CORPORATION" model="NIKON Z 8" mode="14bit-compressed">
            <ID make="Nikon" model="Z 8">NIKON Z 8</ID>
            <CFA width="2" height="2">
                    <Color x="0" y="0">RED</Color>
                    <Color x="1" y="0">GREEN</Color>
                    <Color x="0" y="1">GREEN</Color>
                    <Color x="1" y="1">BLUE</Color>
            </CFA>
            <Crop x="0" y="0" width="0" height="0"/>
            <Sensor black="1008" white="15892"/>
            <ColorMatrices>
                    <ColorMatrix planes="3">
                            <ColorMatrixRow plane="0">11423 -4564 -1123</ColorMatrixRow>
                            <ColorMatrixRow plane="1">-4816 12895 2119</ColorMatrixRow>
                            <ColorMatrixRow plane="2">-210 1061 7282</ColorMatrixRow>
                    </ColorMatrix>
            </ColorMatrices>
    </Camera>

Starting with a fresh debian 12 install, I had to install the following packages to get lens_calibrate.py to run:

$ grep " install " /var/log/dpkg.log | awk ‘{ print $4 }’
python3-pkg-resources:all
python3-numpy:amd64
mime-support:all
libboost1.74-dev:amd64
libboost-dev:amd64
libexpat1-dev:amd64
libimagequant0:amd64
libopenblas0-pthread:amd64
liblbfgsb0:amd64
libopenblas0:amd64
libopenblas-pthread-dev:amd64
libopenblas-dev:amd64
zlib1g-dev:amd64
libpython3.11-dev:amd64
libpython3-dev:amd64
libpython3-all-dev:amd64
libraqm0:amd64
libxsimd-dev:amd64
python3-lib2to3:all
python3-distutils:all
python3-all:amd64
python3.11-dev:amd64
python3-dev:amd64
python3-all-dev:amd64
python3-gast:all
python3-beniget:all
python3-decorator:all
python3-olefile:all
python3-pil:amd64
python3-ply:all
python3-pythran:amd64
python3-scipy:amd64
exiv2:amd64
libboost-python1.74.0:amd64
python3-py3exiv2:amd64
python3-pypdf2:all
libcolord-gtk1:amd64
libgraphicsmagick-q16-3:amd64
liblensfun-data-v1:all
liblensfun1:amd64
liblua5.4-0:amd64
libosmgpsmap-1.0-1:amd64
libportmidi0:amd64
libpugixml1v5:amd64
darktable:amd64
imagemagick-6.q16:amd64
imagemagick:amd64
libnetpbm11:amd64
netpbm:amd64
zip:amd64

I have more of these older lenses so I ultimately would like to be to get this working on my end. I’ll follow up if/when I make progress.

Thanks again

kmilos · April 5, 2024, 6:34pm

F39 was updated to 4.6.1, so that is no longer true.

Peter · April 5, 2024, 6:40pm

Ubuntu three years ago.

kmilos · April 5, 2024, 6:44pm

Btw, your Z 8 NEFs are of the traditional lossless kind?

diopter · April 5, 2024, 7:37pm

Ok. I had to manually update cameras.xml shortly after the Z8 came out, and probably missed that the rpm eventually included it. Currently on 4.6.1-2.

As opposed to the HEIF format? I had tried HEIF some time ago but darktable didn’t seem capable of reading those, which I attributed to perhaps proprietary (Nikon) compression. Really have no idea why it didn’t work and simply went with what did work. I have no jpeg included in the camera settings, just raw. Off topic, but I’m curious why darktable (or rawtherapee) is unable to read .NEFX files resulting from pixel shifting (using NX Studio), which I currently attribute to my inability to fathom pixel shifting in these software packages, or my google-fu is lacking.

kmilos · April 5, 2024, 8:46pm

No, as opposed to HE raw compression. Bit I guess that’s not the case since you can load the NEFs into darktable…

diopter · April 5, 2024, 9:47pm

I see what was and still is causing lens_calibrate.py to hang. Running

lens_calibrate.py vignetting

as user ‘lensfun’:

$ ps aux|grep darkta
lensfun 12439 4.6 2.3 522752 92416 pts/2 Sl+ 14:42 0:01 darktable-cli vignetting/inf/_DSC8099.NEF vignetting/exported/vignetting.xmp vignetting/exported/_DSC8099.ppm --width 250 --core --configdir /tmp/lenscal_rn25y6cp --conf plugins/lighttable/export/iccprofile=image --conf plugins/lighttable/export/style=none

darktable-cli is hanging.

$ strace -p 12439
strace: Process 12439 attached
futex(0x563563b37c70, FUTEX_WAIT_PRIVATE, 2, NULL

Basically blocked. If I kill that process 12439 (which is different than ctrl-c on the python script) I get output from the terminal running the python script:

[rawspeed] rawspeed::Camera(const pugi::xml_node&)::<lambda()>, line 69: Attribute ‘supported’ has unknown value.

Could be a red herring but the Z8 (or Z9) has no ‘supported’ attribute in /usr/share/darktable/rawspeed/cameras.xml.

Remember that I had to add the Z8 definition to cameras.xml (on debian 12).

Doesn’t make a difference if I use a ‘supported’ (and ‘mode’) attribute from another camera; something else is missing that’s hanging darktable-cli, if that’s even what’s causing darktable-cli to hang.

I ran lensfun-update-data which output:

Reading http://lensfun.sourceforge.net/db/versions.json …
Reading http://wilson.bronger.org/lensfun-db/versions.json …

/usr/share/lensfun/version_1/mil-nikon.xml

has no Z8 definition. It does have a Z9 definition.

/var/lib/lensfun-updates/version_1/mil-nikon.xml

does have a Z8 definition:

<camera>
    <maker>Nikon Corporation</maker>
    <maker lang="en">Nikon</maker>
    <model>Nikon Z 8</model>
    <model lang="en">Z 8</model>
    <mount>Nikon Z</mount>
    <cropfactor>1</cropfactor>
</camera>

Thus, on debian 12, I can get lensfun with the Z8, but darktable seems to be missing the Z8, and manually adding it to …rawspeed/cameras.xml doesn’t seem to help.

lensfun-update-data seems to be broken on fedora 39:

lensfun-update-data
Traceback (most recent call last):
File “/usr/bin/lensfun-update-data”, line 46, in
import lensfun
ModuleNotFoundError: No module named ‘lensfun’

Curious if others can successfully run lensfun-update-data on fedora (39 or otherwise).

I know I have to convert my NEF files to DNG and try that with lens_calibrate.py.

diopter · April 5, 2024, 10:39pm

Just tried HE (again). darkroom can’t read those files. They show up in lighttable as a preview, but when opening in darkroom it reports “failed to read camera whitebalance information from <rawfile>”, which is what I suspected had something to do with some proprietary compression. I’m guessing you’re using other software to open HE files?

diopter · April 6, 2024, 3:20am

Curious if anybody has lens_calibrate.py working on a recent distro, notably, with recent python. I managed to get around the darktable errors on debian 12 by running lens_calibrate.py on fedora 39, which almost seemed to work. It finally failed with errors such as:

…
Processing vignetting/30/_DSC8102.NEF …
Generating vignetting data for vignetting/exported/_DSC8102.pgm …
Processing vignetting/10/_DSC8106.NEF …
Generating vignetting data for vignetting/exported/_DSC8106.pgm …
concurrent.futures.process._RemoteTraceback:
“”"
Traceback (most recent call last):
File “/usr/lib64/python3.12/concurrent/futures/process.py”, line 263, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/lensfun/lens_calibrate/./lens_calibrate.py”, line 961, in create_vignetting_correction
calculate_vignetting(pgm_file, input_file, exif_data, distance)
File “/home/lensfun/lens_calibrate/./lens_calibrate.py”, line 800, in calculate_vignetting
plot_pdf(gp_filename)
File “/home/lensfun/lens_calibrate/./lens_calibrate.py”, line 520, in plot_pdf
subprocess.check_call(cmd, stdout=DEVNULL, stderr=subprocess.STDOUT)
File “/usr/lib64/python3.12/subprocess.py”, line 408, in check_call
retcode = call(*popenargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib64/python3.12/subprocess.py”, line 389, in call
with Popen(*popenargs, **kwargs) as p:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/lib64/python3.12/subprocess.py”, line 1026, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File “/usr/lib64/python3.12/subprocess.py”, line 1824, in _execute_child
and os.path.dirname(executable)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “”, line 181, in dirname
TypeError: expected str, bytes or os.PathLike object, not NoneType
“”"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/home/lensfun/lens_calibrate/./lens_calibrate.py”, line 1295, in
main()
File “/home/lensfun/lens_calibrate/./lens_calibrate.py”, line 1288, in main
run_vignetting()
File “/home/lensfun/lens_calibrate/./lens_calibrate.py”, line 1007, in run_vignetting
if f.result():
^^^^^^^^^^
File “/usr/lib64/python3.12/concurrent/futures/_base.py”, line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File “/usr/lib64/python3.12/concurrent/futures/_base.py”, line 401, in __get_result
raise self._exception
TypeError: expected str, bytes or os.PathLike object, not NoneType

I’m not sure if it’s due to differences in python’s concurrent/futures code between when lens_calibrate.py was written and now, but concurrent/futures errors always show up, no matter which format the image files have, or which distro (fedora 39 or debian 12) I use.

Wouldn’t be the first time I’d have to work with an older distro (still have a 20-some year old distro with now obsolete drivers to access Philips ToUCams, if anybody remembers those CCD webcams).

Just curious if anybody has lens_calibrate.py working on something recent, or else I’ll have to try an older distro (@Peter; happen to know which ubuntu release it worked for you?).

Thanks

Peter · April 6, 2024, 6:31am

You have the exact install here for VirtualBox Ubuntu21.zip - Google Drive

Password abc123

The install is not big, so if you have a lot of raw files you will need to increase the disk space or have the raw files stored on another disk.

kmilos · April 6, 2024, 7:21am

No. That’s why I said make sure you use lossless NEF only.

diopter · April 6, 2024, 10:35pm

Using that Ubuntu21 image, I was able to run lens_calibrate.py myself. I don’t have a tool to convert to dng (supposedly need Adobe DNG), but just for testing purposes I used dcraw to convert nef to tiff and lens_calibrate.py was able to work with tiff.

I got somewhat different vignetting correction values than what I had received from Peter. Regardless, I incorporated the vignetting correction values in lensfun’s mil-nikon.xml for a new lens definition, which then shows up in darktable’s lens correction module. However, it doesn’t look like any vignetting corrections are occurring. I tried both Peter’s and my own correction values, but I don’t see any difference. Could be an issue with the calibration shots I took, or it could be an issue with darktable, or even lens_calibrate.py. It’s something I’ll continue to look into.

In any case, thanks for all the help as I now have the ability to run lens_calibrate.py on my own.