Create lens calibration data for lensfun

When I have taken all raw samples for vignetting, I run
exiftool -d %Y_%m_%d '-filename<$FocalLength/$FocusDistanceUpper/%f.%e' .

SkÀrmbild frÄn 2023-04-12 10-39-38

How should I write in Terminal to not get m and mm?

And this one is not a question but something I find helpful for Canon cameras with Magic Lantern. I use it with my 6D. A lua script to load in the camera to take the raw samples fast. This script takes a sample from f/3.5 at every aperture until f/16. Then it jumps to the last one at f/45.

-- f3.5
function test()

    camera.aperture.value = 3.5
    camera.shoot()
    camera.aperture.value = 4.0
    camera.shoot()
    camera.aperture.value = 4.5
    camera.shoot()
    camera.aperture.value = 4.9
    camera.shoot()
    camera.aperture.value = 5.6
    camera.shoot()
    camera.aperture.value = 6.4
    camera.shoot()
    camera.aperture.value = 7.0
    camera.shoot()
    camera.aperture.value = 8.0
    camera.shoot()
    camera.aperture.value = 9.1
    camera.shoot()
    camera.aperture.value = 9.9
    camera.shoot()
    camera.aperture.value = 11
    camera.shoot()
    camera.aperture.value = 13
    camera.shoot()
    camera.aperture.value = 14
    camera.shoot()
    camera.aperture.value = 16
    camera.shoot()
    camera.aperture.value = 45
    camera.shoot()
end

keymenu = menu.new
{
    name = "f/3.5",
    select = function(this) task.create(test) end,
}

exiftool can use Perl alike expressions. The following would strip the space and mm from the FocalLength tag:

${FocalLength;s/ mm//}

Here’s a simple example:

[stasis] jade ~/Temp $ exiftool -a -s -G -p '$FocalLength' tst.1.nef 
50.0 mm
[stasis] jade ~/Temp $ exiftool -a -s -G -p '${FocalLength;s/ mm//}' tst.1.nef 
50.0

Your, somewhat simplified, command:

-r--r----- 1 jade jd 31730789 sep 16  2021 tst.1.nef
-r--r----- 1 jade jd 13172736 jan 11  2022 tst.2.arw
-r--r----- 1 jade jd 25034752 jan 11  2022 tst.3.arw
[stasis] jade ~/Temp $ exiftool -a -s -G -d %Y_%m_%d '-filename<${FocalLength;s/ mm//}/%f.%e' .                              
    1 directories scanned
    3 directories created
    3 image files updated
[stasis] jade ~/Temp $ ls -l
total 12
drwxr-x--- 2 jade jd 4096 apr 13 10:47 28.0
drwxr-x--- 2 jade jd 4096 apr 13 10:47 35.0
drwxr-x--- 2 jade jd 4096 apr 13 10:47 50.0

There’s more info on the exiftool site:

Thanks!

exiftool -d %Y_%m_%d '-filename<${FocalLength;s/ mm//}/${FocusDistanceUpper;s/ m//}/%f.%e' .

Worked great!

I am trying to pass lensfun info for my Nikkor Ai-S 500mm f/4 P ED plus the Nikon TC1.6 (which makes it an 800mm able to partially AF).

I cannot though manage to install py3exiv2 via pip install so this new lens_calibrate script is not able to run. I can run the old scrip the calibraty.py from an older walk through I found in lensfun info on the Internet. It creates a lensfun.xml successfully and I was wondering if this could also do the job.

It seems that pip3 install py3exiv2 brakes as it cannot find a .h file pyconfig.h. boost.python libraries seem to be installed in Tumbleweed

I use lens_calibrate.py. Is that one old? What am I missing? Support for CR3 files?

lens_calibrate.py is the latest one. The calibrate.py I used initially is found on this tutorial/walkthrough https://wilson.bronger.org/lens_calibration_tutorial/

In the mean time I managed to run lens_calibrate.py in another pc running Ubuntu 22.04 and successfully create a lensfun.xml file for both lens configuration.

Yet the TC1.6AF converter (I made another property in the lens.conf file as name = converter but it did not seem to work) does not pass the correct information to the camera (It appears that exiv2 recognises this lens as “Nikon AF Zoom-Nikkor 70-210mm f/4”) and the results came out to be unreadable by RT. It does not read even the crop factor correctly and the distortion, chromatic aberration, and vignetting are greyed out not able to select them.

As far as the 500mm is concerned it came out as a good .xml file but I did not pull the correct a, b and c numbers for distortion. Even the picture I took of a building is not correct or the Hugin Calibrate Lens new software (which seems to find lines automatically) did recognise more lines than I thought that it would.
Vignetting and tca seems to be right.

I am attaching those two .xml files just for reference
lensfun_.tar.gz (923 Bytes)

If you run for the both files 500 and 500 + tc

exiv2 -pt yourfile.NEF |grep -i 'lens\|model\|make'

What do you get?

I have forced lens recognition with my Sigma 500/4.5 that only reported 500mm as lens name in Exif.

You don’t own a Nikon AF Zoom-Nikkor 70-210mm f/4 I suppose?

What does your distortion correction picture look like?

500mm:

Exif.MakerNote.Offset Long 1 13448
Exif.MakerNote.ByteOrder Ascii 3 MM
Exif.Nikon3.LensType Byte 1 (0)
Exif.Nikon3.Lens Rational 4 500mm F4
Exif.Nikon3.LensFStops Undefined 4 5
Exif.NikonLd3.LensIDNumber Byte 1 Nikon Nikkor 500mm f/4 P ED IF
Exif.NikonLd3.LensFStops Byte 1 F5.0

800mm (TC):

Exif.MakerNote.Offset Long 1 13448
Exif.MakerNote.ByteOrder Ascii 3 MM
Exif.Nikon3.LensType Byte 1 (0)
Exif.Nikon3.Lens Rational 4 70-210mm F4
Exif.Nikon3.LensFStops Undefined 4 6
Exif.NikonLd3.LensIDNumber Byte 1 Nikon AF Zoom-Nikkor 70-210mm f/4
Exif.NikonLd3.LensFStops Byte 1 F6.0

Attached the Hugin Calibrate Lens line detection for the 500mm picture I took. The window lines bellow I suppose are the ones that made the problem with distortion prominent.
I am also attaching the RT exported image



for the same lens “corrections” it did for that same profile.

Run the same script but change -pt to -pv to get the LensID nr instead of text. There might be a workaround for you using .exiv2 config file to force lens recognition.

When I create distortion profile I use this target https://youtu.be/G1hvBAiT_Xs?si=2FIcNizssXKhyXju

1 Like

No luck.

I did the following:

exiv2 -pt NIKON\ D4_03-Jun-21_5189_8994.nef |grep -i ‘lens|model|make|focal’

which o/ps"

Exif.MakerNote.Offset Long 1 13448
Exif.MakerNote.ByteOrder Ascii 3 MM
Exif.Nikon3.LensType Byte 1 (0)
Exif.Nikon3.Lens Rational 4 70-210mm F4
Exif.Nikon3.LensFStops Undefined 4 6
Exif.NikonLd3.FocalLength Byte 1 146.7 mm
Exif.NikonLd3.LensIDNumber Byte 1 Nikon Nikkor 500mm f/4 P ED IF + TC1.6AF
Exif.NikonLd3.LensFStops Byte 1 F6.0
Exif.NikonLd3.MinFocalLength Byte 1 71.3 mm
Exif.NikonLd3.MaxFocalLength Byte 1 207.5 mm
Exif.NikonLd3.MaxApertureAtMinFocal Byte 1 F4.0
Exif.NikonLd3.MaxApertureAtMaxFocal Byte 1 F4.0
Exif.NikonFl3.FlashFocalLength Byte 1 n/a
Exif.Photo.FocalLengthIn35mmFilm Short 1 145.0 mm

And:

exiv2 -pv NIKON\ D4_03-Jun-21_5189_8994.nef |grep -i ‘lens|model|make|focal’

with o/p:

0x0001 MakerNote Offset Long 1 13448
0x0002 MakerNote ByteOrder Ascii 3 MM
0x0083 Nikon3 LensType Byte 1 0
0x0084 Nikon3 Lens Rational 4 700/10 2100/10 40/10 40/10
0x008b Nikon3 LensFStops Undefined 4 72 1 12 0
0x000b NikonLd3 FocalLength Byte 1 117
0x000c NikonLd3 LensIDNumber Byte 1 3
0x000d NikonLd3 LensFStops Byte 1 72
0x000e NikonLd3 MinFocalLength Byte 1 92
0x000f NikonLd3 MaxFocalLength Byte 1 129
0x0010 NikonLd3 MaxApertureAtMinFocal Byte 1 48
0x0011 NikonLd3 MaxApertureAtMaxFocal Byte 1 48
0x000c NikonFl3 FlashFocalLength Byte 1 0
0xa405 Photo FocalLengthIn35mmFilm Short 1 145

And changed my .exiv2 conf file to:

[nikon]
3=Nikon Nikkor 500mm f/4 P ED IF + TC1.6AF
92=800mm
129=800mm
145=800mm

Only the byte value/ tag value of 3 is “seen” by exiv2 program. All the rest are ignored. Even just editing and leaving the

3=Nikon Nikkor 500mm f/4 P ED IF

which refers to the original lens without the TC the exiv2 program does not overwrite the rest of the exif properties.
I did not go further digging into tags to see if some other tag value is recognised by this process.
All changes were viewed directly on the Geeqe image viewer that seems to use exiv2 to refer to exif data. To be sure I did run occasionally the exiv2 tool directly.
Finally I did put the first line after the [nikon] declaration as the last line in the .exiv2 config file which made no difference (the only line declaration that is recognised).

Thank you for the info about the distortion on the link. It is clear to me now how to take sample photos. That will complete the process of lensfut.xml as soon as I go out and take some more pictures for the 500mm lens.
For the TCed 500mm I have to rest my case.

I must admit that Nikon is not what I am best at.
Perhaps @paperdigits

There may be some clues here https://github.com/Exiv2/exiv2/blob/130064dc76d2494b6fd473b293f2f32d36eb88f3/src/nikonmn_int.cpp#L2077

Hey @markman8 can you share the raw file that has the tele converter?

Hi @paperdigits

here it is, thank you for the interest.
NIKON D4_03-Jun-21_5189_8994.nef (19.0 MB)

Super, thanks!

Can you do exiv2 --verbose --version --grep config_path and make sure your exiv2 config file is in the correct location with the correct file name?

For me, it works fine:

~/Downloads 
❯ exiv2 -pv NIKON\ D4_03-Jun-21_5189_8994.nef| grep -i lens
LensFocalLength: 145mm 
LensType: AF Zoom 70-210mm f/4 
0x0083 Nikon3       LensType                    Byte        1  0
0x0084 Nikon3       Lens                        Rational    4  700/10 2100/10 40/10 40/10
0x008b Nikon3       LensFStops                  Undefined   4  72 1 12 0
0x000c NikonLd3     LensIDNumber                Byte        1  3 <--- YES THIS ONE
0x000d NikonLd3     LensFStops                  Byte        1  72

~/Downloads 
❯ cat ~/.exiv2          
[nikon]
126=Robin's Sigma Lens
#0=Viltrox,Viltrox AF 85/1.8 Z
3=Some funky shit

~/Downloads 
❯ exiv2 -pt NIKON\ D4_03-Jun-21_5189_8994.nef| grep -i lens
LensFocalLength: 145mm 
LensType: AF Zoom 70-210mm f/4 
Exif.Nikon3.LensType                         Byte        1  (0)
Exif.Nikon3.Lens                             Rational    4  70-210mm F4
Exif.Nikon3.LensFStops                       Undefined   4  6
Exif.NikonLd3.LensIDNumber                   Byte        1  Some funky shit
Exif.NikonLd3.LensFStops                     Byte        1  F6.0

~/Downloads 
❯ nano ~/.exiv2

~/Downloads took 7s 
❯ cat ~/.exiv2                                             
[nikon]
126=Robin's Sigma Lens
#0=Viltrox,Viltrox AF 85/1.8 Z
3=Nikon Nikkor 500mm f/4 P ED IF + TC1.6AF

~/Downloads 
❯ exiv2 -pt NIKON\ D4_03-Jun-21_5189_8994.nef| grep -i lens
LensFocalLength: 145mm 
LensType: AF Zoom 70-210mm f/4 
Exif.Nikon3.LensType                         Byte        1  (0)
Exif.Nikon3.Lens                             Rational    4  70-210mm F4
Exif.Nikon3.LensFStops                       Undefined   4  6
Exif.NikonLd3.LensIDNumber                   Byte        1  Nikon Nikkor 500mm f/4 P ED IF + TC1.6AF
Exif.NikonLd3.LensFStops                     Byte        1  F6.0

You mentioned RawTherapee earlier, are you using a snap or flatpak or some other format that would prevent exiv2 from reading the config file?

What distro or OS are you using?

I haven’t used RawTherapee in a while, and I don’t know if the version I pulled down has lensfun support compiled in. I didn’t see it.

Meanwhile, what needs to happen is that the name you put in the exiv2 config file needs to match the name in the lensfun profile. From the profiles you provided earlier in the thread, the profile lensfun_800mm_new.xml lists Nikon AF Zoom-Nikkor 70-210mm f/4 as the lens name, probably because it uses exiv2 to read the metadata of the raw files. You probably want to change that name to Nikon Nikkor 500mm f/4 P ED IF + TC1.6AF as well.

The .exiv2 file I did edit is the right one.

exiv2 0.28.1
config_path=/home/markman8/.exiv2

Here is the info about my machine

Linux pc-57.home 6.6.6-1-default #1 SMP PREEMPT_DYNAMIC Mon Dec 11 09:46:39 UTC 2023 (a946a9f) x86_64 x86_64 x86_64 GNU/Linux

Running OpenSUSE Tumbleweed.

RawTherapee 5.9 flatpaked. Found in the installation dir of RT and edited the options file so that:

[Lensfun]
DBDirectory=/home/markman8/.local/share/lensfun/updates/version_1

I did run lensfun-update-data and then went to its installation directory (~/.local/share/lensfun/updates/version_1) and added mylensfun.xml file that contains both entries for 500mm and 500mm+TC. RawTherepee does see that file since when I automatically try to recognise at least the 500mm it does pass the right info as shown in the pictures I uploaded (PrtScrn) on previous post in this thread.

The entries in the mylensfun.xml file for the 500mm+TC <lens><model>Description<model><lens> are not matching the ones I have put in the .exiv2 configuration file (Where Description is the 3=value in the config file). I will try this as well in this day and let you know. That might be the case.

What I am afraid of though is that the lens_calibrate script in the first place uses exiv2 data from the RAW image provided and calculates the values for tca and vignetting accordingly. That is for the found focal length (which for the 800mm (500mm+TC) finds 145mm in the RAW file as I saw there is no entry in the nikonmn_int.cpp for the combination of 500mm+TC lens) and thus those o/p values in the .xml file might be wrong. The numbers for the distortion are up to me and Hugin so I suppose it does not matter.

I am uploading the mylensfun.xml here for reference. I am about to change the model in the .xml file to that I have in .exiv2 so they match and let you know.
mylensfun.tar.gz (715 Bytes)

I forgot to mention that in the atached mylensfun.xml file I have intentionally changed the focal entries from focal=“145.0” to focal=“800.0”

*Edit:
I did put the info about the lensID both in lensfun.xml and .exiv2 to be the same for the 500+TC lens. Now automatically the RT recognises the lens as 70-210mm but in the manual lens list it appears with the name I gave to the files. Yet as seen from the PrntScrn manually selecting the lens from the list it behaives as it did in the beginning, thus all options greyed out (distortion, chromatic aberration and vignetting) and a message saying that crop factor is wrong.

Since you’re using the flatpak, I don’t think exiv2 will look for its config file in the same place as exiv2 that is installed on your system. I remember trying to find the right location for this file, but I don’t think I ever found an answer.

Is RT packaged for tumbleweed?

Yes RT 5.9 is packaged for Tumbleweed.

But before I go and install that non flatpak version of it to my machine could you please explain what the correct behaviour of RT would be?
I guess it should first run exiv2 (from /usr/bin) override my lensID by the value of whatever I put 3 toy be equal to and then look in lensfun dir for a match? Thus the entries I mentioned in nikonmn_int.cpp do not matter?

Well I did go and install the packaged RT 5.9 (changed the ~/.config/RawTherapee-AppImage/options configuration file so it can also apply/retrieve lensfun info and the results came to be the same as the flatpak one.
Note that also the Geeqie (which is not related to lensfun info) behaves the same:
Screenshot_20231217_205655

Not sure if this is the right place to post, but I’m trying to compensate for vignetting on some old non-CPU lenses, ultimately wanting to have darktable compensate. This is effectively my first attempt at using lens_calibrate.

lens_calibrate.py init

creates the distortion, tca and vignetting folders.

I’ve created an entry for the lens in lenses.conf.

In vignetting, I’ve created subfolders for inf and various focus distances and placed the raw image files corresponding with each focus distance. I’m not even sure if the different focus distances make sense since no focus distance(s) will ultimately be available, as the lenses are non-CPU (darktable always reports 0.01m focus distance).

When I then run;

lens_calibrate.py vignetting

I get a deprecation warning (running on debian 12), which could be benign, but otherwise the script doesn’t seem to be doing anything and just seems to hang indefinitely. Output is:

lens_calibrate.py vignetting
lens_calibrate.py:50: DeprecationWarning: Please use leastsq from the scipy.optimize namespace, the scipy.optimize.minpack namespace is deprecated.
from scipy.optimize.minpack import leastsq
Processing vignetting/inf/_DSC8099.NEF 


If I hit ctrl-c, it moves on to the other focus distance images with the same (hanging) behavior.

A new subfolder in vignetting called ‘exported’ is created containing vignetting.xmp, which contains entries:

darktable:operation=“basecurve”

and similar entries for colorin and colorout, but effectively nothing that would seem to relate to vignetting or other types of calibration.

Any tips on how to debug or proceed would be appreciated.

Which camera are you using?
I created a lens profile for Canon EF 85/1.2L USM today. This lens reports no focus distance, so I manually put them in different folders.