The ART code seems to be directly derived from RT: Bitbucket
The code tries to split the name into a maker/model pair, but it does not change the trailing portion of the lens name, which is the one that differs in the specific example I have used…
Also, in the call to FindLenses() the ART code does not use the LF_SEARCH_LOOSE flag.
Maybe I should look in some other place in the code?
Thanks!
no, I’m probably misremembering – sorry for the noise
I kind of remember that LF_SEARCH_LOOSE was causing troubles by matching wrong lens.
It was used in 2013, see dt commit lens.c: use loose search · darktable-org/darktable@799bb9f · GitHub and reverted just after lens correction: more consistent behavior of gui and auto-detection · darktable-org/darktable@c7cd6bb · GitHub
I mean … after 6 years … maybe the lensfun code improved? and LF_SEARCH_LOOSE behaves better now?
I will test the LF_SEARCH_LOOSE flag in my code, let’s see if the matching works properly with current LensFun versions…
Thanks for the feedback!
I retired from Exiv2 in September 2019. I have proposed to talk at LGM in Rennes and occasionally answer user questions on GitHub about Exiv2.
I’m not sure I have any good ideas about this. Lens recognition isn’t easy as it isn’t stored in the metadata. The lens has to be deduced from other metadata.
What doesn’t help I think is what appears to be inconsistent syntax in the aperture range in the exiv2 Nikon LensID table. For the same Sigma lens in the Canon lookup, it’s “f/5-6.3”, while in the Nikon table it’s “F5-6.3”. That probably lowers the lensfun score considerably.
RawTherapee seems to be using the MakerNote tag when available: RawTherapee/imagedata.cc at dev · Beep6581/RawTherapee · GitHub
As far as I could see, this seems to give lens names better matching the lensfun DB. I wonder if there is an easy way to translate such code to use EXIV2 instead of the custom RT EXIF code…
@clanmills I know you are taking a very well deserved rest! I just included you in the thread in case you might have some good advice… Thanks!
@Carmelo_DrRaw Thanks. I appreciate being kept aware. I sometimes answer questions on the Exiv2 GitHub site. What I want to avoid is to submit code. The combination of Git, the non-functioning review process and the behaviour of the fuzzing police have broken my will to continue. I’m having a vacation in Australia and NZ at the moment. I’ve submitted a proposal to speak at LGM in Rennes at the end of May.
@ggbutcher I added the exiv2 configuration file ~/.exiv2 in which users can define names for lenses. So there is an existing mechanism for users to fix this for themselves. The table of Nikon lenses was “bought in” and we avoided changing them to ease integration of updates to that table. However, I don’t think the original author supports that code, so we could now change it. Let’s see what ideas emerge from others. My thought was to have a switch “LensFun compatibility” and apply a map from the current lens names to “better”. I’m not a photographer and I don’t know what LensFun does.
However, the biggest impediment to get something done about this is volunteers. If invited to speak at LGM, I will discuss that challenges of maintaining open-source. I’ve solved many issues for Exiv2, however how to attract contributors is a mystery. The guys working on Exiv2 (Luis, Dan and Kevin) are excellent engineers. However they have full-time demanding jobs and other demands on their time. Since I retired in 2014, there have been many 60 hour weeks working on Exiv2. I’m exhausted.
It all boils down to Nikon’s exclusion of a lens nomenclature in their older metadata. For my D7000 and 18-200 zoom, here’s what’s in the file:
glenn@caliente:~/Photography/rawproc$ exiftool -G -n -Lens* DSG_3111.NEF
[MakerNotes] Lens Type : 14
[MakerNotes] Lens : 18 200 3.5 5.6
[MakerNotes] Lens Data Version : 0204
[MakerNotes] Lens ID Number : 139
[MakerNotes] Lens F Stops : 5.33333333333333
[Composite] Lens ID : 8B 40 2D 80 2C 3C FD 0E
[Composite] Lens Spec : 18 200 3.5 5.6 14
The Composite:LensID is the amalgam of tags upon which most softwares’ lookups are based, mostly from a list compiled by Robert Rottmerhusen.
Looks like Nikon finally got the cluebird, here’s a Lens* dump from my Z6-24-70/f4:
glenn@caliente:~/Photography/rawproc$ exiftool -G -n -Lens* DSZ_4168.NEF
[MakerNotes] Lens Type : 0
[MakerNotes] Lens : 24 70 4 4
[MakerNotes] Lens F Stops : 5
[MakerNotes] Lens Data Version : 0800
[MakerNotes] Lens ID : 1
[EXIF] Lens Info : 24 70 4 4
[EXIF] Lens Make : NIKON
[EXIF] Lens Model : NIKKOR Z 24-70mm f/4 S
[EXIF] Lens Serial Number : 20078247
[Composite] Lens Spec : 24 70 4 4 0
Yay, they’re using the EXIF tags now… But still not good, as it still takes a exiftool composite tag to puke a decent nomenclature for an older lens on the FTZ adapter:
glenn@caliente:~/Pictures/2019/2019-10-30 First Light into 70-300 AF-P/NEFs$ exiftool -G -Lens* DSZ_3990.NEF
[MakerNotes] Lens Type : E VR [7]
[MakerNotes] Lens : 70-300mm f/4.5-5.6
[MakerNotes] Lens Data Version : 0800
[MakerNotes] Lens ID Number : 171
[MakerNotes] Lens F Stops : 5.67
[EXIF] Lens Info : 70-300mm f/4.5-5.6
[EXIF] Lens Make :
[EXIF] Lens Model :
[EXIF] Lens Serial Number :
[Composite] Lens ID : AF-P Nikkor 70-300mm f/4.5-5.6E ED VR
[Composite] Lens Spec : 70-300mm f/4.5-5.6 E VR [7]
Lensfun makes the best of it, the match is done by scoring the similarities, field-by-field. I haven’t poked at the guts of LOOSE, but I’d worry about mismatching, especially in variants of the same lens.
Lensfun probably relies on the efforts of the individual calibration contributors, who are probably looking at any of RawTherapee/darktable/exiftool/exiv2, which have different instantiations.
What a mess! I was not aware it was that tricky to get the lens name… is that Nikon-specific, or other camera makers have a similar mess in their metadata?
@Carmelo_DrRaw Yes! Sadly this mess exists with every camera manufacturer. I have no idea why the Exif Standard does not define lens specification.
The Exif Spec defines the “Makernote” tag in which the manufacturer can write arbitrary binary data. Exiv2 can read and rewrite most manufacturer’s “Makernote” tags. However it’s not only undocumented by the manufacturer, the data defining the lens changes for different models. For example, Nikon sometimes use a LensIDNumber (a short integer) and sometimes a string (in ASCII and other encodings). And the same LensIDNumber can be used by several different lenses. Challenging? For sure!
I hope we’ll all meet in Rennes in May to talk about these challenges. Not only metadata challenges, let’s discuss the many challenges facing the Graphics Open-Source community.
My comment is focused on Nikkor lenses, as they’re the topic of the 8-byte Nikon Lens ID endeavor. And, based on a cursory inspection of RawTherapee, darktable, exiv2, and exiftool source codes…
I don’t know the specific history behind this encoding, but it appears to be an open-source fomented thing, in order to provide some order to lens nomenclature sufficient to inform lens correction. Essentially, it’s the construction of a lens ID from 8 other metadata tags, and the maintenance of a large table that maps those IDs to ASCII strings defining unique lens nomenclature.
Thing is, there is more than one source of “canonical” ID-nomenclature mappings. Most seem to have a root in a file created by a fellow named Robert Rottmerhusen, nikonlensid.h. I have a copy of it in rawproc:
#define FMLVERSION "4.4.527.01"
#define FMLDATE "2017-02-27"
to which I added a more friendly lookup function. Of note is that the URL referenced as the file source no longer points to a web server. For the lens that is the subject of this thread, here’s the table entry:
{0x82,0x34,0x76,0xA6,0x38,0x40,0x4B,0x0E,0x03,0x00,0x00, "Sigma", "", "150-600mm F5-6.3 DG OS HSM | C"}
RawTherapee has its own table, located in rtexif/nikonattribs.cc. Here’s its entry:
{"82 34 76 A6 38 40 4B 0E", "Sigma 150-600mm f/5-6.3 DG OS HSM | C"},
Darktable uses exiv2, same with PhotoFlow, more on that in a moment…
The two metadata utilities, exiv2 and exiftool, both use nikonlensid.h as a source. exiv2 has the nikonlensid.h file pasted into src/nikonmn_int.cpp, version:
#define FMLVERSION "4.4.550.02"
#define FMLDATE "2018-03-03"
and the entry for the Sigma lens:
{0x82,0x34,0x76,0xA6,0x38,0x40,0x4B,0x0E,0x03,0x00,0x00, "Sigma", "", "150-600mm F5-6.3 DG OS HSM Contemporary"}
Note the difference between this entry and the one from the rawproc version above.
exiftool has it’s own Perl-encoded table in lib/Image/ExifTool/Nikon.pm. Phil Harvey has this note at the table’s head :
# Note: Sync this list with Robert's Perl version at
# http://www.rottmerhusen.com/objektives/lensid/files/exif/fmountlens.p.txt
and here’s the Sigma lens entry:
'82 34 76 A6 38 40 4B 0E' => 'Sigma 150-600mm F5-6.3 DG OS HSM | C'
Now, all this needs to square up with the lensfun database. For one thing, there’s no single file of lens nomenclatures, they’re all scattered in the correction data. I’d surmise that, based on how calibrations are collected, each submitter is supplying the lens nomenclature they read from their favorite raw converter, or Hugin, as that’s a key tool in analyzing the distortion component. For the Sigma lens, @Carmelo_DrRaw showed us that nomenclature in his first post of the thread.
Soooooo… bear-of-little-brain here makes the following observations:
Again, this missive is focused at the Nikon situation, but probably has applicability to other vendors. I’m particularly interested now because I have a camera/lens from a manufacturer (Nikon Z) that has explicitly stated that software corrections are now in the trade space for lens design. Oh, for that tool, Nikon encodes correction data in their proprietary metadata; ideally, one would just be reading that and correcting the image, no hairy lookups of externally-produced correction data, but that’s another (wide) ditch to jump…
Musings on a snowy day in Colorado…
Hey
Some time ago (a long time ago? Damn, time flies) I wanted to update RawTherapee’s metadata ‘database’ with the latest camera and lens names. However, I don’t know how it was created in the first place, it’s cryptic C++ stuff, and I don’t speak C++ anyway. To a hammer every problem is a nail, so I whipped out my hammer and Bashed ExifTool’s XML output. The inconsistency in things like aperture notation bugged me, and consistency is a nice thing to have. Filing bug reports about that didn’t help, so I ended up fixing it myself on the fly using sed. Seemed like the right thing to do. However, back then RawTherapee didn’t support Lensfun, and now it does, so I slightly regret that choice, as there’s a good chance that there’s already an indecent amount of string fudging and bamboozling going on, on several levels - when people submit data to Lensfun, when people submit data to ExifTool/Exiv2, when any of these try to talk with the other, and who knows where else.
Imagine all these programs were open-source, then we could actually share data, combine efforts, and synergize, and stuff. Ah, a man can dream.
And share workers.
There are plans to build one database to rule them all. Its on our gitlab page. Might be a nice topic for LGM.
Got link?
Let’s assume LGM/Rennes 2021 happens. A BOF discussion about how to cooperate to improve the situation. It would be great to get Phil Harvey to come, or at least join the discussion on video.
I’m not offering to organise anything. I will participate in the discussion. We need a good chairman and one or more proposals for discussion. Any volunteers?