Reverse engineering Nikon Z-series lens correction

paperdigits · April 19, 2023, 4:45pm

The new-ish Nikon Z series of cameras and lenses contain distortion and vignette correction in each raw file, the same as other manufacturers such as Sony and Fuji are doing.

The upside is that you don’t need to profile every lens for inclusion in lensfun anymore, it the downside is that we will need to reverse engineer what that metadata means and the formulas used, as like other manufacturers, Nikon hides this information behind an NDA. And NDAs and FOSS code do not go together. Actually I don’t know that the information is in the Nikon SDK because you have to agree to the NDA before you are considered for having access to the SDK. But I assume the information is in there.

I’ve emailed both Nikon USA and the corporate feedback for the SDK to express my displeasure at the NDA and the lack of freely available information for a camera that I just spent thousands of dollars on. This of course did absolutely no good.

So here I will begin collecting information and try and figure out what the metadata means so that eventually we can have support for Nikon Z lenses and cameras.

Please, Nikon Z users, join me!

paperdigits · April 19, 2023, 4:47pm

As documented in the exiftool source code, the distortion and vignette correction is here:

# extra info found in IFD0 of NEF files (ref PH, Z6/Z7)
%Image::ExifTool::Nikon::NEFInfo = (
    GROUPS => { 0 => 'MakerNotes', 2 => 'Camera' },
    NOTES => q{
        As-yet unknown information found in SubIFD1 tag 0xc7d5 of NEF images from
        cameras such as the Z6 and Z7, and NRW images from some Coolpix cameras.
    },
    # 0x01 - undef[12]
    # 0x02 - undef[148]
    # 0x03 - undef[284]
    # 0x04 - undef[148,212]
    # 0x05 - undef[84] (barrel distortion params at offsets 0x14,0x1c,0x24, ref 28)
    # 0x06 - undef[116] (vignette correction params at offsets 0x24,0x34,0x44, ref 28)
    # 0x07 - undef[104]
    # 0x08 - undef[24]
    # 0x09 - undef[36]
);

Colin_Adams · April 19, 2023, 5:16pm

Can I do anything to help? I have a Nikkor 24-120 f/4 S Z lens.

Jade_NL · April 19, 2023, 5:18pm

I have the 35mm F/1.8 and 24mm F/1.8 S Z lenses.

As Colin, I’m not sure how I can help, though.

paperdigits · April 19, 2023, 6:26pm

I have the following Z lenses:

50 mm f1.8s (this has lensfun correction data available)
24 mm f1.8s
100-400 f4.5-5.6 VR S

paperdigits · April 19, 2023, 6:29pm

First thing to do is to dump the metadata that exiftool has listed.

I’ll put it into a spreadsheet as people start reporting numbers.

Second thing to do would be to try known formulas for correcting lens distortion and vignetting with the metadata dumped from the Nikon raw files.

ggbutcher · April 19, 2023, 7:30pm

I want to help, but I’ll not be with my primary computer for a few weeks.

It is likely the values line up with the parameters required for the Adobe DNG opcode3 distortion correction function. I took a Z 6 NEF and DNG converted it, that’s as far as I got…

Jade_NL · April 20, 2023, 6:05am

Here’s the 0xc7d5 specific datablock for the:

Nikkor Z 24mm f/1.8 (2.6 KB)
Nikkor Z 35mm f/1.8 (2.6 KB) This one is also in the Lensfun database.

Probably not relevant but both shots used are from a Z6ii

kmilos · April 20, 2023, 8:30am

Please note that, once decoded, someone will also need to enable accessing these blocks from exiv2 (for dt and soon RT to benefit).

Colin_Adams · April 20, 2023, 1:00pm

What command do you use to produce this dump?

Jade_NL · April 20, 2023, 1:40pm

I used this:

exiftool -v4 image.nef

You do need v4, v3 will truncate some stuff. It will spit out about 1600 lines in a semi ordered fashion. For both my lenses and using my Z6ii the block of interest sits between lines 158 and 187.

kmilos · April 20, 2023, 1:45pm

Or even -v5, depending on the size of the chunk (so this just keeps pushing the truncation threshold)…

Colin_Adams · April 20, 2023, 1:47pm

Here is the 0xc7d5 block for the Nikkor Z 24-120 f’4 S:

  | | 17) NikonNEFInfo (SubDirectory) -->
  | |     - Tag 0xc7d5 (976 bytes, undef[976]):
  | |        3fa28: 4e 69 6b 6f 6e 00 01 03 00 00 49 49 2a 00 08 00 [Nikon.....II*...]
  | |        3fa38: 00 00 0f 00 05 00 07 00 54 00 00 00 c2 00 00 00 [........T.......]
  | |        3fa48: 06 00 07 00 74 00 00 00 16 01 00 00 07 00 07 00 [....t...........]
  | |        3fa58: 68 00 00 00 8a 01 00 00 08 00 07 00 18 00 00 00 [h...............]
  | |        3fa68: f2 01 00 00 09 00 07 00 24 00 00 00 0a 02 00 00 [........$.......]
  | |        3fa78: 0a 00 07 00 0e 00 00 00 2e 02 00 00 0b 00 07 00 [................]
  | |        3fa88: 0e 00 00 00 3e 02 00 00 0c 00 07 00 0a 00 00 00 [....>...........]
  | |        3fa98: 4e 02 00 00 0d 00 07 00 08 00 00 00 5a 02 00 00 [N...........Z...]
  | |        3faa8: 0e 00 07 00 34 00 00 00 62 02 00 00 0f 00 07 00 [....4...b.......]
  | |        3fab8: 0e 00 00 00 96 02 00 00 10 00 07 00 08 00 00 00 [................]
  | |        3fac8: a6 02 00 00 11 00 07 00 0c 00 00 00 ae 02 00 00 [................]
  | |        3fad8: 12 00 07 00 4e 00 00 00 ba 02 00 00 13 00 07 00 [....N...........]
  | |        3fae8: ba 00 00 00 0a 03 00 00 00 00 00 00 30 31 30 30 [............0100]
  | |        3faf8: 03 01 00 00 9a 59 01 00 00 10 00 00 04 00 00 00 [.....Y..........]
  | |        3fb08: 56 99 01 00 00 00 10 00 74 fc fe ff 00 00 10 00 [V.......t.......]
  | |        3fb18: 47 40 fe ff 00 00 10 00 00 00 00 00 01 00 00 00 [G@..............]
  | |        3fb28: 00 00 00 00 01 00 00 00 00 00 00 00 cf 76 ff ff [.............v..]
  | |        3fb38: 00 00 10 00 00 00 00 00 00 00 00 00 00 00 9b 25 [...............%]
  | |        3fb48: 30 31 30 30 02 01 00 00 9a 59 01 00 00 10 00 00 [0100.....Y......]
  | |        3fb58: 08 00 00 00 94 82 88 00 00 00 10 00 00 00 00 00 [................]
  | |        3fb68: 01 00 00 00 4b 9a 8b ff 00 00 10 00 00 00 00 00 [....K...........]
  | |        3fb78: 01 00 00 00 8f 21 20 00 00 00 10 00 00 00 00 00 [.....! .........]
  | |        3fb88: 01 00 00 00 18 7e 08 00 00 00 10 00 00 00 00 00 [.....~..........]
  | |        3fb98: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 00 [................]
  | |        3fba8: 00 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 [................]
  | |        3fbb8: 00 00 e1 49 30 31 30 30 01 02 00 00 9a 59 01 00 [...I0100.....Y..]
  | |        3fbc8: 00 10 00 00 03 00 00 00 32 02 00 00 00 00 10 00 [........2.......]
  | |        3fbd8: 1f fc ff ff 00 00 10 00 00 00 00 00 01 00 00 00 [................]
  | |        3fbe8: 50 02 00 00 00 00 10 00 03 00 00 00 3a fd ff ff [P...........:...]
  | |        3fbf8: 00 00 10 00 c9 04 00 00 00 00 10 00 00 00 00 00 [................]
  | |        3fc08: 01 00 00 00 3a ff ff ff 00 00 10 00 00 00 01 00 [....:...........]
  | |        3fc18: 00 00 00 00 00 00 00 00 00 00 57 ae 30 31 30 30 [..........W.0100]
  | |        3fc28: dd dd ff ff 00 01 00 00 c7 23 00 00 00 01 00 00 [.........#......]
  | |        3fc38: 00 00 68 2b 81 03 00 00 00 02 00 00 fa 02 00 00 [..h+............]
  | |        3fc48: 00 02 00 00 00 02 00 00 00 02 00 00 00 02 00 00 [................]
  | |        3fc58: 00 02 00 00 00 00 a6 b5 30 31 30 30 f0 03 f0 03 [........0100....]
  | |        3fc68: f0 03 f0 03 40 ac 00 00 30 31 30 30 0c 00 08 00 [....@...0100....]
  | |        3fc78: 40 20 80 15 3d 0e 00 00 30 31 30 30 00 00 00 00 [@ ..=...0100....]
  | |        3fc88: 6c b5 00 00 30 31 30 30 03 00 3c 3b 30 31 30 30 [l...0100..<;0100]
  | |        3fc98: 46 30 00 08 00 08 00 08 00 08 22 00 b7 2a aa 9d [F0........"..*..]
  | |        3fca8: e0 72 5e 91 18 f5 1c 99 65 82 f0 af bf 20 d2 d2 [.r^.....e.... ..]
  | |        3fcb8: 2f c6 c1 02 a7 86 c5 5a 21 58 d8 a6 ca 36 ec 4c [/......Z!X...6.L]
  | |        3fcc8: 30 31 30 30 00 00 00 00 00 00 00 00 9e a3 00 00 [0100............]
  | |        3fcd8: 30 31 30 30 f2 12 bf 29 30 31 30 30 c1 0f c2 28 [0100...)0100...(]
  | |        3fce8: ce f0 54 79 30 31 30 30 9f d5 1a 23 00 ca 9a 3b [..Ty0100...#...;]
  | |        3fcf8: 7b 84 25 17 00 ca 9a 3b 5d 74 66 fe 00 ca 9a 3b [{.%....;]tf....;]
  | |        3fd08: ed 8f d8 0c 00 ca 9a 3b 3f 78 b2 3e 00 ca 9a 3b [.......;?x.>...;]
  | |        3fd18: d4 c1 0f f0 00 ca 9a 3b cf f3 98 00 00 ca 9a 3b [.......;.......;]
  | |        3fd28: d1 28 15 f5 00 ca 9a 3b 37 97 3b 4b 00 ca 9a 3b [.(.....;7.;K...;]
  | |        3fd38: 7f 80 00 00 30 31 30 30 c4 09 51 92 01 00 a0 86 [....0100..Q.....]
  | |        3fd48: 01 00 17 64 04 00 a0 86 01 00 b8 0b ed d0 01 00 [...d............]
  | |        3fd58: a0 86 01 00 de 7f 03 00 a0 86 01 00 a0 0f 03 4c [...............L]
  | |        3fd68: 02 00 a0 86 01 00 f0 92 02 00 a0 86 01 00 88 13 [................]
  | |        3fd78: d5 c3 02 00 a0 86 01 00 71 40 02 00 a0 86 01 00 [........q@......]
  | |        3fd88: 70 17 c4 08 03 00 a0 86 01 00 91 f8 01 00 a0 86 [p...............]
  | |        3fd98: 01 00 40 1f 49 7d 03 00 a0 86 01 00 c7 c6 01 00 [..@.I}..........]
  | |        3fda8: a0 86 01 00 10 27 26 e1 03 00 a0 86 01 00 7d a3 [.....'&.......}.]
  | |        3fdb8: 01 00 a0 86 01 00 00 00 00 00 00 00 00 00 00 00 [................]
  | |        3fdc8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
  | |        3fdd8: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [................]
  | |        3fde8: 00 00 00 00 00 00 00 00 00 00 00 00 0a c0 00 00 [................]
  | | + [MakerNotes directory with 15 entries]

Colin_Adams · April 20, 2023, 1:51pm

Hm. It seems to differ according to focal length. That one was for 24mm. Will I need to take photos at a range of focal lengths?

Jade_NL · April 20, 2023, 1:51pm

Good point!

I jumped the gun with -v4 based on my 2 lenses, with both have a block that is the same size.

The amount of data seems to be mentioned (undef[448] for both my lenses and as I see now, undef[976] for Colin’s lens.

EDIT

@Colin_Adams:

Zoom vs prime lens, I guess. Not sure what to advise you about this.

kmilos · April 20, 2023, 1:59pm

As this is just another TIFF IFD structure, exiftool smartly parses it further if you give it the -u option:

Nikon_NEFInfo_0x0005            : 0100..šY...y†.Wãþÿ.Ir..[...]
Nikon_NEFInfo_0x0006            : 0100..šY......v...)Ó.[...]
Nikon_NEFInfo_0x0007            : 0100..šY...œúÿÿ.r...Sÿÿÿ..[...]
Nikon_NEFInfo_0x0008            : 0100h...þÿÿ.±œ
Nikon_NEFInfo_0x0009            : l.........Õ6
Nikon_NEFInfo_0x000a            : 0100ð.ð.ð.ð.@¬
Nikon_NEFInfo_0x000b            : 0100..@ €.=.
Nikon_NEFInfo_0x000c            : 0100lµ
Nikon_NEFInfo_0x000d            : 0100.<;
Nikon_NEFInfo_0x000e            : 0100F0...."·*ªàr^‘.õ.™e‚ð¯¿ ÒÒ/ÆÁ.§†ÅZ!XØ¦Ê6ìL
Nikon_NEFInfo_0x000f            : 0100ž£

albeit w/ unknown tags (also seen in -v4/-v5 output).

Going by existing Nikon MakerNote fields, the 4 first bytes usually denote a version of some block/struct, so “0100” probably means “1.00”.

Colin_Adams · April 20, 2023, 2:05pm

Hm. That is more readable?
Anyway, it’s not the lens firmware version, as that is 1.10.

I see:

Nikon NEF Info 0x0005 : 0100…<9A>Y…V<99>…t.G@…[…]
Nikon NEF Info 0x0006 : 0100…<9A>Y…<94><82><88>…K<9A><8B>…<8F>! […]
Nikon NEF Info 0x0007 : 0100…<9A>Y…2……P…[…]
Nikon NEF Info 0x0008 : 0100

.#.h+
Nikon NEF Info 0x0009 : <81>……
Nikon NEF Info 0x000a : 0100....@
Nikon NEF Info 0x000b : 0100…@ <80>.=.
Nikon NEF Info 0x000c : 0100l
Nikon NEF Info 0x000d : 0100.<;
Nikon NEF Info 0x000e : 0100F0…"*<9D>r^<91>..<99>e<82> /.<86>Z!Xئ6L
Nikon NEF Info 0x000f : 0100<9E>
Nikon NEF Info 0x0010 : 0100.)
Nikon NEF Info 0x0011 : 0100.(Ty
Nikon NEF Info 0x0012 : 0100<9F>.#ʚ;{<84>%.ʚ;]tfʚ;<8F>.ʚ;?x>ʚ;.ʚ;<98>[…]
Nikon NEF Info 0x0013 : 0100.Q<92>.<86>…d.<86>...<86>.…<86>.…L.<86>.<92>.[…]

Colin_Adams · April 20, 2023, 2:09pm

BTW, lensfun has data for this lens.

Jade_NL · April 20, 2023, 2:10pm

Those are not the correct tags that are shown by -u. Or more specific the tags we are looking for.

The first one (0x0005), for example, is this one in the -v4/5 output:

| | | 2) WhiteBalance = AUTO1
| | | - Tag 0x0005 (13 bytes, string[13]):
| | | 890c: 41 55 54 4f 31 20 20 20 20 20 20 20 00 [AUTO1 .]

kmilos · April 20, 2023, 2:13pm

The two bytes just after the “Nikon\0” header seem to also indicate some version number of this directory:

          - Tag 0xc7d5 (976 bytes, undef[976]):
  | |        3fa28: 4e 69 6b 6f 6e 00 01 03 00 00 49 49 2a 00 08 00 [Nikon.....II*...

So it would be “1.03” here. I’m e.g. seeing “1.02” on the Z 9 sample on RPU…

(For the traditional MakerNote way below it is 02 11 for “2.11”.)