Yeah I was hoping it was already reversed engineered.
Exiftool just sees this:
80D:
AF Micro Adj Mode : Disable
AF Micro Adj Value : 0
Vignetting Corr Version : 0
Peripheral Lighting : On
Distortion Correction : On
Chromatic Aberration Corr : On
Peripheral Lighting Value : 0
Distortion Correction Value : 100
Original Image Width : 100
Original Image Height : 0
Peripheral Lighting Setting : On
Chromatic Aberration Setting : On
Distortion Correction Setting : On
Peripheral Illumination Corr : Off
Auto Lighting Optimizer : Off
Highlight Tone Priority : Off
Long Exposure Noise Reduction : Off
High ISO Noise Reduction : Off
77D:
AF Micro Adj Mode : Unknown (3)
AF Micro Adj Value : undef
Vignetting Corr Version : 16
Peripheral Lighting Setting : On
Chromatic Aberration Setting : On
Distortion Correction Setting : On
Peripheral Illumination Corr : Off
Auto Lighting Optimizer : Off
Highlight Tone Priority : Off
Long Exposure Noise Reduction : Off
High ISO Noise Reduction : Strong
Hopefully they are also embedding the actual profiles.
I would be surprised if RAW files from Canon EF cameras have correction data embedded. Nevertheless, there are some tricks to figure out what the state of play is.
First, try developing your RAWās in a commercial application which is capable of correcting distortion. Then, run this under either a debugger or something like strace to see what files it opens up. Look for anything that seems like a correction profile. If so, try deleting this profile and seeing what happens. If corrections still get applied then youāre in business.
Another approach Iāve found helpful when sniffing for vignetting data is to take a sequence of images in manual mode at a range of apertures. The idea here is to exploit the fact that while there is likely vignetting wide open, it will decrease as you stop down. Then open up the raw files and begin by marking out any regions (byte ranges) which Exiftool knows about. From here diff whatever remains. Look for any patterns. If a single region keeps changing then it is a possible candidate for correction data.
What you observe through the OVF will always be the distorted image. This is one of the reasons why older lenses designed often suffer from less distortion than newer designs.
Live view depend on the camera. On mirrorless cameras, in my experience, both the EVF and live view are both corrected (hence why it is essential for correction data to be stored in the lens firmware). I am unsure about live view for DSLRās.
Freddie (@fdw). Please explain to me (and the community) why you didnāt say Thank You for my 40 hours of work on your behalf to make this possible. Please also explain why my contribution to your success has not been acknowledged.
Apologies, I missed your first message. I am of course thankful for your help in this matter (and exiv2 in general), and me not acknowledging it was purely an oversight on my part ā and one which I am happy to correct.
Moreover given that this entire thing represents no more than a few hours work on my part, you are more than welcome to the credit, especially given the time youāve invested.
Thank You, @fdw. The credit for applying the metadata to the photo is entirely yours. Iām not a photographer. I work on the metadata engine without knowing what the metadata means!
@KristijanZic asked about Nikon and Canon, and if similar metadata is stored, presumably it can be applied. Reading Canon CR3 files is implemented in the tvisitor.cpp code in my book and I hope one day it will be ported to Exiv2. @1div0 is working with me on tvisitor.cpp and I hope support for CR3 will arrive in Exiv2 in 2021. I intend to finish the book and retire on 2021-01-18. IMaEA
Itās amazing what can be achieved with team-work. 1+1=3. And itās amazing how progress can be obstructed by unfounded legal challenges. 1-1=-1.
@KristijanZic I donāt know. I know very little about photography or the meaning of metadata. However I know everything about how metadata is stored and safely manipulated in C++. Iām like a vehicle mechanic who cannot drive.
Preferably for both CR2 and CR3 but in this instance Iām asking about CR2 as there a high likelihood that CR3 has correction data embedded if other mirrorless cameras do that also.
Iām curious about if thatās also the case for canon DSLRs that produce CR2.
So Iāve done some Googling to see what the commercial raw developer applications are offering. Here Iridient Digital - Iridient C-Transformer claims āAutomatic lens corrections for distortion, chromatic aberration and vignetting. Corrections are based on native Canon lens information specified in their CR2 metadata.ā which indicates that there may well be correction information baked into the CR2 files. Of course, it is also possible that it is simply looking for the lens ID and then looking up the correction information in a database.
The way to unpick this would be for someone who owns a copy to run the program under a debugger and see if the program opens up anything which looks like a correction database whilst it is running. If not, then the data may well be coming from the CR2 file.
I donāt think EF lenses include correction parameters in the lens firmware. Usually to have correction in camera you need to download the lens parameters to the camera using Canonās software, for example.
Many third-party lenses also do. I know that Sigmaās MC-11 explicitly advertised vig/dis correction (I do need to double check in the data), and Iām 90% positive all of the native-E Sigmas also have vig/dis data.
Newer Canon lenses/bodies appear to support this, along with third party lenses - there was news a couple of years ago where apparently, Sigma lenses had an error in the correction data that they reported which would make one of Canonās latest bodies crash. (I canāt find the link at the momentā¦)
Wouldnāt a body/lens combination that can use lens correction data show that in the ooc jpeg (and by extension in the jpg embedded in the raw file)? In that case, Iād expect to see a difference between the uncorrected raw and the corrected ooc jpgā¦
Not a guarantee that those correction parameters are stored in the raw file, but a strong hint they might be (or the manufacturerās raw converter could be left out in the cold)
This is absolutely correct. Indeed, RawTherapee can exploit this fact to try and derive a correction model on the fly. It does this by considering what distortion is required to make the RAW file currently being processed look like the embedded JPEG preview. Results in my experience are hit or miss, however.
It would certainly be nice if the correction data was also stored in the RAW file, and based on what I am reading, may also be quite likely. What is needed now is someone with a Windows system, a developer application which fully understands CR2 files, and a sample RAW.
Iridient C Transformer seems to run perfectly in Wine (Ubuntu 20.04 + Wine dev branch). Nothing special to do, just install wine devel from the winehq repo and run wine [installer.exe]. I donāt know how to debug it tho. There is winedbg but I donāt know how to use it yet.
I can provide the samples, just tell me what you need and what camera lens combos to provide.
I have access to Canon EOS 77D, Canon EOS 80D and Canon 5D mark IV. And I have access to the following lenses:
FF:
Canon EF 50mm f1,8
Canon EF 80mm f1,8
Canon EF 16-35mm f4 IS
Canon EF 70-200mm f2.8 III
All of these produce CR2 but 5Dmk4 also produces those Dual Pixel Raw (cr2) images.
I think Iāve found something, check this out:
Original CR2 (5Dmk4 + Canon 70-200mm III): IMG_6440.CR2 (33.8 MB)
Converted to DNG with Iridient C Transformer with all the corrections applied: IMG_6440.dng (66.4 MB)
I think the distortion is corrected in the dng, compare the clouds near the top edge.
Okay, lets get this party started. The first thing I like to hunt down is vignetting. Take a picture of a while wall at a fixed focal length at a range of apertures with manual focus. Then, look to see if any corrections are being applied (it is interesting that the tool can apparently apply corrections to RAW data, something I have not seen before). What you want to confirm is that vignetting is being corrected by the converter.
Beyond this, I have a question for @clanmills: Have you ever known Canon to employ encryption and/or checksums in their Markernotes for CR2 files? I know Nikon and Sony encrypt certain parts (albeit poorly).
Alright, here are my findings so far, Iāve tested 18-135mm, 10-18mm and 50mm on EOS 77D.
Iridient doesnāt correct anything.
Iāve tested one image that I took with 70-200 on a 5Dmk4 and Iridient definitely corrects distortion. It also seems to grey out some green CA and Iām not yet sure about vignetting. I can test tomorrow, I donāt have FF setup with me now.
But here is the interesting part. Iāve taken some of the sample raw images from dpreview taken with the EOS R and Canon RF 15-35.
Iridient corrects perfectly for vignetting and distortion. Again Iām not sure about CA because the lens doesnāt appear to produce much CA and I canāt make a test image myself.
But it seems like both CR2 and CR3 might have at least some correction data embedded.
@fdw The metadata parsing engine in Exiv2 (TiffVisitor) can decrypt/encrypt tags. This feature is only used by Sony tag 0x9402 (FocusPosition) which is ciphered (lookup table byte replacement).
I read somewhere that Nikon used to cipher the tags in a NEF. I have a D5300 and the metadata in my NEF images is not ciphered. Exiv2 does not have such a file in the test suite. Perhaps somebody in the Community could provide one.
I was hoping it was already reversed engineered.