What's the problem with CR3 format files?

Software RawTherapee
1

I’m pretty sure Canon has been the leading maker of cameras for many years (more likely decades) now. As such it does seem that the number of CR3 format files being produced may easily exceed any other format. One might conclude that should make support for them a top priority for any product intending to support the significant variety of raw files being produced by all/many makes of digital cameras.

Also, even though CR3 format files are new to me they’ve been around for quite some time now.

Furthermore, Rawtherapee (RT) is NOT the only public license (i.e., freeware) product that attempts to support a wide variety of different camera formats for raw file post processing. None of these other products, that I either use regularly or have experimented with, are lacking support for CR3 metadata. These products include raw file post processing programs like Darktable, Art (which is a fork of RT BTW), viewers of both images and metadata like XnViewMP, or simply metadata processing software such as ExifTool and Exiv2.

This begs the questions, “what’s the problem with RT?”, “Is there an expectation that this problem will get fixed?”, if so “when could we expect that to happen?”.

Note: I have been downloading development versions to include as recently as yesterday, which looks like it was created on 3/29/2022, with the hope that the fix is included.

2

To my knowledge, Canon’s CR3 files are supported in the latest development version. Or at least most of the different CR3 types.

There are many topics about this here on Pixls. Here’s the last known issue, which was fixed within days and is now part of the master branch.

You do need to use the latest development version (Direct download links), the “latest” stable version is over 2 years old and should not be used any longer.

1 Like
3

One should call canon and suggest that they release the spec for their format publicly, and not require an NDA to get such details. Or just not buy canon at all and be very vocal about the reason why.

5 Likes
4

I can develop my CR3 files using RT. That’s NOT the problem. The problem is that, apparently, it can NOT read the metadata. As a result, features like dynamic profiles do NOT work. However, what’s troubling me the most is that metadata gets lost. When I output .tif files (assume same is true for other formats) the Makernotes in the metadata are completely gone.

I have tried the latest development version (posted 4/7/2022 I think) and it still does NOT work.

5

Haven’t had time to do that yet. But why hasn’t that been a problem for all of the other software I’m using? Including ART which, I think, is a fork of RT.

6

Art includes exiftool to read cr3 metadata and darktable requires a new version of exiv2. RT hasn’t released a build with either of those yet.

7

Sounds like the answer. Why is it taking so long? There are lots of these (CR3) files being created. I’d guess more than any other kind of raw files.

8

Short answers:
Q: What’s the problem with RT?
A: Some developers have been occupied with other things.
Q: Is there an expectation that this problem will get fixed?
A: Yes.
Q: When could we expect that to happen?
A: After the 5.9 release.

Long answer:
RawTherapee’s metadata handling code is adapted from ExifTool. In order to support new formats, someone needs to find the relevant parts of ExifTool and insert them into RawTherapee. The process is “tedious and error prone” and the task has mostly been done by one person, Maciej Dworak @Morgan_Hardwood (He is also the one who has been responsible for releasing new “stable” versions of RawTherapee). Rather than try to include the CR3 updates from ExifTool, the plan is to offload the metadata handling to exiv2. There has been progress on this mostly thanks to Alberto @agriggio (the creator of ART) and it is almost ready. However, it is a big change with many possible side effects. It’s best to leave it out of the next release (5.9). The release needs to happen first, then the exiv2 update can be incorporated into dev to receive feedback from more users.

I can’t give you a timeline of when it will happen. We are stuck on the release because Maciej is unavailable and none of the other developers have intimate knowledge of the process.

4 Likes
9

Because apparently none of the developers have the need for it, and other people are just waiting or something.

It comes back to the expectation of opensource development.

A question ‘why is it taking so long’ seems to imply that

  • it’s easy
  • the developers that are active agree with your priority assessment
  • they the time having past is indeed 'a long time '.

The basic rule to remember - always - is that the programs are developed how the devs want them to be. Not how you think they should be. That doesn’t mean it needs to support the latest and greatest , it doesn’t mean it needs to compete with other software , and it doesn’t mean that an ‘obvious’ feature for you is put up high in the list.

Personally , the statement ‘canon has been the leading camera maker for the past decade’ made me giggle. They finally caught up is how I see it, but ok.
But the question of new file formats affect all manufacturers, so it doesn’t even matter.

Edit: I think I’m confusing Darktable and Rawtherapee things in the story down here, but the principle still stands. I believe it was Darktable using exiv2, why Cr3 support in Darktable took a while. Rawtherapee uses exiftool , and I don’t know the state there. But the principle is the same, all the dependencies need to aligned and need to be used in a build to get something working.

In the case of Cr3 - i believe, I didn’t follow it with much attention because I don’t care about Cr3 - is that the ‘base’ format which contains the data is based on more recent container formats. So, it isn’t “like a TIFF file but with raw data” but it is more “like a MP4/mov file but with raw data”. That means metadata is written in another way.

Now, the thing is, that there were some questions and observations about that base format having certain patents or copyrights around it , that made the authors of metadata libraries be careful about wanting to implement it. Which is of course fully in their control.

And the exiv2 library is used by Rawtherapee (or was it Darktable?) to read and parse information about the files it opens.

Now, there has been support here and there. I believe other programs like art and/or Darktable made an exception for Cr3 files to not use exiv2 for just Cr3 files… or they changed to another metadata system all together. I really don’t know .

I believe exiv2 has even the correct support added ? But that would mean you would need a recent build of Rawtherapee, that was build using a recent version of exiv2, and probably a build of exiv2 that had the special support enabled (which might not be the case by default ).
So that are a lot or maybes that affect if Rawtherapee is now able to read Cr3 files or not .
Or they just left it at reading the raw data and not bothering with metadata, who knows .

The common answer to 'why is this obvious feature taking so long ’ is that the feature isn’t as obvious as you might think , or the devs have very little time, and relative simple stuff takes a lot of time.

Very often a combination.

If i was developing software to use myself, feature requests for a file format which is of no relevance to me won’t be high on the todo list.

10

To be honest I do know that Canon does NOT itself put much effort into customer support. They operate forums such as this that are community based (i.e., depend on their users to support other users). This works good for many things but NOT this kind of issue.

However, my own hunting around on this issue did discover something that looks like it might offer additional detail. The ExifTool website references this link which deals with specifics of the CR3 format.

11

Thank you Lawrence!

Your long answer caused me to revisit my own findings which in the case of ART and Darktable are based on some trials that I performed. Not having prior knowledge of either they pretty much involve using only default settings.

In the case of ART the metadata that ends up being added to the .tif file produced from raw development contains pretty much the same metadata as Rawtherapee (RT). The main difference is that, unlike RT, ART does NOT have any problem figuring out what camera produced the raw file and therefore the dynamic profiles as well as camera and lens identification for automatically selecting input profiles works as expected. However, my complaint about losing Makernotes still happens.

Note: correction from original post is that I’ve now learned that Art does offer some options for what metadata gets preserved in the files it produces. By using an option called “Unchanged” the original metadata can be preserved and seems to have no problem including Makernotes.

Likewise Darktable has no problem identifying my camera even though it uses a method of acquiring camera specific input profile information that I don’t understand. As best I can tell it does NOT use ICC profiles (nor DCP profiles for that matter). In my sample output .tif file the Makernotes are still missing. Furthermore, it added in a whole lot of what look to be its’ own XMP metadata that I see no value in having. I suspect this could turned off via some kind of options/preferences/settings.

In summary, I implied that that these products provided the normal kind of support I was looking for when it fact that is NOT the case. They do seem to be suffering from similar problems to RT.

12

As I’ve demonstrated in my YouTube videos, Canon 1DXMk3 and R5 files (those are the only cr3s I have access to) open fine in RT, it just can’t read the exif.

And, if you use the free AdobeDNGConverter and make a dng file and open that in RT then it CAN read the exif - all of it!

1 Like
13

Andy,

Yes, I agree, I can develop CR3 files with RT V5.8 (latest stable release). Mine are from an EOS R camera that is new to me but I think it’s been available for at least a couple of years. I wouldn’t say that my CR3 files “open fine”. I think because of the problem reading metadata RT can NOT even figure out what kind of camera produced the file. That makes several features unusable (e.g., dynamic profiles, auto matched camera profiles). These are kind of convenience features that I would say are NOT terribly important.

I have tried Adobe DNG Converter (ADC) with the hope that it would solve the problem. However, unlike what it is generally promoted to be there are significant revisions in a .dng format file. In that, the revisions are NOT confined to a change in format. Unfortunately, I know of NO way to tell how accurately the raw data is transformed but it fails to preserve both some of the metadata and the preview image from the camera. More discussion can be found here.

Something I have become curious about as a result of my experience trying to develop CR3 raw files with RT is "where does the metadata that is included in developed image files (.tif in my case) come from? It seems there is some metadata labeled EXIF included in developed .tif files but supposedly RT cannot read metadata from CR3 format file. In my case a big thing that is missing is something called Makernotes which constitutes a much larger portion of the camera supplied metadata than what is included.

14

Makernotes is often a bunch of non-descriptive binary data, embedded in the file.

You need to know how to parse that information to get anything out of it, most of the time.

In other words , makernotes are not standardized. It’s a field of metadata that says “het you go, here is 10kb of makernotes, good luck with it”.

So it requires knowledge and support to harvest anything useful out of it . If the file format is not fully supported by a program, it doesn’t know how to interpret that data.

I imagine the makernotes from more recent Canon camera’s are not that different compared to older models. It’s more that programs ‘need to be explained where to find the blob on the raw file, and explain that they probably support it just fine’. That’s why running the files through Adobe DNG converter can often help. The same makernotes are put into the DNG container, And suddenly programs think ‘DNG, exif, canon , makernotes… I know this !’. While if that starts with ‘ISOBMFF, canon , makernotes’ the program thinks “I am confused”.

There is an opensource program to do the Adobe DNG converter thing , just for Cr3 files. So it converts Cr3 files into DNG files that Rawtherapee might open or read better. I don’t know the state of it, and I don’t know if it helps with your missing metadata.

Ps , if a tool like exiftool is able to read the makernotes , you can always learn to copy metadata from the original Cr3 file to your processed tif if you want. If you get handy with it, copying over the stuff you miss and removing unwanted stuff that got added should be possible.

And a recent version of exiftool is easier to get and play with than to get a recent Rawtherapee build with all the correct options ticked in the dependencies to get it working for you , it seems.

15

Yes! Phil Harvey offered some help on how to transfer Makernotes from .CR3 file to .tif file. I do know how to write scripts for MultiCommander that allow such command line tools to be executed via GUI. Do not yet have it working but expect that will provide an effective workaround.

I doubt I’m the only one affected by this problem. Still think getting RT to properly support CR3 would benefit a lot of users both present and future. As the correction to my prior post indicates ART seems to having it working nicely NOW. Maybe RT developers should consider a little reverse forking.

My initial impression is that ART works pretty nicely. My reason for hesitating on complete dependence is that there basically is NO documentation. Rawpedia is fine for the things that are the same but they need to do something about their new refinements if they’d like it to become popular (which may NOT be the case).

16

If you care to share a raw file then I can at least compare a linear dng from it against a linear profile of the raw in Lightroom/ACR

17

Undoubtedly yes, but “I’ve concluded you all should do more work to support me” isn’t super motivating.

18

Perhaps obvious, or a particular cr3 raw file is needed here, but I have used these sets of CR3 from Canon EOS R and Canon EOS R5 cameras, for RT experiments. Some frames are jpg only, but some are both RAW and JPG.

Canon EOS R files can be found here.
View Canon EOS R sample gallery from DPReview.

Canon EOS R5 files can be found here.

Some notes:

  • I go to Color Tab | Color Management | Custom: ‘Canon EOS R5.dcp’ to set up input profile manually.

  • There is a Firefox bug which labels raw files ‘.MP4’ instead of '.CR3’ when downloading them, at least from these sites. I change file extension manually to ‘CR3’, if needed. Other browsers are not affected.

  • I use development RT, compiled from git source: 5.8-3085-g59a36d8f8