Call for testing: RawTherapee metadata handling with Exiv2 (includes CR3 support)

Software RawTherapee
,
1

Background

The RawTherapee team is planning to integrate the Exiv2 library to handle all metadata. This is a big internal change which will facilitate quicker support for new cameras and file formats while reducing the development time required from contributors.

Metadata handling is important throughout RawTherapee, including the display of EXIF and IPTC data, file browser filtering, dynamic profile matching, automatic lens correction profile selection, and writing of metadata to saved images. Adoption of Exiv2 can improve or break any of these functions. It is critical that the Exiv2 integration does not introduce major bugs into the stable releases.

How you can help

We need people to test the Exiv2 RawTherapee and report any regressions (things that are working properly in the latest stable release (5.9) or development version but not in the Exiv2 RawTherapee). Download one of the available builds and check that it is correctly handling metadata for your images. Some of the things to look for include:

  • Files of every type (ARW, CR2, CR3, DNG, JPEG, NEF, RAF, ORF, PEF, etc.) for every camera/imaging device you use
  • Information displayed beneath thumbnails in the file browser
  • The quick info overlay in the editor preview
  • Exif and IPTC data in the editor’s metadata tab shows all important information, and editable fields such as the EXIF copyright can be edited without issues
  • Metadata-dependent features, such as dynamic profiles and Profiled Lens Correction automatic selection, work as before
  • Saved images have all the required metadata regardless of file type (JPEG, PNG, TIFF), including metadata you’ve added/edited in the Exif and IPTC tabs

There may be others. If something not listed here is important to you, don’t forget to verify that it works so you won’t get a nasty surprise when the next stable release comes!

Let us know what file formats, cameras, functionality, etc. are working properly. If you come across any issues, be sure to provide a detailed report, preferably by commenting on the pull request. Attach sample files if applicable.

A big thanks to all who participate!

Notes on using pre-dev RawTherapee

Pre-dev builds such as the ones for Exiv2 RawTherapee are designed to have settings and caches separate from both the stable releases and development versions. However, RawTherapee by default will write PP3 sidecars next to the source image. That may cause your existing PP3s to be overwritten. It is recommended to tell pre-dev RawTherapee to write PP3s to the cache by going into Preferences > Image Processing > Processing Profile Handling > Processing profile saving location, and selecting “Save processing profile to the cache”.

More information about pre-dev builds is located here.

Downloads

The direct links are sufficient for most people.

Direct links

Here are direct links to the latest Exiv2 pre-dev RawTherapee builds.

Pre-dev releases

Find the list of all pre-dev releases here. The Exiv2 builds the ones begining with “RawTherapee_Beep6581_metadata-exiv2”. At the moment, only Linux (AppImage) and Windows (portable .zip) are available. See the direct links for the macOS download.

GitHub Actions

These are the current and past automated builds available to those with a GitHub account. Refer to this topic for instructions on how to obtain them. The pull request is here.

Build it yourself

For those who know how and want to build it from source, the branch name is metadata-exiv2. You will need to install the Exiv2 library (and the corresponding development package for certain OSs such as Ubuntu).

6 Likes
2

hello @Lawrence37

Thank you for updating the metadata-exiv2 branch.

The metadata-exiv2 branch works for me. I tested a few Canon EOS R and R5 CR3 files. The editor window display exif data, the Color → Color Management → Auto-matched camera profile and transform → Lens/Geometry → Automatically Selected, picks up profiles and lenses.

I built exiv2 from source like this, on top of Debian stable, to get a local exiv2 lib for my build, with CR3 support:

git clone https://github.com/Exiv2/exiv2.git

cd exiv2

git checkout v0.27.5

mkdir build
cd build

cmake -DCMAKE_BUILD_TYPE=Release -DEXIV2_ENABLE_BMFF=On -DCMAKE_INSTALL_PREFIX="$HOME/camera2/test2/rt-exiv2-test" ..

make -j8
make install

Edit:
Further testing shows that also color → Output profile is respected in result from RT output queue. The output is fine, with correct output color profile. This was a issue I had with old metadata-exiv2 branch.

1 Like
3

Hello, I tried the appimage with nefs, jpgs and tiffs.

Okay.

Okay.

Okay. One thing to note: there’s a little font issue when I change the User Comment or the Image Descripion field. The Backspace and Del keys works, but the old string stays visible. Typing a new comment produces a somewhat messy result.

user.comment

Works.

Works as well.

4

Hello
I tried Windows, with various images…

I did some tests, and at first sight, it seems to work well.
To be confirmed of course.
Excellent work

Jacques

5

Did tests with many Nikon cameras, DSLR and mirror-less. See the PR for the results.

In general: All looks to be good except for the Nikon Z series mirror-less cameras, which have lens detection issues (which was also the case in the old situation and has an older open bug report).

6

It’s so so for me.

pre-dev with this PR:
grafik

dev:
grafik

So there is improvement.

While

pre-dev with this PR:
grafik

dev:
grafik

The 50-250 is a native Z-Mount lens. The 17-50 is an adapted Sigma.

With the Sigma:

Nothing there:

grafik

Partially there:

grafik

Down there it is:

grafik

7

@apostel338

Nikon changed the way metadata is stored when the Z series arrived compared to the DSLRs, which explains the issue. But even (some of) the Z mount lenses are not seen correctly with this PR in place.

Maybe add your experience to the PR? Might narrow down the way this can be fixed. I did test a bunch of Z series cameras with Z and F mount lenses, but definitely not all possible combos :grin:

8

@Lawrence37 Thanks a lot for your effort!
Did some checks with .CR3 from PlayRaw and Fuji with stone age S6500 and X-H2.
Everything looks good!
Metadata in the exported jpegs are reasonable

9

@Lawrence37 it looks fine here with windows, except manually added attributes seem not to be exported.
Thanks for your efforts!

btw: also having lens detection issues as @Jade_NL. Here on a Sony with Sigma lenses:

grafik
grafik1190Ă—670 140 KB

grafik

Attaching these lenses to my former A100, they are recognised well.

1 Like
10

I also tested it with CR3, 5 different lenses: all have been recognized, including one picture with flash light (flash has been recognized.
Exiv data for dng can be read, and exported exiv data to jpg are correct.
IPTC data can be edited.
Dynamic profile and lens correction works.

11

I downloaded the Exiv2 pre.dev RT win64-release build, but the executable doesn’t run with Windows 10 because it misses the library libexiv2.dll.
I had a similar problem with a missing DLL for the ladenoise pre.dev build, which Jaques Desmis fixed today.

I would really like to test the Exiv2 version for handling IPTC data, because I have encountered problems with RT5.9 not writing IPTC data and with both RT5.7 and RT5.9 not correctly reading UTF-8 encoded IPTC data with 2 byte encoded characters (German Umlaute ä,ö and ü).

12

Seeing something interesting here in the File Browser (Windows 11).

Runs fine and initially (first visit) all the CR3s in a folder show EXIF. When I open a file and check the Metadata tab, it’s all there. But when I go back to the File Browser, that file is the only one showing EXIF. If I open another file, its EXIF is in the Metadata tab, but not in the File Browser, until I close and reopen RT. Then that file additionally shows EXIF (and so on). Simply changing folder does not work.

For example:

image
image1601Ă—450 202 KB

(I’ve since fixed the “Y” error in the date variable)

When I first opened this scratch folder, all CR3’s showed metadata on the thumbnails. I opened IMG_5161 (second row, fifth image) and all was OK in the Editor. When I came back to the File Browser, it was the only one showing metadata. I opened IMG_5155 (second row, fourth image) and all was OK in the Editor but back in the File Browser still only IMG_5161 showed metadata. I restarted RT and then both of these two files showed metadata (but no others). The same behavior repeated when I opened IMG_1692 (first row, fourth image) - All OK in the Editor but it took a restart to show its metadata in the File Browser.

Toggling CR3 filtering off / on in the File Browser works properly.

Thanks.

13

Thank you all for the responses so far. I will try to address the found issues as soon as I can and give updates on their statuses.

@schorschbey, I added the missing DLLs. The same download link has the latest version, so try it again. Note that I have not tired to run it on a real Windows machine yet, so don’t hesitate to inform me of any additional issues. Thanks for bringing it to my attention. Turns out, Exiv2 only works when… Exiv2 is present :laughing:

1 Like
14

Windows 10, with the latest version and its missing DLLs.
It’s right.
Thanks

15

I tested handling of IPTC data on Windows 10 with RAW files from my Sony DSC-RX100.

I opened a RAW file, entered plain text for the IPTC tags Caption, City, ProvinceState, CountryName and ObjectName and saved as JPEG, TIFF 8 bit, TIFF 16bit, TIFF 16bit float, PNG 8 bit and PNG 16 bit. Opening the files in GIMP 2.10.6 and checking the metadata showed that all tags were correctly stored for all file types. I also checked saving as JPEG with a maximum length entry for ObjectName and foung it correctly stored. This shows that the problems I encountered with RT5.9 not correctly writing IPTC data have been solved.

I then entered text with German characters ä, ö, ü and ß for the same IPTC tags, saved as JPEG, TIFF 8 bit, TIFF 16bit, TIFF 16bit float and PNG 8 bit and opened these files with Exiv2 RT. All IPTC data was correctly displayed in RT which shows that RT corectly writes and reads IPTC data with characters that are 2 byte encoded in UTF-8. This shows that the problems I encountered with RT5.7 and RT5.9 not correctly reading UTF-8 encoded IPTC data with 2 byte encoded characters have been solved.

I also opened a TIFF file containing IPTC data, amended the data for one of the IPTC tags and stored as JPEG for all three metadata copy modes. Opening these JPEG files with Exiv2 RT showed that the IPTC data was stored as intended, i.e. as described in RawPedia for the metadata copy modes. This shows that the problems I encountered with RT5.7 and RT5.9 not writing IPTC data for the copy unchanged mode has been solved.

A good job done for proper handling of IPTC data. I am looking forward to RT5.10.

I just have one suggestion for changing the behavior of the metadata tab. Up to now, the default setting for the neutral processing profile has been the copy unchanged mode for metadata. Therefore, I may not be the only person who has created his own processing profiles from the neutral profile, not changed the metadata copy mode, entered IPTC data for a RAW file and found the stored processed file to not contain any of the entered IPTC data. This unexpected and to me embarassing behavior would also happen when using the old processing profile with Exiv2 RT. I think it is not a proper behaviour for a GUI to let a user enter data for a program mode where the entered data will not be stored. In my opinion, it would be more appropriate to make the IPTC data fields non editable for the Copy Unchanged Mode, with the data from the source file displayed grayed out. Editing IPTC data fields should be enabled only for the Apply Modifications Mode and changing between both these modes should toggle between displaying the source file data for Copy Unchanged and displaying the modified data stored in the pp3 sidecar file for Apply Modifications.

Finally, I noted an odd behavior I have observed for opening RAW files in this pre-dev build. Double clicking a RAW file in the RT file browser opens the file in the editor at the usual speed until I open the Metadata Tab for the first time. From then on, opening a RAW file from the file browser takes much longer: The Histogram and the Navigator with the thumbnail picture and the color channel data at the cursor position are displayed right away, but the editor window stays black for 25 seconds until the frame is displayed. This delay only occurs for RAW files and not for TIFF or JPEG files. Furthermore, it occurs not only after selecting the Metadata Tab for a RAW file, but also for selecting it for a TIFF or JPEG file, and it occurs without making any changes in the Metadata Tab.

16

Hello, with the appimage on Linux this works fine, that is: as usual (raws open in 2 - 3 sec).

17

Preliminary tests (appimage) works well. Metadata is shown correctly and loaded correctly from xmp sidecar files, supported lenses are automatically detected in lens profile module.

Correctly reading .xmp metadata is HUGE! This is an enormous and important improvement to Rawtherapee! No other feature could possible compete. This will enable me to tag raws rather than tag and retag output jpegs for every export. This is days of work saved in a year.

Will test in depth export/import when I find the time. The simple gui for picking tags to export seems adequate. This feature does make separate export pp3’s (as in ART) even more important though because you might want to limit metadata depending on output. Being able to store a preset and apply at export would be gold.

18

What am I missing? How do I get it to read from an xmp sidecar file?

19

It just did. There are settings to choose standard or darktable style. Note it’s about tags etc not edits.

20

It’s available in settings:
Settings>Image Processing>Metadata>Metadata Synchronization