Call for testing: RawTherapee metadata handling with Exiv2 (includes CR3 support)

@Andy_Astbury1 We really need those crash reports, screen shots of problems and errors, problematic raws and pp3s.

Regarding the command line screenshot, try open -a photoshop

Already tried that, but it won’t work.
The correct command line is the one that works in Terminal - the open -b version works in Terminal on my machine, and open -a does not.

Real frustrating, because open -b in the 5.9 release version I can revert to in Time Machine functions correctly.
And the interface does not have those buttons at the bottom


When did these creep in?
Here’s the working version

@Andy_Astbury1 Interesting. Not a photoshop user so I have no clue how to fix that.

The Exiv2 branch contains a recent version of dev because that will make it easier to integrate it into dev when the time comes. The extrenal editor changes you noted have been in the works for almost 2 years and made it into dev at the beginning of the year (delayed in favor of the 5.9 release). The two main improvements are (1) the ability to save multiple custom external editors and (2) a drop-down for changing the external editor without going into preferences.

There was a report of Photoshop and another program not opening correctly on Windows after this change, but there is a workaround in place. The same issue may be happening for you on macOS. I will implement the same workaround for macOS. Let’s see if that helps.

2 Likes

@Lawrence37 Thank you, looking forward to trying it when you’ve worked your magic :+1:

That camera may be recording the correct lens in the metadata somewhere.


I made a slight modification to the reading of the lens names for Nikon Z cameras. Let me know if there are any problems.

RawTherapee_Beep6581_metadata-exiv2_release.AppImage freezes.

Running on Linux, OpenSuSE Tumbleweed.

I’m running rawtherapee 5.9 with an empty cache and navigate into a directory with some .jpg, .arw and .orf (Olympus TG4) images. RT 5.9 works, console output is:

(process:15458): Gtk-WARNING **: 10:26:54.747: Locale not supported by C library.
Using the fallback ‘C’ locale.
(rawtherapee:15458): Gtk-WARNING **: 10:26:54.763: Theme parsing error: gtk.css:1649:16: ‘-gtk-icon-size’ is not a valid property name
(rawtherapee:15458): Gtk-WARNING **: 10:26:54.763: Theme parsing error: gtk.css:1652:16: ‘-gtk-icon-size’ is not a valid property name

Running RawTherapee_Beep6581_metadata-exiv2_release.AppImage console output is:

(process:17382): Gtk-WARNING **: 10:45:02.852: Locale not supported by C library.
Using the fallback ‘C’ locale.
(AppRun.wrapped:17382): Gtk-WARNING **: 10:45:02.876: Theme parsing error: gtk.css:1649:16: ‘-gtk-icon-size’ is not a valid property name
(AppRun.wrapped:17382): Gtk-WARNING **: 10:45:02.876: Theme parsing error: gtk.css:1652:16: ‘-gtk-icon-size’ is not a valid property name
Gtk-Message: 10:45:02.896: Failed to load module “colorreload-gtk-module”
Gtk-Message: 10:45:02.897: Failed to load module “window-decorations-gtk-module”
** (AppRun.wrapped:17382): WARNING **: 10:45:31.015: atk-bridge: get_device_events_reply: unknown signature
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.

Selecting an image gets RT freezing.
The OlympusCs warnings don’t appear if I start RT with an already existing cache. But it still freezes when I select any of the images.

Thanks for adding support of .xmp sidecars. As soon as this is working reliable, I can get rid of lots of bash scripts, which helped me to keep ratings, colors, geo-data, … in sync.

Running strace ./RawTherapee_Beep6581_metadata-exiv2_release.AppImage the console is flooded by:

futex(0x7f70400500a8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7f70400164b8, FUTEX_WAKE_PRIVATE, 2147483647) = 0

Last lines before the flood:

openat(AT_FDCWD, “/usr/lib/locale/de_DE.UTF-8/LC_NUMERIC”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, “/usr/lib/locale/de_DE.utf8/LC_NUMERIC”, O_RDONLY|O_CLOEXEC) = 4
newfstatat(4, “”, {st_mode=S_IFREG|0644, st_size=54, …}, AT_EMPTY_PATH) = 0
mmap(NULL, 54, PROT_READ, MAP_PRIVATE, 4, 0) = 0x7fd285e11000
close(4) = 0
openat(AT_FDCWD, “/usr/lib/locale/de_DE.UTF-8/LC_CTYPE”, O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, “/usr/lib/locale/de_DE.utf8/LC_CTYPE”, O_RDONLY|O_CLOEXEC) = 4
newfstatat(4, “”, {st_mode=S_IFREG|0644, st_size=357744, …}, AT_EMPTY_PATH) = 0
mmap(NULL, 357744, PROT_READ, MAP_PRIVATE, 4, 0) = 0x7fd281b2c000
close(4) = 0
futex(0x7fd28547bf38, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7fd28547bf38, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7fd28547bf38, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7fd28547bf38, FUTEX_WAKE_PRIVATE, 2147483647) = 0

Hi @Jens1. I’m not able to reproduce the freeze. Does it also freeze if you have just one image in the directory? Are you able to apply the neutral profile by right-clicking and using “Processing profile operations”? Is it also freezing with the normal development version?

Development release 5.9-146-g3d67ca450 has no issues. It also has less Gtk warnings.

./RawTherapee_dev_release.AppImage

(process:7723): Gtk-WARNING **: 08:41:14.105: Locale not supported by C library.
Using the fallback ‘C’ locale.

(AppRun.wrapped:7723): Gtk-WARNING **: 08:41:14.127: Theme parsing error: gtk.css:1649:16: ‘-gtk-icon-size’ is not a valid property name

(AppRun.wrapped:7723): Gtk-WARNING **: 08:41:14.127: Theme parsing error: gtk.css:1652:16: ‘-gtk-icon-size’ is not a valid property name
Gtk-Message: 08:41:14.144: Failed to load module “colorreload-gtk-module”
Gtk-Message: 08:41:14.145: Failed to load module “window-decorations-gtk-module”

** (AppRun.wrapped:7723): WARNING **: 08:41:41.940: atk-bridge: get_device_events_reply: unknown signature

With a small subset (14) of .orf images, the problem disappeared, but is still existing with the bigger collection (2500) of .tif, .arw and .orf (same 14) images.
When I played around, I got

** (AppRun.wrapped:6596): WARNING **: 08:33:50.825: atk-bridge: get_device_events_reply: unknown signature
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
g_mutex_clear() called on uninitialised or locked mutex
Aborted (Speicherabzug geschrieben)

It seems that there is a raise condition, simply triggered more reliably by lots of images.

Hi @Lawrence37 ,
I used official RT 5.9 to apply the “neutral profile” to the images. After that RT Beep6581 is working!
Then I recovered all .pp3 files and all images from backup. RT Beep6581 is still working!
I will use RT Beep6581 from time to time to check for the issue to reappear. How often will you generate an updated appimage? (to check for a new download)
Greetings, Jens

It’s currently in the beta testing phase, so there will only be updates in response to bugs and usability concerns. You can track the latest changes in the pull request mentioned in the original post. I will also share any significant updates here.

Hello,

Everything works fine in the Exif and IPTC data in the metadata tab of the editor display all important information, and editable fields such as EXIF copyright can be edited without any problem.
With DNG(PENTAX K-30), and NEF(Nikon D7100)
I edited an exif tag and it is saved

grep Benoit IMGP0372.DNG*   
IMGP0372.DNG.pp3:Artist=Benoit
IMGP0372.DNG.xmp: <rdf:li>Benoit</rdf:li>

I just compiled after an update of the source in the branch : Nothing to report

% git branch 
  dev
* metadata-exiv2

cmake ../RawTherapee -DCMAKE_BUILD_TYPE="release" -DCACHE_NAME_SUFFIX="5-dev" -DPROC_TARGET_NUMBER="2" -DOPTION_OMP="ON"

About->Version :

Version: 5.9-232-g294c6167a
Branch: metadata-exiv2
Commit: 294c6167a
Commit date: 2023-03-26
Compiler: cc 10.2.1
Processor: x86_64
System: Linux
Bit depth: 64 bits
Gtkmm: V3.24.2
Lensfun: V0.3.2.0
Build type: release
Build flags: -std=c++11 -march=native -Werror=unused-label -Werror=delete-incomplete -fno-math-errno -Wno-attributes -Wall -Wuninitialized -Wcast-qual -Wno-deprecated-declarations -Wno-unused-result -Wunused-macros -fopenmp -Werror=unknown-pragmas -O3 -DNDEBUG -ftree-vectorize
Link flags: -march=native
OpenMP support: ON
MMAP support: ON
Build OS: Linux 5.10.0-21-amd64 x86_64
Build date: Sun, 02 Apr 2023 17:43:20 +0000 UTC
Build epoch: 1680457400

Hello,
with the support of Exiv2 lib my guess is that the amount of supported cameras and lenses will be increasing.
It is already hard in lens correction to find the correct camera and lens, the menus are pretty large. For stuff not found by lensfun, the situation will get even worse.
It would be great to be able to filter the provided profiles to the cameras and lenses, which are in use. No automatic filter, but something configurable (e.g. filter for “Sony” or “Olympus”).

1 Like

I agree. While the scrolling menus are workable, the sheer number of entries has gotten to a point where (without filtering) a pop up dialog box with a scrolling combo list would be easier from a UI point of view. Maybe a dialog with an incremental-match search field. For example, as you type:

S - Sigma, Sony, Soligor
So - Sony, Soligor
Son - Sony

… etc.

Of course I have no idea how much dev effort that represents, though.

The incremental-match is the generic solution, but requires mouse + keyboard interaction.
Since the cameras and lenses of each user are limited and static, I would prefer a fixed filter entry in the config file (like setting my favorite tools tab by vi, or like in the preferences → file browser tab you can configure the allowed file formats in a graphical way).

We’re throwing developer time and effort around :slight_smile: but as long as we’re “wishing”, a GUI with a both full and favorites lists would meet both wishes. One small, user-editable list of favorite lenses and another full list of all supported lenses.

I usually work on images from only my lenses, but not always. Between friends, Play Raw and other resources I virtually “touch” various lenses from time to time and having to edit a file (then presumably restart the editor) would be clumsy IMO. Plus, I’m already using a mouse / keyboard while editing anyway.

But what’s already implemented does work. However it will reach a scaling threshold at some point…

I agree that some sort of dialogue/filter UI solution would be nice. Meanwhile, what I’ve been doing is creating a few partial profiles for my manual lenses. Then, it’s just a matter of selecting the right partial profile file when I open a raw file taken with a particular lens.

Thanks for testing @benoit!

@Jens1 @lphilpot I have also noticed it can be hard to find certain cameras and lenses when the lists are long. GTK probably supports list filtering, so adding a search feature could be as easy as flipping a switch. As noted by @MarcosC, partial profiles can be used as a substitute for a favorites list. Combining them with dynamic profiles allows for automatic selection if the lens and camera details are present in the metadata. If you only need lens corrections for your own cameras and lenses, you can create a copy of the Lensfun database, stip the unneeded profiles, and tell RawTherapee to use the modified database

1 Like