advanced photo selection with exiftool + jq

Software
1

For HDR i wanted to find all photos, that were taken using exposure bracketing while shake reduction is turned off. A combination that usually means, the photo was made using a tripod. digiKam’s advanced search can do the former but not the latter.

The general idea is to read the EXIF data and add a keyword to every matching photo, using a XMP sidecar file.

I used “exiftool -json” in combination with “jq”, to have a little more control over what exactly gets changed. Don’t wanna touch the DNG’s by accident, while playing with exiftool. jq is a query language for JSON data.

Install Requirements

sudo apt install exiftool jq

Create a big JSON file with all EXIF data (adjust file pattern as needed)

exiftool -json -g 20??/*/IMGP????.DNG > db.json

From the JSON tree select all files that match auto bracketing and disabled shake reduction. This jq query matches Pentax K-1 EXIF data, it might vary with other vendors or camera models.

jq -r '.[] | select(.MakerNotes.ShakeReduction == "Off" and .EXIF.ExposureMode == "Auto bracket") | .SourceFile' db.json > filelist

Given that every DNG file has a existing XMP sidecar file, add a single keyword “forHDR” to allow filtering in digiKam/darktable.

The XMP “subject” tag might not exists and should not already contain the keyword, to avoid duplicate tags. The needed check is made with exiftool’s -if parameter.

Test the condition and print the matching filenames, without adding the keyword.

while read L; do exiftool "${L}.xmp" -if 'not defined $subject or not $subject =~ /forHDR/i' -filename ; done < filelist

WARNING - WRITE OPERATION: Add the keyword to all matching sidecar files

while read L; do exiftool "${L}.xmp" -if 'not defined $subject or not $subject =~ /forHDR/i' -XMP-dc:Subject+=forHDR ; done < filelist

Start digiKam / darktable to re-load the sidecar files.

Pros:

  • The JSON “database” only needs to be created once, any jq queries after that are fast. No need to read all EXIF data again and again.
  • exiftool has the best EXIF support there is, including obscure vendor specific tags.
  • Elegant and powerful jq queries.

Cons:

  • The JSON database needs to be updated with every new batch of photos.

The database can also be split into multiple JSON files. For example one for each year of photos, so only the current year needs to be updated.

Please share use-cases and jq queries in the comments!

1 Like
2

What is the advantage of this over sqlite?

3

Make the sqlite workflow and queries and compare? Has anyone done this with sqlite?

4

Most of the applications that use a database can use sqlite, darktable and digikam for sure.

5

Whatever EXIF data digiKam supports is limited and already exposed via the advanced search. As stated before, “MakerNotes::ShakeReduction” is not supported by digiKam.

$ sqlite3 digikam4.db ".schema ImageMetadata"
CREATE TABLE ImageMetadata
                    (imageid INTEGER PRIMARY KEY,
                    make TEXT,
                    model TEXT,
                    lens TEXT,
                    aperture REAL,
                    focalLength REAL,
                    focalLength35 REAL,
                    exposureTime REAL,
                    exposureProgram INTEGER,
                    exposureMode INTEGER,
                    sensitivity INTEGER,
                    flash INTEGER,
                    whiteBalance INTEGER,
                    whiteBalanceColorTemperature INTEGER,
                    meteringMode INTEGER,
                    subjectDistance REAL,
                    subjectDistanceCategory INTEGER);

darktable stores even less.

The excellent EXIF support from exiftool was the very reason to use it.

6

Another JSON query to find all photos taken with a camera’s internal HDR feature.

With the Pentax K-1, the DNG file with HDR data contains three exposures, while the EXIF data of the first exposure has the HDR tag we are looking for.

To extract the EXIF data of all embedded photos in the DNG, call exiftool with -a and -g4 parameters.

exiftool -json -a -g4 20??/*/IMGP????.DNG > db.json

The JSON query simply filters all photos that have a .HDR tag with HDR not being turned off.

jq -r '.[] | select(.Copy1.HDR != null and .Copy1.HDR != "Off; Auto-align Off; n/a; 0").SourceFile' db.json > filelist

The HDR tag itself encodes details about the settings used.

Examples:

HDR 3; Auto-align On; 1 EV; 0
HDR 3; Auto-align On; 2 EV; 0
HDR Auto; Auto-align On; 1 EV; 0
HDR Auto; Auto-align On; 2 EV; 0
HDR Auto; Auto-align On; 3 EV; 0
1 Like