XMP files & Darktable

DavidB · April 16, 2023, 2:25am

Hi, all! Recently, I’ve been trying to take a deeper dive into how Darktable writes to XMP sidecar files, and I have a few questions.

Consider this excerpt from the history stack of one of my XMP files:
xmp

How exactly is params encoded? I know that XMP files follow RDF/XML syntax, which imposes some restrictions on how data is stored (base64, I think?), but I’ve also heard that Darktable compresses param hex strings with gzip before writing - I’m not fully sure as to how this all happens and why.
Are blendop_params written the same way, and if not, why?

I’ve started digging through the source code to answer my own questions, but I think it would be much easier if someone with more insight could walk me through exactly how these parameters are written to XMP files and the rationale behind this process.

Thanks!

ayro · April 16, 2023, 3:19am

Here is a python project i created a few years ago to understand this very thing. Essentially the params are represented as a packed struct. This project is somewhat outdated, but it should still be relevant to understanding how the fields are packed.

If we peek inside, say, hotpixels.py, you can see how the fields are arranged. The first 8 bytes are two floats and the remaining 8 bytes are to integers.
The fields orders are defined in structs like these: darktable/hotpixels.c at master · darktable-org/darktable · GitHub. Each iop in darktable has something similar.

For modules with lots of parameters, darktable will compress the struct.

github.com

groutr/darktable-xmp/blob/master/src/dtxmp/params.py

import binascii
import base64
import zlib

dt_exif_decode = binascii.unhexlify
dt_exif_encode = binascii.hexlify

def dt_exif_compress(source):
    rv = zlib.compress(source)
    factor = min(len(source)//len(rv)+1, 99)
    output = bytearray(map(ord, f'gz{factor//10}{factor%10}'))
    output.extend(base64.standard_b64encode(rv))
    return bytes(output)

def dt_exif_decompress(source):
    return zlib.decompress(base64.standard_b64decode(source))

def dt_exif_from_xmp(source):
    if source.startswith(b'gz'):
        return dt_exif_decompress(source[4:])

This file has been truncated. show original

has routines for compressing and decompressing the params.

github.com

groutr/darktable-xmp/blob/master/src/dtxmp/blend.py

from ctypes import Structure, c_int, c_float, c_char
from ctypes import c_uint32
from enum import Enum

class blend_mode(Enum):
    BLEND_MASK_FLAG = 128
    BLEND_DISABLED = 0
    BLEND_NORMAL = 1 # deprecated as it did clamping
    BLEND_LIGHTEN = 2
    BLEND_DARKEN = 3
    BLEND_MULTIPLY = 4
    BLEND_AVERAGE = 5
    BLEND_ADD = 6
    BLEND_SUBSTRACT = 7
    BLEND_DIFFERENCE = 8 # deprecated
    BLEND_SCREEN = 9
    BLEND_OVERLAY = 10
    BLEND_SOFTLIGHT = 11
    BLEND_HARDLIGHT = 12
    BLEND_VIVIDLIGHT = 13

This file has been truncated. show original

I think these cover the blendops (or at least attempts to). Another packed struct.

Hope this helps.

Coco · February 7, 2024, 2:22pm

Hi ayro,

this is a part of my xmp:

     <rdf:li
      darktable:num="5"
      darktable:operation="temperature"
      darktable:enabled="1"
      darktable:modversion="3"
      darktable:params="00c01f400000803f0080cc3f0000c07f"
      darktable:multi_name=""
      darktable:multi_priority="0"
      darktable:blendop_version="11"
      darktable:blendop_params="gz14eJxjYIAACQYYOOHEgAZY0QVwggZ7CB6pfNoAAE8gGQg="/>

after almost one year, I have the same problem…don’t really understand how to convert xmp params to human readable value.
If I want to decompress like descriped in your def “dt_exif_decompress”, I will get an error:
zlib.error: Error -3 while decompressing data: incorrect header check

Maybe you can show me how to convert " darktable:params=“00c01f400000803f0080cc3f0000c07f”" with you script?

Kind regards,
Coco