New version of darktable-cli changes output of jpg

Hello,

I updated darktable from version 3.8.1 to 4.6.1 yesterday.

I use darktable-cli to automatically generate jpg’s from ARW’s. The command is

darktable-cli “$ARW_FILE” “my_darktable_settings.xmp” “./jpg/$JPG_FILE”

I noticed that the new version 4.6.1 generates a jpg that is looking different to the jpg generated by the old version 3.8.1, although there were no changes to ‘my_darktable_settings.xmp’.

Can you explain that? Do you have a recommendation how to generate identical jpg’s with different versions of darktable-cli?

Thank you!

my_darktable_settings.xmp (5.0 KB)

What happens when you use that xmp via dt instead of the cli?

Also, why not use 4.8.1 (our current version of dt)?

There was a bug reading old filmic rgb settings. It’s fixed in the development version, but not in 4.8.1. This may be the cause of the issue.

Same result with xmp via dt: The exported jpg looks different than those of the old versions.

(I use 4.6.1 because it comes with the repositories of my new Ubuntu system.)

I do not use the filmic module, so this can not be the cause I think…


Please have a look at the two photos I have uploaded: old vs. new.

When I run dt in debug mode, I get the following messages:

missing mandatory module rawprepare for image id=1 `…ARW’

missing mandatory module temperature for image id=1 `…ARW’

missing mandatory module demosaic for image id=1 `…ARW’

missing mandatory module flip for image id=1 `…ARW’

missing mandatory module colorin for image id=1 `…ARW’

missing mandatory module colorout for image id=1 `…ARW’

missing mandatory module gamma for image id=1 `…ARW’

Could this indicate the cause?

Your XMP file only contains some extra modules that come on top of the default one.

In between 3.8 and 4.6, there where some rework in the white balance area that could cause that problem. Since the white balance is part of the default modules, you get that result.

I suppose to solve the problem, the best would be to have a XMP that does not rely on the default modules.

1 Like

I think a bug report should be raised. The XMP migration should take care of defaults (may not be feasible in all cases, though; I’m not a darktable dev, but they would know).

It’s not really an XMP migration problem in the sense that it’s a not the development XMP stored alongside an image. In that case, yes, the migration should apply and there shouldn’t be any difference.
The posted XMP file only contains a subset of modules (like a style in DT) that adds on top of the existing development:

  • lens correction
  • color balance RGB
  • bilateral filter
  • exposure correction
  • highlights reconstruction
  • denoise profiles
  • sharpen
  • haze removal
  • crop

As you can see, no demosaic, or white balance is set by this module. There is no way for the system to know it should apply another basic development than the default one.

1 Like

Oh I see. Sorry, you are right.

1 Like

Forgive my noise if my suggestion is way off the mark. If I remember correctly and I may not, early version of the color calibration module default were not set to as shot in camera but now they are. Could this be the issue?

if the style doesn’t contain explicitly set parameters then the defaults are applied - these can be different with different releases of darktable.
That differs from the xmp sidecar files: these contains the specific parameters set while editing to give same results (backward compatibility of edits)

1 Like

Others have already pointed out the problem (incomplete XMP file, so darktable applies defaults, and yes, those can change between versions). Your XMP file does not provide all processing steps – but you should be aware of that, as you’ve manually commented its contents. Since there are no settings, there is nothing to migrate.

no, it’s not an incomplete xmp - using styles just to enable default settings is a valid approach - and in newer darktable release you can explicitly set this behaviour while creating a style.
you simply get what you defined :wink:

1 Like

Hello, thank you very much for your help!! :smiling_face:

The solution that I derive from your comments is to include ALL modules and the corresponding settings in my XMP file - correct?

If so, how do I determine the missing modules? Are they the ones listed above from debug mode: rawprepare, temperature, demosaic, flip, colorin, colorout and gamma?

And how can I determine the respective settings / params of V 3.8 for my XMP file?

Thanks again for your help!