The masks code is being reworked to a new internal representation (awfully code-named Flexi).
See here for the initial discussion, in case you missed it:
Before we even consider this as a serious replacement for existing masking allowances, we have to make sure that every existing edit can be migrated without any loss: your masks must render exactly as they do today, with no action required from you.
I’ve tested that this is the case against my own library — 2,466 edits, 27,803 shapes — and I verified that I can successfully migrate all my edits without issues (i.e., all masks are bit-by-bit identical, or within rounding error tolerance).
But this is just one person’s library, one machine, one GPU. My masks are not your
masks. If you use shapes, feathering, parametric masks or raster masks in ways I don’t, those combinations are currently untested.
What I’m asking for
- Run one command that exports your masks (parametric elements, shapes, inversions, what not) to a file (it takes seconds)
- Share with me the output file
I will use it to verify that your masks are interpreted and migrated correctly, as explained below.
The more of you contribute, the more I can be confident that every possible edit under the sun will be migrated flawlessly.
Another way to help is by tagging in this thread the most audacious mask builders that you know, so that they are aware of the initiative and hopefully can contribute.
What’s in the output file
It is a plain JSON file. You can open it and inspect its contents before you send it.
It only includes numbers and darktable module names: shape geometry, blend parameters, opacities, mask modes, and the pixel dimensions of each image.
Deliberately not in it: no file or folder names, no film-roll names, no image content or thumbnails, no EXIF, no timestamps, no GPS or location data, and none of the names you typed for shapes, groups or module instances. Images appear only as pixel dimensions and a sequential index.
What I will do with it
Your file gets replayed through flexi’s migration and blending code: every mask
rendered before and after migration, on both CPU and OpenCL, and compared pixel by pixel.
Your actual photos are never involved — the images are replaced by a generated test pattern, because masks are stored resolution-independent and
don’t need your pixels. Any mismatch is a bug I can fix before this reaches
you in a release.
If your library contains one of the rare configurations I have almost no test
data for, the command says so on the spot — those files are especially valuable,
so please mention it when you send it to me.
Why the replay must be done by me (instead of you)
Because if I find a mask that is not migrated exactly I can modify the migration algorithm directly, which is more efficient for all of us.
What it does to your setup: nothing
The command opens your library.db strictly read-only, writes an output file, and exits.
It runs before darktable’s startup exists at all. It never takes the library lock, never upgrades your database schema, never touches your XMP sidecars, never creates or modifies a config directory, and never opens an image. Nothing about your installation changes, and your normal darktable is unaffected.
It returns and exits before any of the code that could write to your setup is even reached.
The terminal output will look like more or less like this:
$ /Applications/darktable-flexi.app/Contents/MacOS/darktable --library ~/Documents/Photos/darktable/library.db --harvest-masks ~/masks_harvest.json
[harvest] read-only scan of /Users/dudo/Documents/Photos/darktable/library.db
[harvest] history entries scanned : 144227
[harvest] edits with masks : 2466
[harvest] images involved : 544
[harvest] mask forms : 27803
[harvest] wrote /Users/dudo/masks_harvest.json
[harvest] wrote /Users/dudo/masks_harvest.json.gz (12.4 MB, from 150.2 MB)
[harvest]
[harvest] This library contains rarely-used mask combine modes:
[harvest] inclusive mask combine: 12 edit(s)
[harvest] These exercise migration paths that almost no real edit reaches,
[harvest] so this file is especially useful to us.
[harvest]
[harvest] The output is plain JSON containing only numbers and module names.
[harvest] It has no file names, folder names, shape or group names, or image content.
[harvest] Please open it and check before sharing it.
[harvest] Read /Users/dudo/masks_harvest.json, then send /Users/dudo/masks_harvest.json.gz -- they hold the same thing.
Running the harvester
Grab the build for your platform and install it as you normally would.
Then, run darktable with the --harvest-masks option, pointing it to the output file path.
Example command lines
These are indicative, the actual command depends on where you copy/install the package. For simplicity, I am assuming that the package is installed in your home directory, and that you want the output file to be written there. Adjust as needed to suite your needs.
Linux
~/darktable.AppImage --harvest-masks ~/masks_harvest.json
macOS
~/darktable.app/Contents/MacOS/darktable --harvest-masks ~/masks_harvest.json
Note: the binary is not signed, so before the actual command you will have to run:
sudo xattr -dr com.apple.quarantine ~/darktable.app
Windows
"%USERPROFILE%\darktable\bin\darktable.exe" --harvest-masks "%USERPROFILE%\masks_harvest.json"
IMPORTANT
By default, the binary will look for your library in the standard place
(~/.config/darktable/library.db, or %LOCALAPPDATA%\darktable\library.db on
Windows). If you keep your library somewhere else, or want to harvest a second
one, use the --library flag to point it to the right library. E.g., on linux:
./darktable.AppImage --harvest-masks ~/masks_harvest.json --library /path/to/library.db
Sending it to me
You get two files: masks_harvest.json, which is the readable one, and
masks_harvest.json.gz, which is the same content compressed — usually about
12× smaller (mine is 150 MB uncompressed, and 12 MB after compression).
INSPECT the .json, but SEND the .gz.
The .gz is normally small enough to send it as a private message here on Pixls, which is by far the simplest route.
If it’s too big to attach (I would be surprised), upload it to https://litterbox.catbox.moe — no account, no email address, and you choose how long it lives (72 hours is plenty). Upload it there and send the link to me privately.
Can I also use the build for playing with the new masks panel?
Yes! Just remove the --harvest-masks <path> flag, and run it as you normally would. If it’s the first time that you try out flexi, it will copy your configdir and data to a temporary directory and run darktable against that, so that you can try out the new masking UI without writing to your precious DB. Also XMP files are saved to the temporary directory to avoid polluting your archive.
Thanks a lot for your help, it means a lot!
Thanks!
