Yes, good regularization, extrapolation and sufficient color space coverage in the samples is really challenging!
I’ve also found that, even when using highly saturated test patterns, estimation for extreme color values often just doesn’t work well. On very bright and / or saturated gradients, visible artifacts occur.
I wonder if this is really due to our algorithms or maybe a problem in the sample data generation process.
The handling of out-of-gamut colors / highlight clipping in the color space transformations in the pixel pipeline of Darktable might be an issue, for example, if there is no bijective mapping between the resulting colors of the RAW and JPEG samples anymore, for example.
In the latest release, I’ve added some more options and also a few charts to output for visualization. Have fun (see the Readme and the --help option)!
Also, I’ve tuned the optimization and extrapolation a bit (still not a sophisticated approach, but let it be for now) and added a simple test image generator for sample generation.
IMO it is really important not to overparameterize. E.g. a 9x9x9 Cube already produces visually identical results to my camera’s JPEGs. Whereas this is a low-resolution cube, it still has 729*3 Parameters! Given the fact that images are far from a uniform sample distribution over the colors, this is already a lot and good generalization perfomance is hence difficult to achieve.
Hmm, have you tried to tune the ECC alignment parameters? Interesting that it does not work for you. For my images, it works flawlessly.
EDIT: Here are Fuji LUTs generated with the current version and the sample pattern generator. It is important to remember setting the interpolation to trilinear and the color space to adobe rgb in the 3D lut module
fujis.zip (33.7 KB)
PS: have also added the link to your script to the readme; great