hi, i created this topic for getting deep in tech discussions surrounding the development of film simulations currently in spektrafilm, vktd and ART. feel free to join in with suggestions, ideas, and you are more than welcome to be critical and peer review 
splitting here Spectral film simulations from scratch - #678 by hanatos from the original thread on spectral film simulations
nice, let me start then
so i found these NIST skin reference spectra online:
so i was thinking, maybe the metamers are more of an issue than the windowing, at least here. your plots in the other thread also seem to point that direction: the xy shift made more of a difference than the window if i read this right. since in general we don’t know any of the spectral shapes, maybe it’s useful to special case the skin tones to these characteristic spectra. i suppose it’s not hard to blend spectra for colours with approximate skin tones to some say PCA-based model reproducing these characteristic peaks / the dip around 550nm.
about your other question: the upsampling could easily use D55 of course, but it was my impression this is used during printing. exposing is likely done from a raw file that mostly comes with a D65 input device transform. maybe useful to devise blendable D65/tungsten tables again, like the dng input transforms or vkdt’s spectral IDT. probably a separate issue though, multiplying an illuminant to a skin spectrum is kinda easy.
Is there no interest from the darktable community in this idea? Just wondering?
I would suspect the community per se would be thrilled but currently there seems to be nobody capable and available to champion that project.
just played around a bit with the dev/main branch of spektrafilm.
two questions:
for film sensitivities there is no point in going past 780 either, even less than that would actually suffice. the upper head-room in spektrafilm wavelength domain is to accomodate the dye absorption of developed film and sensitivities of photo paper, that do not have to be connected to human vision and thus make smart use of the red side for purer print transfer.
after some more experiment i have to say that the window can go a long way (exploiting a better regularizer that tries to flatten the tails of the corpus cloud). the small effect is only when we constrain the window to not cut into the visible.
here another experiment with max channel errors on the corpus and more freedom for the window to cut in the visible:
i am kind of very curious to see how spectra optimized with a window would perform, i believe that the window will force the metamers to make less use of the lobes and be more compact in the visible in a gradual way (i don’t know if this will mean more difficult sigmoid solutions, or maybe not possible with zero error). i believe this “compactness” would be absolutely a good behavior for our problem.
i agree that skin tones should have a particular treatment. film photography was developed with attention to memory color,s and human vision is very sensitive to skin tones, so i would put a bias on it. in the sense that if the reflectance of a brick wall and of my skin give the same XYZ, we would not have any way to generate two different solution in our problem, so i would lean to prioritize accuracy for the skin (we are still talking about 20th of EVs error that are not a lot for perception of generic material colors i guess).
that is another viable solution indeed, and could be interesting to explore
my question was coming from reading the data-sheets of film and in general other technical documents about film in which usually a sentence like this is reported: D55 represents typical daylight average conditions with sun 40deg above horizon; or D55 better simulates practical day photograpy than D65, film standards prioritize real-world shooting conditions. straight from of my obsidian notes, but i bet there must be references.
a few more plots, “just because”…
here a few more result from the same experiment above. i raised the weight of the skin spectra, and i added the colorchecker babel dataset to anchor saturated memory colors, so the skin dataset bias does not swamp to much yellows and reds and does not hurt too much the neighboring greens. we pay a little price on otsu and munsell, but i am pretty sure there is still a little gain to reach a better compromise (ie weights and reg tilts), everything was manually tuned here (i will send an optuna hyperparameter search one of the next nights when i will be happy with the regularization 
).
apparently the forest color library is just easy to guess (i should reduce its weight to let more freedom to otsu/munsell).
there is not, but will have a look into it soon enough, i have not much experience in this kind of things. there is an issue on github on a similar note.
it makes use of some automatism from Pillow.ImageCms that if i remember well is able to retrive the monitor transform only in Windows. there is a little note in the tooltip of the gui control.
here a small additional analysis on the upsampling discussion. i am making some statistics on the real spectra and the upsampled spectra, detaching a bit from the the correction problem, and just look at metameric mismatch and where it might be symptomatic.
i’m taking the reflected spectra by D55, normalize by Y, then computing XYZ and upsamling, then comparing the two pools.
i divided the corpus pool of spectra with two lines in the xy chromaticity space, “purples” and “greens”. and i compared also with the full corpus.
first i took the purple pool. here are 100 randomly selected spectra from the otsu pool.
we can see the effect of the near uv and near ir lobes.
below, the mean spectrum show more clearly the overshooting.
from the residual plot we notice that the upsampled purples miss some energy from 450-500 nm and a little from 600-630 nm, and boost below 440 nm and above 660 nm. there is a little boost also 550-580 nm (but i suspect it should partially resolve by flattening the energy losses and normalizing by Y). this means that whatever funky film sensitivities we decide to use, a more realistic upsampling alg should try to compensate for those. the more we go to the uv and ir the less problematic because also film sensitivities will decay (they are just a bit wider than 1931 cmfs).
in other words, fixing those boosts and cuts will make for a metameric upsampled base that on average matches better the real measured corpus (for the purple pool). and from this point of view, compacting the spectra in the visible, thus filling the energy losses is a good idea for a more realistic rendering by default, especially in the near uv side.
things get slightly more complicated if we average the full corpus of spectra.
for full otsu, the energy loss at 450-500 and 600-630 nm is still there, although less strong. skin rendering shows strong energy loss in the same spectral areas. interestingly the far red side of full otsu residuals flattens. forest spectra are more structured and have small loses at 450 and 650 and peaks at 500 and 600 nm, then a gigantic energy pit above 700 that is not really seen by typical daylight sensitivities (of course might be relevant for ir sensitive film).
apparently the energy boost at 500 and 600 nm in the forest do not interfere too much with the current version of hanatos2025 (i.e. easy to compensate the error from previous experiments), possibly because they matches the dips in between RGB sensitivities. this also suggests that we should take extra care in the rendering of the spectra in the region 450-500 and 600-650 nm because it is where film “sees more” the spectra (the 550 center is also important for green of course, but i guess zero errors on cmfs should already help out there).
if the full otsu corpus shows reduced energy losses compared to the purple sub pool, it means that the green side should behave better. below is a similar analysis for the green sub pool.
due to the nature of the parabola-sigmoid of hanatos2025 now we lack energy above 630 nm, and that is the reason of the flattening above on the full otsu corpus. overall the blue-uv side behvaves quite well. greener forest spectra still do their thing.
some considerations:
hanatos2025 with a window should push the solver in moving the parabolas or the flipped parabolas a bit inside the visible, thus a “good window” might be able to partially solve the energy lossesfinal plot for this round, i promise.
if we make an experiment and we optimize the exposure errors on portra 400 with bandpass+suface and the otsu/munsell corpus we get the following errors on the other spectra sources. errors on ostu/munsell squish quite ok in the pancake. forest also hides pretty well in the core of the cloud.
we notice that there are residual tiny problematic tails in the otsu/munsell library (mainly a narrow yellow-green tail). with a tuned regularization i suspect that we will be able to partially fix the tails that sit at the edges of the cloud.
skin is a totally different beast. it hovers above and below very dense areas of the red and blue clouds, making it impossible for a surface to really fix both. pulling the blanket on one side will create distance from a cloud or the other. skin green channel is ok as expected.
Thanks for the quick answer.
thinking about this, i don’t want to do pca, though it’s probably going to fit skin super well within only like 3 or 4 basis functions. but then i’ll have to upload / access data again: eigenvectors + coefficients per chromaticity. there’s a ton of literature specific to measuring and fitting skin spectra that seems to indicate pca is one of the better ideas.
these are some preliminary results trying to find a closed form regression to the data (wavelength, cr, cb → normalised spectral power). sorry my plots are nowhere near as polished an nice as yours. doesn’t work very well so far. my beautiful dip at 560 is gone. thick lines (first four) are reference data examples, the ones with the points are the fits. the thin lines are an alternative expression, and the thin yellow is the average of all skin specimen:
the yellow average has the dip, so i’m guessing the pca would resolve it really fast.
especially above 600nm the fit is pretty terrible. this doesn’t appear to affect chromaticity coordinates much though, i.e. colour normalised to overall luminance scale.
also possible that the chromaticity coordinate doesn’t hold enough differentiation power to tell the spectra apart.
i agree that the chromaticity alone might hold little information about how the specific skin spectrum from the pool behaves after 650 nm, or the capability to select a specific spectra from the four you plotted for example (in general this is true for the other categories of the corpus i believe). i think we should aim to have an average behavior that is acceptable, and contain the exposure errors within a reasonable range.
so under this consideration, your model here already does not look that bad.
what is still pretty oscure to me is what kind of tricks are used to blend a model like this one inside the sigmoid spectra, in a nice and smooth way. i mean such that it does not create problems on film sensitivities across the xy plane.
for example if i compute the exposures on portra 400 and LMS sensitivities from the otsu2018 upsampling pca-based algorithm across the xy plane, i start seeing clear “solution” domains, while hanatos2025 is always impeccably smooth. i guess this is because of the algorithm is targeting to respect chromaticities only. the additional energy on the purples for blue/red in hanatos2025 is also clear.
LMS are decently smooth for otsu2018, too. but the sigmoid spectra are soooooo smooth
also i found some more natural sources of spectra to compare the trend of the purples pool (below the “purples” line). some purple fruit, leaves and flowers from japan (Spectral dataset of daylights and surface properties of natural objects measured in Japan). they are from japan so they must have been photogenic too.
just to confirm that the otsu dataset is representative somewhat for the blue/uv behaviour. natural plant matter seems to generally reflect much more ir compared to the otsu dataset (that was predictable anyways, but does not seems to be a huge problem at least for greenery stuff).
the structure around 400-630 is approximately conserved. this just reinforces the intuition that trying to structure the sigmoid spectra solutions to this general behavior is probably good for realism on any sensitivity set.
hm portra is super highly peaked in the reds. will work a bit to get a better match to the skin spectra in this range.
i’m working on the assumption that i can use sigmoids throughout and then have a small/statically defined blend weight blob for just the skin tones, where i’d smoothly blend over one spectrum to another. since they should be metamers (for the cie observer anyways) this should not be visible under human observation and smoothly change behaviour to more realistic/skin like output under portra 400.
defining where skin tones are may imply some sort of known illuminant. if it works under daylight i suppose we can fix the chromatic adaptation later.
ok, that sounds like a very good idea!
very excited!
one nice thing is that the skin spectra seem to be roughly aligned with the black body line, thus i expect some slack with the chosen illuminant. but it does not remove the problem. focusing on daylight sounds more than sane for now.
current run. thin: reference. bold: approximation based on only cr,cb (derived from 1931 cmf, i assume that’s what we have after input device transform, normalising away the effect of different cameras).
every other plot is just the residuals of the 10 spectra in the primal plot (reference - approximation).
reference spectra are the first 20 averages of the nist skin reflectance dataset.
i’m fitting a more complex model too to see how far i can push this. other than that i think maybe this is close enough to attempt an integration test, blending this with the sigmoidal spectra.
now i’m also curious whether this could in general improve white balancing of skin tones…
looking great! i see the dip!
spectra overshooting in the ir seem more difficult, but overall we have a very clear shape match.
same, super curious! happy to do some testing later on with the exposure error clouds, and on sample portraits.
from the plots above in the thread, skin on portra 400 has a red exposure shift of approx -0.2ev from the otsu/munsell plane, while the blue channel apporx +0.15 stops.
with a simple script we can guess the visual appearance of such a wb shift.
this are exposures on portra 400 for random nist skin reflected spectra on d55 (brutally represented as sRGB).
this are deltaE if we apply the exposure shifts to the linear rec709 rgb from the same reflected spectra.
a deltaE of about 4+ should be reasonably noticeable. and skin tones will be by default a bit warmer compared to otsu/munsell standard objects (and midgray reference). i think i find myself adding a little nudge of warmth for skin tones more than i don’t (but possibly just confirmational bias in my memory, who knows).
hmm \Delta E\approx4 sounds a tad on the high side. considering you measured it against portra and the input comes from the cie cmf red/blue i’m not sure i can expect more. here’s what the most expensive model i tried so far says:
(((((((cos(cos(((x2 * (((x0 + cos(x0 + x0)) * 0.1119975) + -66.24202)) * (x2 + 0.4276783)) + 13.629569) + 0.47141406) * (cos(x0 * ((x1 + -0.33281153) * (x2 * 0.9949248))) + (x1 * (cos((x2 + inv((x0 * 0.05602691) + -3.7870345)) * ((x2 * x0) * 0.30428776)) + (((((x2 + (x1 * 2.7831793)) + -1.1299064) * x1) * x0) * x1))))) + cos((x0 * 0.05609102) + 3.0112202)) * x1) * 3.7629797) + -71.9527) + (x0 * 0.12663795)) * ((x1 * 2.7151208) + ((x2 * 1.0508473) + -1.2616391))) + ((cos(cos(cos(x2 * ((((x0 * 0.13795727) + -68.15959) * 0.48809084) + cos((x0 * 0.046920378) + -3.4562838))) + -0.13253786)) + x1) + x2)
where x0 is the wavelength in nanometers, and x1 and x2 are cr and cb, respectively, with c_{r,b} = \frac{r,b}{r+g+b} computed from bt2020 tristimulus values r,g,b.
this might not be the best trade-off in terms of error vs. compute, but might be a starting point for implementation testing.
[cough writing this i noticed i used the wrong colour matrix, so cr cb are something else and i need to do the fitting again…]
i just eyeballed the exposure shifts from the “otsu surface”, it’s maybe closer to -0.15ev red and+0.1 blue. anyway, deltaE of 3-4 sounds like good gains on the table to grab. especially when natural and juicy rendering of skin is one of the main targets of film photography.
can i ask you what is the rationale behind the construction of such a model? i’m kind of curious. 25+ parameters sounds like a fit for skilled people.
even if i am not sure if “window+surface sensitivity adaptation” will make the cut, i pushed a bit the tuning of the regularization and managed to get better results for the squishing of the otsu+munsell cloud. i removed skin, is too much of an outlier, and forest colors is not really a problem (for a the stocks that i tried).
we get much flatter error clouds!
also just the bandpass can be pushed to get quite a reasonable level of squish, without addressing the tails.
flowers and fruits are kind of fun: the funkiest spectra. nevertheless, the very worst ones are still within half ev.
among the residual problems on the ostu/munsell squish, we have the yellow-greens tails of the blue channle (partially also in the red). the edges of the cloud seems to split and the greener part overshoot more uv that the yellower one.
i am codenaming in my notes this issue “the fish”.
i will have a closer look to the shape of the spectra in that region and see if fruits/leaves/flowers behave similarly or if is just a fluke of the otsu+munsell patches pigment composition, that have the uv fish behavior (aka drastically different uv reflectance for close chromaticities).
later tonight I’ll make a few examples in the main thread that shows in photos what this actually mean, so we’ll have also a perceptual intuition on the impact on skin tones, and overall color rendition/accuracy. and if the surface correction is messing up or improving “the feel”.
i hope to not see fishes in the output images. 
