nah, it’s just lazy. i mean i could have used rational splines or what but this is just pysr/symbolic regression running on the spectral data and the cr/cb projections directly. i confined the search to +,*,/,cos() other than that it’s just the optimiser defaults. sometimes this reveals structure in the data, this time it doesn’t have melanin() and hemoglobin() to make sense of the values 
I wondered: would measured camera-SSFs “help” your RGB-to-Spectral algorithm @hanatos ? As in: improve the spectral guess?
Forgive me if it they were useful in creating the algorithm in the first place, I haven’t read the paper yet.
@Dissipatio has posted in the other thread a comparison of Film vs. Alexa and for the Alexa I think someone has measured and published the SSFs iirc.
Our @ggbutcher has a nice collection at Github SSF data including the Alexa.
Or maybe all of this is trivially obvious knowledge already? Sorry.
I have a larger collection, reticent to post some without a clear license. Thinking about that, using such to generate a camera profile would be okay, not sure about transforming it to another format like dcamprof json…
initial integration of skin spectra into spectral upsampling:
2 Likes
didn’t know about this kind of regression, sounds fun though!
i do not think this is trivial and obvious. the way i see the framework of the problem at this stage is: limit our input to RGB/XYZ from RAWs >> have strategies to reduce the roundtrip exposure errors on measured reflected spectra, while keeping the smooth rendering across the xy plane.
from the film side we are already fully considering the sensitivities.
how the specific camera sensitivities can inform the upsampling alg to make use of the additional information that might be present is a cool direction. but sounds less universal and not very portable.
as far as i understand it, the sigmoid spectra are obtained from an elegantly defined problem (and fitting strategy) that just considers 1931 cmfs and XYZ roundtrip error. so it is very self contained and not aware of the camera sensitivity transform at all.
1 Like
i had to watch this video several times in a row. so satisfying to see the smooth changes. and so cool to have an interactive way to visualize it in vkdt!
i believe that we might want to have a transition that is as smooth as we can across the xy plane. maybe even smoother than the actual distribution of the nist-dataset chromaticities. but data and output will speak about this.
if we ever feel in the need of more spectra (with possible more variability) there is also the ISSA dataset The International Skin Spectra Archive (ISSA): a multicultural human skin phenotype and colour spectra collection | Scientific Data with 15k spectra on 2k subjects. they pitch it as a modern multicultural human skin phenotype dataset.
i can try to get them and plot the xy distribution and see if it can help in this.
3 Likes
absolutely less universal.
But when comparing RA-4 scans (what’s the scanners SSF? maybe less important for sampling the film dye densities as scene-spectra have already collapsed into a CMY triplet? not sure.) to a camera-RGB of the same scene, it might be useful for tuning to have an accurate idea how the camera responded to a spectral input.
The “worst” benefit would be that minimizing the roundtrip error converges faster, no?
yeah absolutely agree! and this gets outside of my expertise really fast, but i am happy to grow and learn.
maybe trivial set of comments, just reasoning loudly.
we are not working with an hyperspectral camera or spectrometer as input data for the simulation, thus we are accepting information loss in the projection of the the scene on a tristimulus. any upsampling from a tristimulus will work with reduced information and give out a “smooth/average” guess of the scene spectra. let’s say we work with raw readout of a camera and we upsample with the best possible alg. let’s define a good alg to have zero round trip error on the camera SSFs (similar to the sigmoids on 1931 CMFs). there will be a family of such algorithms, all sharing zero-error, but with slightly different characteristics. which one is the best and why?
any zero-error alg will give a deterministic solution to the RGB_raw → spectrum problem. one triplet in, one spectrum out. now, real life scene spectra will always have metameric variability (same projections on SSFs but different shapes, equally for CMFs of course), and real life is complex. thus the projection of the upsampled spectra on film SSFs will have a “metameric noise distribution” compared to the real projection of the scene on film SSFs. among the classes of zero-error algs, the best one is the one that will minimize the metameric error distribution cloud. but we will never have zero-error because of the information loss in step one.
to define the error cloud, and converge on an algorithm for the second step, we need to define a good corpus of realistic spectra that mimic what we are actually imaging. there must be an infinite family of algs that minimize the error on infinite corpus of spectra that we might assemble (maybe imaging Mars rocks with a skin optimize alg will give intolerable errors).
but essentially we are trying to find a good solution for the transform camera SSFs → film SSFs (i believe there is ton of science and expert smart people in this that might have very good opinions including you guys). we want to minimize the metameric errors that inevitably will be there. so we should also probably be aware that working to correct issues that have smaller errors might just act inside of the error cloud and not change too much the transform.
starting from an alg with zero error on 1931 CMFs, is a similar story as the step one and two on camera SSFs. i guess there is a certain level of similarity in the error clouds of camera SSFs and 1931 CMFs, since both mimic the human vision. so the big question is, how impactful is starting from the camera SSFs and how much would this minimize the total metameric exposure error? the answer might be that the error difference is within the spread of the round trip exposure error clouds, or maybe it would be very impactful, especially for SSFs that differs more form the 1931 CMFs.
many words to say that, of course i don’t have any answer in this. and i am probably the less knowledgeable person here to have the most accurate opinion. 
and I agree that scanning RA-4 and CMY dyes should make the issue quite “stable”, because we have a very well defined class of scene spectra.
3 Likes
and, here a small analysis of the ISSA spectra.
they have a little more variability than the NIST dataset and they cover a slightly larger chromaticity area. they are delivering in the promise of a multicultural dataset i guess.
(ISSA are dark brown dots, NIST light brown)
here some random spectra from the dataset, and the Y-normalized upsampled twins.
here some more stats on the full corpus and distance from hanatos2025
the standard deviation band of the ISSA spectra is not that much larger than the NIST one. but it is indeed broader. also the mean has tiny differences.
overall the ISSA dataset seems to be a refinement on top of the NIST if we want to upgrade, i believe it might not require changes in the symbolic regression model, just fine tuning. i like that it has a softer shape in the xy plane.
2 Likes
1 Like
awesome. i was worried my c_r, c_b gaussian is so small. will switch over to the larger set.
yes. there’s a thing about pysr assuming full dataset evaluation at every step of the optimiser. but they have a mini-batching switch that i already use for the nist dataset. also in the grand scheme of things this optimisation didn’t take all that long.
camera ssf can be very different from the 1931 cmf. after good input device transform this is really only about metamer mismatch. the spectral input transform vkdt uses has better colour accuracy, i can especially see that for the Canon 5D Mark II (i should recreate it with the skin spectra too…). but the killer feature is that it doesn’t produce impossible colours (negative L) as the matrix input device transforms routinely do.
unfortunately limiting this feature to cameras with known/good measured ssf rules out quite a few camera models. i’d err on the side of portability here for the film sim.
3 Likes
Measuring camera SSF isn’t insurmountable, so I would instead consider this feature to be an incentive…
2 Likes
But analog film has rgb-sensitivities which are (deliberately) different from their print dye absorption spectra, which leads to inherent metamer-mismatch (if i understand metamer-mismatch to mean what I think it means, could be wrong on this).
Repeated analog round tripping would lead to very very wrong colorimetry fast.
On top of that their sensitivities also are different from 1931 cone fundamentals AND different from camera-rgb-sensitivities.
Therefore I think one has to be very careful with the optimization goals.
If RA-4 scans shall be matched…well, that needs some kind of reference scan to be sure what XYZ values actually are in the RA-4 scans.
All of this with the caveat that I hope I understood everything that is going on! Again, I could be wrong.
than that sounds great! and we will have automatically a slightly broader gaussianoid in the xy plane.
i agree with the fact of being careful about matching strictly the RA-4 scans. i think there is no will in having a 1:1 match or to fit the model on them. the idea is to use them as a guide, and doing it in the most fair possible. if considering the scanner SSFs will be necessary, then it is the way to go.
the beauty of spektrafilm is the self contained all-physically-based-model that starts from only spectral densitometry and have everything embedded through modeling and a small set of parameters (not so small ok ). the json profiles could be also compressed with simple models without major loss in the output i believe (would be fun to do actually).
it is like slowly distilling the film look by modelling the essence behind it. we can probably reach up to a certain level of accuracy (maybe not much higher than what we have right now), and i believe we should be happy with it. everything produced by the model is physically believable and act withing known physical rules.
ps: having the full parametric model of the jsons would also create another dream that i have in mind >> having a way to blend in between stocks or having a continuum between all the stocks. further rationalizing and experimenting in this direction might bring us even further with additional creative freedom of generating families of fantasy stocks with intended behavior linked to similarities in the parameters bundles (maybe through pca or similar, who knows). and in this picture, having a LUT-ish strategy correcting on RA-4 scan, or some similar procedure would go against this grand plan. instead understanding why major shift between RA-4 and output are present, might inform the modeling and assumptions within it, that is the real precious knowledge we are after (my personal opinion ofc).
5 Likes
Evolving for 10000 iterations... 0%|▏ | ETA: 1 days, 9:28:2 Info: Full dataset evaluations per second: 1.20e+02. Press 'q' and then <enter> to stop execution early.
uhm… thanks for the hint^. i might have to make use of this option.
sigh i know you two are right. and in fact the way the spectral input device transform clut creation works is by hallucinating spectra for the whole lut range in the first place. these spectra are then multiplied by known ssf and compared against cie cmf reference to create a mapping cam rgb → xyz. it’s kinda straightforward to skip the mapping part and just output the spectrum which would have created the given camera rgb values (together with known ssf).
i don’t want to drop support for unprofiled cameras, but in terms of self respect i think this direct codepath is a necessity
4 Likes
this is a loooong optimization!
fine tuning could be an option. or you could randomly select a subset of spectra to reduce the size to a manageable problem. but i am pretty sure you’ve already come up with a super smart solution
to the contrary, was swimming in the lake, it’s hot here. error didn’t really reduce during that time, played with 3d polynomial regression for all of a minute or so… had no trust in managable number of coefficients and then gave up and it’s now running on a random 5% subset of the data. we’ll see…
noice! i played beach volleyball, swedish May and feeling too warm is such a treat!
i added a few more spectra dataset to the corpus just to make it more general and see how a larger landscape of natural reflectance would cope (compared to mainly munsell patches).
i added:
- a repository of minerals/rocks/soils/vegetaion/organic-compounds/liquids (USGS https://www.usgs.gov/publications/usgs-spectral-library-version-7)
- a repository of plants/animals/stones (https://figshare.com/articles/dataset/Spectral_dataset_of_natural_objects_reflectance_from_the_Southern_cone_of_South_America/25705380)
- munsell glossy from
colour-science - soon i will receive the ECOSTRESS library that i kindly asked from JPL (https://speclib.jpl.nasa.gov/)
skin is in the full corpus for the minimization in this case.
you can see a sort of skin colored horn protruding from the center of the cloud in f2 blue channel, in the red channel it is more like a blob. human skin is really strange.
the new corpus ruined the optimization of the hyper parameters (thus we do not have the flattest pancakes for now), but it shows that with more variability from the additional spectra, the surface correction becomes a little less relevant. i still think it is a worthy gain, though.
1 Like
so there’s a full story about going back and forth with the optimiser… and i don’t think it’s over. but i have some initial test images:
even though this is downsampled for web (srgb, lowres, etc) i think you can tell the difference. this is portra 400/portra endura which gives this nice and warm glow to everything. the vanilla spectral upsampling looks a bit inconsistent in the skin tones, methinks. has spots with pinkish tones, not this consistent warmth throughout. the NIST spectra stay high from 700 to 800 and have one more characteristic dip in this range. ISSA only provides data up to 700nm and in coarser intervals, so for now i’m going with NIST. the range around 600-700 seems to be important for portra, so i think actual skin spectra might be a great feature.
not sure about interaction with non-skin makeup + illuminant. the spectra here are really made for D65 and round trip through cie XYZ for now (via input device transform).
5 Likes
this is great!
and it is a very good sign, the additional warmth is exactly what we are looking for. in my screen (calibrated now with a second hand colormunki spectrometer) the portrait is a bit to yellow. i think the realism of skin tones really shines and makes a difference when the withebalance is more neutral.
the portrait on the left is more rich and “healthy” looking. love it!
3 Likes