The average film stock is 90kb uncompressed. Zip (deflate) is aprox 36kb. The largest we have is kodak_2302 and doublex, sitting at 135kb uncompressed, 51kb compressed.
They are pretty lightweight! It is the spectra_lut.f32 the one that weights something like 6mb. Just one file that the module needs, for every film and print, present and future. Although, it could get newer revisions coming from spektrafilm original author itself, which already happened a couple of times. And it could contribute to some changes in the photos across different, newer versions.
A possible 2D-delta encode applied to spectra_lut.f32 will cut its size by half and make any compression algorithm (like deflate for ZIP files) work even better.
I’m leaving some snippets for experimentation. This one goes into export_pack.py replacing the LUT write block
# FNV-1a over the raw bytes; the engine reads this value, it does not
# recompute it, so any stable hash will do. The hash MUST stay over the
# ORIGINAL values -- it is the table's identity, and the engine compares
# it against the one stamped in edit params, so it must not change just
# because the storage encoding changed (v3 delta vs v2 raw).
h = 2166136261
for b in lut.tobytes():
h = ((h ^ b) * 16777619) & 0xFFFFFFFF
lut_id = f"{lut_name}@{version}".encode("utf-8")[:255]
# SFS2 v3: the float16 payload is stored as a lossless 2-D delta code --
# 2nd-order deltas along the flattened chromaticity grid, then 1st-order
# deltas along wavelengths within each spectrum (anchor: wl index 0 of
# every spectrum, first two spectra raw). Neighbouring spectra and
# wavelengths are near-identical, so deltas are tiny and the payload
# deflates to ~1/3 of the raw float16 bytes. Pure modular arithmetic:
# exactly reversible, decoded payload is bit-identical to v2.
flat = lut.view(np.uint16).reshape(-1).astype(np.int32)
n, s = flat.size, lut.shape[-1]
d = flat.copy() # pass 1: grid deltas
d[2 * s:] = flat[2 * s:] - 2 * flat[s:n - s] + flat[:n - 2 * s]
wl = np.arange(n, dtype=np.intp) % s != 0 # pass 2: wl deltas
d[wl] = d[wl] - d[np.arange(n, dtype=np.intp)[wl] - 1]
delta = (d & 0xFFFF).astype(np.uint16)
with open(out / "spectra_lut.f32", "wb") as fh:
fh.write(b"SFS2")
fh.write(struct.pack("<i", 3)) # header version (3 = delta float16)
fh.write(struct.pack("<iii", *lut.shape))
fh.write(struct.pack("<i", 1)) # 0 = float32, 1 = float16
fh.write(struct.pack("<I", h)) # content hash of ORIGINAL values
fh.write(struct.pack("<i", len(lut_id)))
fh.write(lut_id)
fh.write(delta.tobytes())
print(f"wrote {out / 'spectra_lut.f32'} shape {lut.shape} float16 v3-delta "
f"id={lut_id.decode()} hash={h:08x}")
And these two go into spektra_sim.c. Keep in mind I’m assuming a bump and conditionals for a new version (v3) while keeping it working with previous spectra_lut.f32 file. Else, just skip it and simplify these conditionals deleting the f32 branch (:
a) Accept v3 in the header validation (line ~651):
|| fread(&hdr_version, 4, 1, fh) != 1 || (hdr_version != 2 && hdr_version != 3)
b) Decode the deltas before the f16→f32 loop (the dtype == 1 block, line ~681):
if(dtype == 1) /* float16 -> float32 */
{
uint16_t *h16 = malloc(count * sizeof(uint16_t));
ok = h16 && fread(h16, sizeof(uint16_t), count, fh) == count;
if(ok)
{
if(hdr_version == 3)
{
/* SFS2 v3: delta-coded float16. The exporter wrote 2nd-order deltas
along the flattened chromaticity grid, then 1st-order deltas along
wavelengths within each spectrum, all modulo 2^16 -- exactly
reversible. Undo in reverse order; both passes are sequential. */
const size_t stride = (size_t)dims[2]; /* SF_NWL */
for(size_t i = 1; i < count; i++) /* undo wavelength deltas */
if(i % stride) h16[i] = (uint16_t)(h16[i] + h16[i - 1]);
for(size_t i = 2 * stride; i < count; i++) /* undo 2nd-order grid */
h16[i] = (uint16_t)(h16[i] + 2 * h16[i - stride] - h16[i - 2 * stride]);
}
for(size_t i = 0; i < count; i++) pack->spectra[i] = _sf_half_to_float(h16[i]);
}
free(h16);
}
All this to save ~4mb of disk space and download! Although, arguably so, if there’s going to be newer versions of this spectra_lut, and backwards compatibility is needed, better be sure we don’t waste space.
Speaking of new film and paper pack releases, I’m pretty sure we as a community we will start creating them and sharing them in some way, so installing them will be a matter of download, copy and paste onto the install folders. Maybe they can be submitted to the designated repository or wherever this data-pack is going to live, so they will be globally available at some point if/when approved for everyone. This approval is something to think about, as it won’t be a good idea to just accept new packs just because. We need them to be used and tested by community members, for quality purposes.
Travelling, Will hopefully test tomorrow!
Not at all meant as a critique. Just curious.
