I find the source of that formula tricky to follow, but not because of the maths - certain ‘leaps’ seem to be made without explanation. Still reading it on occasion…
If the optimal settings result in values being clipped and you’re looking for a way to compress rather than ‘lose’ them, it suggests the curve/formula itself has an issue for your input.
Edit: some explanation of it may help
The formula written in another way is just (K/J^p)^c where J represents the set of all pixels (but this equally applies to any one pixel individually). Rewritten it can be K^c * J^(-pc) and given that K is some constant of your choosing and we can normalize to any range, it may as well just be J^(-pc).
In other words, this is equivalent to reciprocal then gamma, or simply a negative gamma exponent : J^-g
A demonstration of this using gmic… let’s start with a cat image, negate in the ‘digital’ sense then normalise to your input range:
gmic -sp cat -negate -n 0.1,0.5
At that point it’s negated. Now try a negative exponent and normalise at the end:
gmic -sp cat -negate -n 0.1,05 -^ -0.2 -n 0,255
That’s really all that’s happening with the formula.
Just to be clear, it’s basically saying negate with 1/x instead of 1-x. Then you can apply a usual gamma/tone curve in whatever editor you like so long as it has sufficient gamma range. That of course ignores any colourspace or normalisation…