Any interest in a "film negative" feature in RT ?

#1

Hi, this is my first post on the forum. I’d like to ask the RT developers if there is any interest in adding a feature to RT, to facilitate the task of DIY “scanning” of color film negatives using a digital camera.
The traditional methods described in the RawPedia “Negative” page, have some annoying drawbacks (inverted control sliders, need for manual tweaking to get the colors right, etc). After some tinkering with the RT source code, i found a simple solution that looks promising.
This method is different from the one used in Darktable’s “invert” and digiKam’s “film negative” modules, and seems to give better results (see below for details). My idea is to work directly with the raw values from the sensor, upstream of white balance, and to add a “Film Negative” tool panel in the Raw page, where the user can select the film type, or manually enter the necessary parameters for the formula.

So, what do you think? Should i go on and try to implement this? Would it be a desireable feature in RT? Or do you think it would just increase software bloat, for a functionality that is rarely used? I realize that diy film scanning is a very niche problem, so i don’t know if such a feature really belongs in RT.
Maybe it would be better to do that in an external “pre-processor” program, creating an intermediate file? That would be closer to the unix philosophy… in this case, see the end of the post for a ready-to-use gmic command line that already works pretty well.

In any case, i’d like to hear some opinions. Thanks :slight_smile:

== Details ==

The approach i found is nothing new, actually i took it word by word from Wikipedia:


this article contains the following sentence:

[…] the transmission coefficient of the developed film is proportional to a power of the reciprocal of the brightness of the original exposure."

This is different from what happens when we apply the negative HaldCLUT, for example: in that case, we’re doing MAX - v (where MAX is the maximum value, and v is the current value for a channel).
Instead, the article suggests we should be doing k*(1/v)^p, which is much different.
In fact, lurking around in the forum i’ve also found an old discussion pointing out this exact issue:

Then, i had a look at the Darktable and digiKam source code (both have a film negative module), and to my surprise i discovered that both seem to use inversion (MAX - v).

Darktable:

digiKam:

So, i modified the RT source code in RawImageSource::scaleColors with a quick and dirty patch in order to compute the formula. For now, i read the exponents and coefficients from a text file, just to try things out (no GUI or integration with settings data).
Then i created a LibreOffice spreadsheet to calculate the parameters based on known values sampled from a test picture.

The most annoying thing to do with the traditional “inversion” methods, in my opinion, is getting white balance right: most of the time i pick a light gray spot for WB and all seems ok, but then i notice that another darker gray spot has become somewhat red. So i pick that one, and the previous brighter spot becomes blue-ish. So i adjust the RGB Curves and after some tweaking, i finally get perfectly balanced grays all across the range, BUT… i am now bound to that exact brightness level; if i make a slight change in exposure comp., brightness, contrast, tone curves or whatever, the histogram moves along the X axis in the RGB Curves, and the balance is gone. I can only use Lab controls from this point on.
The same applies when i process another negative: if i change the light source or exposure slightly, the RGB Curves created before need to be retouched.

So i decided to concentrate on this white balance problem; to make a cheap test, i took a picture of a color checker displayed on my PC monitor (i don’t have a real one; at least the screen is factory calibrated :smiley: ) with a Kodak ColorPlus 200 film roll.
Then i digitized the developed film, using a Sony A7 and a speedlight as a light source (xenon).

In the attached spreadsheet you can find the channel values from the 6 gray patches in the bottom row of the checker. These were obtained by reading the channel values inside RawImageSource::scaleColors (so they are normalized 0…65535), and averaging an area of 32x32 pixels.

negativeCalc_curve.ods (25.5 KB)

The “p” and “k” values are the exponent and coefficient, respectively, for each channel. The B channel was used as the reference. These parameters are calculated based just on the first and last patch values, not taking into account intermediate vaues. In the graph on the right, you can see the results: the curves are not perfectly overlapped, but not too bad, either.
At this point i fed the parameters in the formula in RT (multiplying by a global factor to re-normalize the output in the rage 0…65535), and the test chart looked pretty good. If i change exposure comp. or white balance, or brigthness/contrast, now everything remains stable.
I had a couple of color patches that were definitely off, but to my surprise, those were easily fixed by using my camera DCP profile, and enabling the look table. I thought that those corrections would not work after mangling the channel values… anyway, this doesn’t matter: i could also have fixed those patches via Lab hue equalizer.
The final result was not bad. I’ve also tried digitizing the same checker negative using my smartphone (which produces raw DNG) and, with the same parameters, the output was good (using the DNG-embedded camera profile here, too).
Then i’ve tried some other negatives (also from previous rolls of the same type), and the results seem quite stable.
Here you can see two examples showing the difference between inverting the tonce curve and using the formula described above.

inv

As you can see, the light gray patch has a blue cast, while the dark gray tends towards red. Impossible to white balance without touching the RGB curves.

recip

This instead is using the formula with the parameters calculated in the spreadsheet, as it appears “out of the box”, with just white balance and exposure comp. The WB is much more constant across the entire gray range.

ex1 ex2

Here are other 2 examples from a different roll of the same type, processed using the same parameters calculated from the checker above. No tweaking, just WB, exposure and contrast.

As a bonus, below you can find a G’MIC command line that implements the same formula, with the same parameters from the spreadsheet. For example, try to download the CR2 raw file from this tutorial

http://www.frogymandias.org/imagery/camera-scanning-dcr.html

(download link on top of the page). Get a linear tiff from the raw file using dcraw:

dcraw  -T -4 -r 1 1 1 1 -o 0 -h /tmp/60D_11930_negative.cr2

and finally, run this G’MIC pipeline:

gmic /tmp/60D_11930_negative.tiff \
  -fill '[(R^-1.57969792927765)*149.305039862836,(G^-1.15851358827476)*3.91704903038636, B^-1 ]' \
  -fill '[R*1.3,G/1.3,B/3.2]' \
  -cut 0,{ic*4} \
  -apply_gamma 2.2 \
  -adjust_colors 0,30

The first “fill” command contains the formula. Here i used negative exponents instead of doing 1/n , it’s the same.
The second “fill” command does white balance, those coefficients were just eyeballed.
The “cut” command limits the maximum value to something not too far from the median value of the picture. Then gamma and contrast to taste :slight_smile:

Note that the same parameters calculated for the ColorPlus 200, also work fine with the Kodak Gold 200 used in the tutorial, and with a different camera used to perform the “scanning”.

That’s all. Hope this is useful to somebody
Sorry for the long post, and my terrible english :smiley:

alberto

7 Likes
#2

No, it is well written, both the English and the content.

Yes, that is what I typically do. It makes a lot more sense energy-wise anyway. Other equations have a similar form; e.g.,

image

Go for it. The beauty of open source is that anyone can contribute.

1 Like
(Flössie) #3

I can’t speek for all “the RT developers”, but I’m personally interested in such a feature (and was talking about the color film problem with @heckflosse lately). I can offer you my help for getting this into RT.

Definitely. I don’t think DIY film scanning is a niche, there are so many negatives lying dormant and waiting to be recovered. A ready-to-use solution in RT would bring that to the masses. :grin:

While we’re all fans of the UNIX philosophy this wouldn’t help John Doe (who isn’t aware of something like the command line).

Hmm. :wink:

(Shush! Look at this. Maybe you could ask the one with the color target to lend it to you.)

Great! :+1:

That’s a lot. Now let’s make it useful for everybody. Fork RT on GitHub and make a pull request. When you need a helping hand (or two), don’t hesitate to ask here, in a PM, or in the PR at GitHub. Really looking forward to it.

Best,
Flössie

2 Likes
#4

Cool! Thank you for the replies. Ok, I’ll try to do it. I don’t have much spare time so I can’t really commit to a deadline, but I’ll try to get this done.
Stay tuned :wink:

6 Likes
#5

Here is an improvement on the G’MIC pipeline. I added a rudimentary form of auto WB, because you have to manually white balance the resulting image anyway.
Basically i’m reading the median for each individuall channel, and using them as coefficients to keep the median values roughly aligned. This way the resulting image will be almost balanced, and yow only have to make slight adjustments by hand.
This way, there’s no need to have the coefficients as parameters, so a film type will be characterized by just 2 exponents.

film_neg:
  # Calculate formula with Kodak ColorPlus 200 parameters
  -fill [(R^-1.57969792927765),(G^-1.15851358827476),B^-1]
  # Split the image in separate channels and read channel medians
  --split c rk={ic#1} gk={ic#2} bk={ic#3}
  # Discard single channel images
  -rm[1,2,3]
  # Divide each channel by its median, take them all roughly in the same ballpark
  -fill[0] [(R/$rk),(G/$gk),(B/$bk)]
  # Clip away outliers; threshold arbitrarily chosen as 6 times the global median
  -cut 0,{ic*6}
  # Gamma and contrast to taste
  -apply_gamma 2.2
  -adjust_colors -10,10
  # Normalize to 16bit range in case you want to save the result
  -normalize 0,65535

This is more or less what i plan to implement inside RawTherapee. Turns out G’MIC is a perfect tool for prototyping!

I’ve tried digitizing some very old Kodak negatives from 1969; the color cast is different from more modern film. Using the exponents calculated for the ColorPlus 200, the result is quite bad, and of course i can’t get a colorchecker shot on a 1969 film because i don’t have a time machine :smiley:
BUT… i noticed you can still get decent results by using the beginning of the film as white/black patches to calculate the coefficients. You know, the part of film that sticks out of the roll when you buy it. That part has been outside and is overexposed. The part inside the roll, before the first frame, is completely blank. For example, using this (they cut the dark part very short in the lab, and put a sticker on it) :

start

…yields this:

neg2

Far from perfect of course, but not too bad for a starting point.

2 Likes