In the next few months I plan to digitize a large collection of old family pictures and I am quite convinced that the best way to go is camera scanning, i.e. using a DSLR to take pictures of the negatives. It is a lot faster than a flatbed scanner and I’m hopeful that the quality will be a lot better once I figured out the workflow. I hope to post something about my setup at some point, but for now I would like to ask for help about the processing of the negatives using darktable.
Basically, I can divide it down to three problems:
The first really weird thing is that it makes a big difference if I use darktable’s “invert” module on the raw file or if I create an intermediate tiff export and invert the tiff. When inverting the raw directly, the colors are really dull and I have been unsuccessful so far to understand what is going on. I suspected it’s due to the fixed module order, but even when disabling base curve and color balance, going via the tiff export still results in a noticeable different image. I also noticed there are different code paths for mosaiced and non-mosaiced pictures, but I don’t have the experience to understand what’s going on and if it’s a bug or supposed to be like that. I would really like to find a workflow that does not need an intermediate export.
Even with the intermediate tiff export, the colors are far away from what the automatic exposure mode of the flatbed scanner is producing. Showfoto seems to do a bit better there (see examples below). In order to process thousands of pictures, I need to find a (semi-)automatic way to produce a result that is comparable to the scanner.
Besides finding a workflow that produces acceptable results (semi-)automatically, I would be really happy about some advice of how to process the pictures to make them look great. I lack experience and so far have not been happy with the results I got.
Find two example pictures with the raw files below. Feel free to edit them and share the results. More RAW samples from somebody else can be found here. I’ve had the same issues when trying to invert those with darktable.
Of course I have and I have also found this thread and tried to understand it. It might be my lack of experience with photography and color spaces etc, but I do not see how the discussion there will help me with the questions I asked. Most of it is not about darktable and it also talks about inverting raw data from flatbed scanners, not scanning using a DSLR.
the problem you are facing is mainly the orange mask. I do not know if you need to start with the raw images or if a TIF image from the camera would be sufficient.
Just to show what is needed I opened your house image with RawTherapee and exported it as a TIF. Then I used a dedicated software for scanning (SilverFast HDR) and processed the negative image there. Since I do not know the emulsion I used the default, so colours are not optimized. SF HDR does the inversion and the filtering of the orange mask (which is essentially a white balance). Here is the result:
Then I did the whole thing manually with imageJ starting again from the TIF-output from RT to show the steps needed. First I inverted the image. Then I used the histogram adjustments to set the black- and white point for each channel. This is sort of a white balance and already gives acceptable results. The output from imageJ was then treated in RT for a better white balance and an adjustment of contrast and brightness. Here is the result:
In principle one would also want to include an emulsion-specific colour profile (see above) here also.
I hope this helps to show the steps necessary to develop a dedicated workflow with darktable or some other tools.
There is one thing I would like to point out, because it is important for my workflow. Using a digital camera is fast, but it lacks the tool of IR cleaning from dust and scratches.
thanks a lot for taking time to look at my samples and doing tests. I really appreciate it!
No, of course not. I want to use the solution that is best in terms of quality, efficiency and workflow. But I plan to do further editing of some of the pictutures in darktable, therefore using darktable (and without intermediate exports) will have the advantage that I will have to only have to store the .cr2 and the .xmp file.
Can you tell me what tools you used and how you processed the image?
I know and as far as I understood it, removing the orange mask when inverting is exactly what the “invert” module is for. That is why I am so surprised about the results I got.
Wow, the result is quite impressive! I would be really interested to reproduce the steps. I am really happy to have the proof that it is possible to turn my raw file into something that is much better than what the scanner did automatically. I hope it will also be possible to find a reproducible non-destructive workflow that will be almost as good as what the scanner produces and can just be applied to a few thousand images without too much manual tuning.
That is true, but that doubles the scanning time and even scanning without ICE would be a tremendous effort for the number of negatives I plan to scan. That is why I am looking for a solution that gives acceptable results fast and still leaves the option to do fine tuning for individual photos later.
removing the orange mask when inverting is exactly what the “invert” module is for
Well, inverting and filtering are two separate things. But I do not know this “module” you are talking about.
can just be applied to a few thousand images without too much manual tuning
What I did with RT (develop the raw and export as TIF) and imageJ (setting white point and black point) is easily cast into a batch job, albeit with two separate software tools, but that should not matter. Tweaking the colours etc. is the a manual job. From my experience, it is an illusion to handle thousands of images in an automatic way with optimum results.
Here is what I get with no manual intervention for the polar bear image (this time inverted with Adobe raw converter and then using imageJ to do the inversion and histogram setting with the “auto-mode”:
Yes. I have to admit, though, that @Jossie 's solution is much better than mine
However: my suggestion is much faster It is based on RawTherapee – not because rt is better than darktable, it is just that I know how to handle Rawtherapee somewhat better.
In short:
Open image in rt
Select Standard film curve ISO low
Film simulation Negative
White balance on the back of the bear.
Change exposure L to +64 or after taste.
Later this evening. I will try to make a one-command-line of it all. Will be an interesting piece of exercise.
well, same argument holds for me, too: I know imageJ better than RT so it was faster for me that way. So as usual, many roads lead to Rome, as we say in Germany.
I gave the house-image a little bit more treatment with Nik-collections’s Viveza:
Hermann-Josef
@Claes Could you please tell us in which menu the relevant RT commands can be found? E.g. where do I find “standard film curve ISO low”?
You can also start Rawtherapee, open icebear.cr2 (or whatever scanned image you want to convert)
and ask Rawtherapee to use the profile makePositive.pp3.
Interesting question @dani_l. I also used RawTherapee. Just two tone curves in the Exposure module and switching on the local contrast module and white balance adjustment. This is the processing file: Inversion.pp3 (10.9 KB)
And, here are the result when it was applied to the given images (some extra rotations to make it viewer friendly!):
@Claes, @shreedhar: Thanks a lot! I will start to play around with RawTherapee and look at your pp3 files carefully.
Could you please tell me how to do the automatic histogram adjustment with ImageJ? I tried “Image > Adjust > Color Balance” and the automatic mode gives me strange results. Adjusting it manually works, though.
But thanks to your description, I discovered that the automatic level adjustment in GIMP (“Colors > Levels > Auto”) gives quite similar results. I think this is also what the scanner software is doing to get a result that looks 80% correct 80% of the time.
Still have to figure out how to best integrate it in my workflow and how to automate it, but at least I know it is a problem that can be solved.
And that is why it is nice to have pixls.us instead of a discussion forum for a specific software. I’m still hoping to get some answers for my darktable specific questions, though…
Could you please tell me how to do the automatic histogram adjustment with ImageJ?
Here are the three steps I did in imageJ:
Crop the image. This is important as otherwise the unexposed areas influence the histograms.
This shows the histogram after load the image (as an example the R-channel):
Click “Auto” and then “Apply” to get this histogram:
Repeat for all three channels and save as TIF.
That is all.
I also tried “Image – adjust – colour balance” and get the same results.
These two steps can easily be recorded as a macro and then applied e.g. to all images in a folder.
This procedure is to first approximation the white balance algorithm called “grey world”, which assumes that the Center of gravity for the histograms in the three channels are the same (see here).
@Claes I cannot find “negative” under “film simulation”. There is no way to change something under this tab in my RT version (5.4). Instead I used the RGB-curves for inversion and then adjusted the tone curve. White balance I did just after inversion on the grey wall in the background.
Because the thread title originally refers to darktable, here’s my quick try with it.
Tone curve for inverting, white balance manually set to the lateral frame stripes and a second S-shaped tone curve plus exposure correction and mirroring. No further optimization done.
I did not really try to achieve the coloring of the original print but the result looks decent in my eyes.
Without having used ImageJ before, it was totally not clear to me that I have to enable select the different channels and use that individually. After I found “Image > Color > Channels Tool”, I was able to reproduce your result.
If I understood the GIMP auto levels code correctly at a first glance, it searches for the first (and last) point in the histogram that clips at least 0.6 % of the pixels (and is not an empty point in the histogram) and adjust each channel to that range. So it’s not terribly complicated… I also spent a few minutes digging through the ImageJ code, but didn’t immediately find the autoscaling code. I doubt it will be much different.
That is aresome and really helpful. So what actually works is to use the tone curve to invert the image and not the base curve or invert. I gave up too early on the tone curve because I kept “scale chroma” on automatic instead of setting it to manual and inverting the a and b curves like you did. I also found out that “automatic in RGB” also does the trick.
Another question for someone who understands darktable: From what I can tell, the color balance module in darktable can be used to set the black and (indiretly via gain) white points for each RGB channel, so is could be used to apply the RGB histogram adjustment directly in darktable. A) Is this correct? B) Is that the only module or did I miss something? C) Is there an automatic adjustment in darktable, similar to GIMP’s auto levels or the ImageJ auto adjustment?
So at this point, I see 3 options to get to some automated workflow:
Crop the image in darktable (can be done in automatically with a fixed negative holder), export tiff, use GIMP in batch mode to auto-adjust colors. (I’ll post scripts later.)
Edit one picture similar to what @Thomas_Do did in darktable and just use that on all pictures without any auto adjustments. I’ll have to test how the results are with different pictures.
Crop the image in darktable, export tiff, implement my own histogram adjustment and write the calculated parameters for the color balance module back to the xmp sidecar. Sounds like a lot of effort, but it might work out.
