Dust spots in flat fields

Hi.
I know this is a long shot but I thought I’d ask anyway.
Obviously, when capturing the actual flat fields, the sensor should be clean.
In the real world, the sensor will never be 100% clean, unless it’s an out-of-the-box camera.
Some dust particles just won’t go away, while others are not even dust but abrasions that are there to stay… Well, there shouldn’t be any kind of problem with those types of “permanent residents” in the flat fields, at least until we try to use them on OLD photos, photos that were captured BEFORE some sort of acid etched itself on the OLPF.
If we capture the flat fields now and apply them to OLD photos, there will be all sorts of bright spots all over the frame, because the NEW flat fields have in them dust/scratches/acid droplets that were not present on the sensor when the OLD photos were taken.
Of course, there’s the “Blur radius” but it can do only so much.

The most sensible way to approach dirty flat fields would be to actually “clean them up”. If there was a “spot removal” tool like the one in Lightroom and we could use it on flat fields, that would be “problem solved”. But since RT does not allow any local adjustments, I wonder if there’s another way to clean up the mess.
Isn’t there ANY software that can modify the actual pixel data in RAW files (NEF). I’m aware that those aren’t even pixels but you know what I mean.

I’m open to any other suggestions too.

Thanx!

2 Likes

I’m not sure it’s what you’re after, but Darktable has a spot removal module. As per all of its modules, it’s non-destructive. I think you may even be able to set a preset with it, for if the spots are consistenly located.

You’re not sure what I’m after because you’re thinking of a different software. I was talking about RawTherapee. So, unless Darktable offers vignetting correction via flat-field application, it is of no use to me.
Thanks anyway.

Hey David

I don’t know of any programs which let you paint/clone/heal the undemosaiced raw data and then save it still as undemosaiced raw.

Why is the blur radius not enough? Works fine for me.

I realise Darktable is a separate program to RawTherapee, but wondered if you might want to try taking the demosaiced/processed result from RT into it to try its spot removal. DT offers lens profile-based vignetting correction, but not flat-field-based as far as I know.

Because you lose accuracy.
There are spots in my flats that can only be mitigated with blur radius values of around 85. I had to run a few tests to figure out just how bad values like that affect the end result. As it turns out - not that bad but it’s not like having the blur radius at 3 or even 30. There IS noticeable difference and when it comes to stitching panoramas ANY noticeable vignetting is bad.

RT is great at removing vignetting, and I do mean that. However, it needs some basic spot removal tool badly. Both, to be used in the photos and on the flat fields. Being able to actually clean up the dust in the flats would be so much better than having to bump up the blur radius, which is nice that it’s even there, with the values allowed to go up so high.

So whoever is developing RT: Spot removal tool ASAP, please. It should be there, even if it’s the only “local adjustment” tool. Frankly, in my workflow that is the only tool that I miss in RT.

Lens profile-based vignetting correction and flat fields are not in the same league. As far as I’m concerned, they’re not even the same game.
Thanks anyway, man.

There isn’t any software for dust removal and saving this in RAW, but you have some luck as I see you are using a Nikon. The software of Nikon Capture NX2 can remove dust spots and save the modified photo (flat field) in NEF format.

Hope this helps.

Can you please expand on that? Because I’ve tried taking that route but all I got was NEF files with embedded instructions in them, with no change to the actual RAW image data whatsoever.
From what I understand, the Nikon software simply saves the “edit” instructions inside the NEF files (as opposed to storing them in separate, sidecar files). So what happens is, as long as I view or edit the NEFs inside Nikon software, the “edits” are applied and I see the photos with the spots removed.
However, these instructions are simply ignored by other applications, including RT. So whichever edits I apply in ViewNX or CaptureNX are simply ignored. So yes, all the dust is still there, even if I’ve “removed” it in Nikon software.
I’m not an expert on the subject and for once I’d love to be wrong about something. Please tell me that I am and that it can be done.

Thank you for reading through the entire thread and understanding the issue I am dealing with.

At least Canon’s software (DPP) works just like you described, dust removal data are saved in metadata and usable only with (the same version of) Canon’s program.

Edit: Same thing with Lightroom (5.7) and exported DNG.

[quote=“the-old-david, post:7, topic:924”]
Lens profile-based vignetting correction and flat fields are not in the same league. As far as I’m concerned, they’re not even the same game.Thanks anyway, man.
[/quote]Out of curiosity: What exactly are you missing? I found them to work quite well if measured correctly (and taking into account focal distance).

Hello,
I am also interested by that kind of feature in RT.
When I have to deal with dust spots, I have no other solution than editing the exported picture in Gimp with duplicate & corrector tools. Such a time-expensive work.

Beside that it would be a nice addition to have something like a “clone-brush” in RT which Ialso would love to have.

@the-old-david

From what I’ve learned so far there is no RAW editing software, which does manipulate the RAW itself. So everything you do inside any of those RAW editing software is always stored as “settings” for the program itself. So as long as you don’t prozess a RAW to tif, jpg, what ever you only would see the things you’ve done inside the RAW editing software. Those programs differ in the way they do things and therefore they store different “settings” and also the way they store those are different, different formats as pp3, xmp, dop and other and different places and ways as side car files, inside a database or even inside the xmp section of the image itself.

Well even - at least from my point of view - if you open a RAW file in which ever RAW editing software, you don’t look at the real sensor pattern, you already look at some prozessing this pattern into a more viewable image. Well at least I havn’t seen any RAW editing software displaying i.e. the Bayer pattern (scroll down to the first third of this article about zero noise, there is a pic of what I mean).

So my answer to your question would be NO, I don’t know any software which would manipulate the sensor pattern.

darktable has a pass through demosaicing mode. It is originally meant for cameras with the CFA removed, but of course you can enable it on other images, too.

Thanks @houz good to know - as a win user I havent invested much so far in darktable but I should do from what I’ve seen so far.

@Hombre was working on a spot removal tool, but quit his involvement in RawTherapee to pursue other interests before it was completed.
https://github.com/Beep6581/RawTherapee/tree/spot-removal-tool

RawTherapee can skip demosaicing to show the pattern:
http://rawpedia.rawtherapee.com/Demosaicing

Thanks for this tip @Morgan_Hardwood