Rescuing thousands of images from a blemished life

The Story

Over the last few years, my wonderful and tireless mother has scanned thousands and thousands of negatives (approx 7,500). Family, holidays, friends – all these images have been digitised for future generations after an incredible amount of work.

I saw her first twenty or so rolls in the early days. Magnificent results. Clean, and well organised, she was on a very good path. I’ve heard her stories over the years of some of the old negatives she’s found and seen one or two of them. Yes, they were grainy, but everyone expected that, right?

Anyway, she has been beavering away over a period of seven or so years, scanning all these photos.

I happened to have a really good look at some of the later ones (that are more modern photos) about a week ago.

Maybe you can imagine the sick feeling that came over me when I realised that the scanner seems to never have been cleaned.

This work that she’s so justifiably proud of, for the sheer effort, has every photo marred with dust and fibre artefacts. And quite badly. It’s heartbreaking. Especially when she didn’t really see these issues as things that (a) could be fixed or (b) needed fixing in many cases.

It’s clear that these images are going to live on for years – they are not going to be re-scanned.

Here’s three photos, to show what’s happened. They are relatively new films - and scan very well when the scanner is clean. I have circled some common blemishes between the three images that are obviously from the same unmoved fibre stuck on the scanner; sadly the dust specks are also perfectly evident.



I have googled, played with digikam / g’mic / imagemagick / gimp, looked at filters and tools and lost sleep, and am still feeling sick about the relatively poor results for the massive effort that was put in to scan these images. And that could have so easily been avoided.

The challenge.

I don’t want to make these images perfect. What I do want to do is remove some of the bigger artefacts automatically if at all possible.

Automatically inpainting holes seems to work really well on most of the specks, so that is sorted.

To be rid of some of the fibres, I think I could use one of the InPaint filters, but am not sure how I could go about this. While the artifacts are very similar in each photo, they move around ever so slightly (I imagine because of image deskew when saving).

Having outlined the probably impossible challenge, does anyone have any idea what they’d do in this situation? To throw the images out is just not possible, for the thousands of person hours that have gone into them.

I imagine the only way this could reliably be done in some batch scenario is to somehow analyse a series of shots made in a given couple of months, try to find some commonality in the artefacts, and somehow and use it as a mask for other InPaint filters, but surely I’m wishing for a miracle here. And if that was an approach, I’d have no idea how to go about it.

Of course, one of the big problems is that the orientation and flipping has been changed on most of the shots to one of 8 configurations, as you can see above. The shots can be in each of the four orientations, multiplied by two because of possible image flipping. And we only have jpg images. No corresponding IR images come with them, probably for obvious reasons.

Another approach would be just to use another kind of “hole” filter that could work on “elongated” holes to cater for the fibres, but again, I’m wishing for a miracle I presume.

Something tells me that I’m not the only person who has wished for this. And with much home scanning going on, I imagine there will be a strong desire for some kind of solution for this in the future.

I’m keen to hear anyone’s thoughts that aren’t “throw the images away and start again.” :cry:

There’s an old tool called ‘Polaroid Dust & Scratch Removal’ that seems to make a good job
https://www.dpreview.com/forums/thread/3836494
https://www.kenrockwell.com/tech/photoshop/polaroid-dust-filter.htm

1 Like

Perhaps you’ll want to peruse this thread: Scanned image scratch removal with “ICE”

1 Like

Thanks for this. I did see this link earlier in my googling, but it is soooo far above my knowledge (so many new concepts in there) that I’m not sure I understand how I can use it. I am definitely going to look into gmic scripting, but that code looks unfamiliar - still, something that I’m sure the wiki pages will help with. It’ll be an interesting learning curve, if nothing else.

There is already some inpaint code in the thread. YMMV, but have at it :slight_smile:

1 Like

Hi,

I sincerely apologize for this - I don’t want to take up anyone’s time, so please just shoot me to another page rather than explaining, if you see what’s wrong here.

If i take this picture (terrible though it is):

…and run it through:
gmic {image name} +_fx_remove_scratches 72,2,4,3

I get no discernable change.

I then opened gimp to see if I could see what the output after the first threshold step was, and it doesn’t matter what parameters I input, I can’t get any “flecks” in the Threshold output, so I suspect that’s why it’s not working.

Can you tell me why that would be? I’ve looked here https://sourceforge.net/p/gmic/wiki/Tips%20and%20curiosities/#6-the-threshold-command and seen that if there is no normalise command, you get a blank screen.

When I look in the gmic config for the “routine” I don’t see a normalise statement after the threshold statement. Something tells me though, that because @David_Tschumperle wrote it, it’s probably not needed. :slight_smile:

You must cringe at newb comments like this, but I’ve done a few hours of googling trying to find the answer and I’m none-the-wiser. :frowning: Sorry… PEBKAC, I s’pose. :slight_smile:

Thanks again.

From memory, the scan scratch removal was based on having an intact IR channel (which basically records scratches) in the image. I assume you don’t have that here.

In fact this seems like a worst possible case; no IR channel, no original images and the scratches are rotated and translated at random due to inconsistent placement in the scanner. I’m not certain there’s an easy way to handle it!

Possibly the first task is to find a good way to create a separate image/layer of the scratches. Given that, the above removal methods could be used…

Some “hot pixel” inpaint might cover some of it.

The OP at dpreview link given by @pphoto suggests the free polaroid dust and scratch removal tool does a similar job to photoshop’s dust and scratches filter. I don’t have the polaroid tool, but can give an example of what it might look like using the dust and scratches tool in photoshop (hope this isn’t against forum policy, I know ps is not FOSS, but the polaroid tool is). This was a pretty fast and crude job, to give an example:

1 Like

@Jeff Please post your examples without the circles. That way we can try removing the artifacts. Edit Preferably as high quality as possible. E.g., you photo in post #6 is super compressed, so much so that I can see tiny square compression artifacts when I zoom in. Those really interfere with image enhancement. However, it may be what you have on hand, then that is all you can provide.

Hi @afre,

Thanks heaps for helping.

Unfortunately, that’s close to the highest quality - when i resaved with the circles and privacy for peoples faces, quality will have only reduced a small amount i suspect. I’ve chosen three images that have those blemishes and have uploaded them untampered.


I really appreciate your help. Very much. I’d love to see if this works with G’MIC - I don’t have easy access to a windows computer (all Linux here) so i imagine getting the polaroid stuff to work with Wine might be problematic. Anyhow, let’s see how it goes. :slight_smile:

Cheers,
Jeff

Maybe it would be a good idea to live with the images as they are for some time. AI in image processing is making great progress. The task could be relatively well done today manually but getting good results by automatization is still challenging. Maybe in 5 years the rescuing “can be done in passing”.

As it looks to me the dust is always at the same place wouldn’t it be possible to overly all images and then generate a mast out of it. If you have the mask you could use gmics inpainting to fix it.

Yes, @Tobias’s suggestion is most sensible. If you can mask reoccurring blemishes, it can go a long way to remove them. Workflow:

1 Mask or select blemishes.
2 Use heal brush, clone brush or in-paint all images. Try selecting all masked areas and then filling them in with large radius brushes or in-painting.


PS No matter how I automate the masking, it always covers areas that are perfectly fine.


PPS @Jeff I made 2 filters for you to try.

G’MIC plugin name (CLI name)
– Clean Photo 0 (afre_cleanphoto0)
– Clean Photo 1 (afre_cleanphoto1)

They are untested and incomplete but I decided to commit them anyway. Their performance depends on the image. Sometimes one works better than the other and vice versa. Remarks:
– “Recovery” → lower value means more changes; higher value means less.
– “Clean Photo 1” → preview can fail on very low recovery values. All you can do is output to new layer to compare and undo if you don’t like the result (at least that works in GIMP).

Hi again,

As always, thanks for the help.

As you’ll probably see if you flick the images in a sequence, those dust marks move, due to minor readjustments that the scanner software has made when it saves the image.

I do think that Thomas’s thoughts might be where I’m leaning, however I’m not sure that this particular issue will be fixed with AI in image processing. I can vacillate though on this. :slight_smile:

I was toying with the idea of working out whether some routine could be written to analyse images to find “runs” of white-ish pixels that are, say, never get narrower than “n” pixels, never get wider than “w” pixels and that are between “x” and “y” pixels in length. Or is that waaaaaaaaay too processor intensive or too hard to figure out?

Cheers,
Jeff

Wow @afre, just seen your PPS.

I’ll give them a try tonight! :slight_smile:

So brilliant - thanks.

Hi @afre,

I updated the filters, but didn’t see your plugins sadly.

Where did you commit them / how do I get access to them?

Thanks again. (Please feel free to just shoot me a link if i’ve missed something obvious.)

Cheers,
Jeff

Ok, i updated on the command line and have them.

So, I’m trying commandlines out now. Thanks, I’ll have a bit more of a play. :slight_smile:

Cheers,
Jeff

Sometimes the plugin doesn’t update properly. Don’t expect too much from the filters though. They aren’t very smart and only meant to give you perspective.

1 Like

Thanks again @afre.

Just quickly, and I’ve been looking for about an hour on this one - what would I append to my gmic command to output a file with the same basename, but a png version. From what I’ve been reading, I feel it has something to do with -output png:{[image],b}, but I’m not sure how to write it…

Cheers,
Jeff

Try outputx.