Mean/Median Stack of multiple dark frames before demosaicing.

Software RawTherapee
12

Looks like I found a solution.

Kandao Raw+

I tried it, and it legit spits out an undemosaiced noise reduced raw stack file that I was able to successfully use for removing amp glow in the BMPCC without adding more random noise.
Success!

13

Warning: I’m 90% certain Kandao’s RAW+ does a tile-based align-and-merge similar to Google’s HDR+ algorithm, which may cause unusual artifacts.

You want a straight mean/median stacking approach that does NOT do any sort of alignment - siril can do this. (In your use case, you’d want to treat the dark frames as being normal frames, since you want to feed the preprocessed darkframe to RT). However since it’s highly astronomy-focused it spits out a FITS file as output, which you then need to convert to DNG. You can use imagemagick’s convert command to create a 16-bit linear TIFF, rename to DNG, and use exiftool to add the appropriate metadata. I posted an example exiftool command in another thread here a few months ago, I’ll try and dig up the thread.

2 Likes
14

“You want a straight mean/median stacking approach that does NOT do any sort of alignment”

Well, I was aware that that was the ideal.

I tried out Kandao Raw+, knowing full well it might have issues because of the align-merge function, and ran some experiments, and the file it spit out did work to remove the amp glow and some of the hot pixels, so I kind of guess it defaults to no alignment when fed dark frames.

Nonetheless this is the ideal solution that I have been looking for for YEARS! Thank you so much!

“siril can do this. (In your use case, you’d want to treat the dark frames as being normal frames, since you want to feed the preprocessed darkframe to RT). However since it’s highly astronomy-focused it spits out a FITS file as output, which you then need to convert to DNG. You can use imagemagick’s convert command to create a 16-bit linear TIFF, rename to DNG, and use exiftool to add the appropriate metadata. I posted an example exiftool command in another thread here a few months ago, I’ll try and dig up the thread.”

1 Like
15

Still need to dig up my tagging approach, but GitHub - ifb/makeDNG: A utility for converting mosaiced/Bayer TIFF files to Adobe DNG might be slightly less painful if you change the parameters to match your camera as far as converting from FITS back to DNG after processing in siril

1 Like
16

Thanks, I did successfully use Imagemagick to convert the fits file into a DNG, but couldn’t figure out how to even get exiftool running and doing commands on metadata, using Windows.

On further inspection, I realized that the output of Kandao Raw+ successfully removes amp current, but the complex AI stacking algorithm messes with the hot pixels.

I wil also give Makedng a go.

17

BTW, I did find the exiftool based flow I use often - Loading .raw images (from industrial cameras) - #7 by Entropy512

Note that your ColorMatrix will definitely be different (although for darkframe stacking it may likely not matter as I would assume RT will ignore the ColorMatrix metadata if it’s being used as a dark frame)

1 Like
19

I noticed that in your exiftool script the black level had four numerical entries presumingly for each channel of the bayer matrix, but in the metadata for a real BMPCC raw file, there was only one entry for the black levels. Which should I go by?

20

Ok, so I went with the four value black level, inputting the value found in the metadata for the real camera raw file. When I did the dark frame subtraction, I noticed that the result was clipped black until I applied negative raw blackpoint compensation in Rawtherapee at a value of -500. However, after application, it looks like the hot pixels and amp current were fixed.

Is this behavior an inevitable result of subtracting a darkframe with a raised noise floor, or does this mean I need to fiddle with the black point setting in Exiftool a bit more? The fact that this blackpoint compensation is needed when subtracting another camera original raw file suggests that it is inherent to the dark frame subtraction, and the metadata settings I inserted were correct, and changing the black point in Rawtherapee after the subtraction is the correct workflow. What are your thoughts?

21

Hmm, I’m not sure on this one, as I’ve never used the darkframe subtraction before.

There might be a corner case/bug that occurs if a non-DNG raw file is used with a DNG darkframe, not sure. What happens if you use a single original raw file?

1 Like
22

Ok, so I noticed the same issue mentioned with black levels when I used an original raw file, and the issue is present but hot pixels get fixed regardless of if the subtraction file is the ‘real’ one or the averaged ‘cludged raw’ file, so my intuition tells me that it is an inevitable quirk of the darkframe subtraction with the BMPCC, based on elevated effective black levels. If this is true, then I should just continue to add the negative black point compensation in the raw tab after enabling subtraction. But the other less plausible but still possible option is that the black point in metadata manually inserted for the subtraction frame is not correct and I should continue fiddling with it. I was just curious if you had any opinions any one way or the other.

Nonetheless, I got the pseudoraw darkframe effectively working, in large part thanks to your much appreciated guidance, and question posed above is more theoretical minutia about the most optimal way of doing things than technical obstacle of necessary surmountation.

23

Hmm… There might be a bug in RT regarding black level handling when DFS is performed.

24

This looks suspicious:

If someone can provide example files from a camera with a black level > 0 I can check what’s wrong…

Edit: iirc at this stage of RT pipeline the data is still before black level subtractions, means each file is something like BL + data. If we now subtract image1 (the darkframe) from image 2 we would get image2 - image1 (means after subtracting black level). For this reason the blacklevel is added to the result in the code I posted above).

25

For examples you’ll need to talk to @Waveluke, although at least reading that - that should be re-adding the black offset so it remains after subtraction.

Unless W/H are mismatching for some reason, in which case the black level correction wouldn’t be applied.

26

But then the darkframe subtraction also would not be applied.

27

ERROR, insufficient caffeine at this time.

Good call.

28

My latest PlayRaw NEF is from a camera with metadata black > 0, Nikon Z6=1008:
https://glenn.pulpitrock.net/Individual_Pictures/DSZ_4168.NEF

29

I also need a dark frame…

30

Care about shutter speed, ISO, etc?

31

shutter speed and ISO should be the same as the other image. Lens and aperture should not matter in this case.

32

Here’s one, 1/3sec, ISO 400, same as DSZ_4168.NEF:
Lens cap on, dark room, camera wrapped in a black towel to avoid leakage…

https://glenn.pulpitrock.net/Individual_Pictures/DSZ_4437.NEF
This file is licensed: Creative Commons, By-Attribution, Share-Alike.

I call it “Black Cat Eating Licorice in a Coal Mine”. As a PlayRaw, I’d be interested to see what folk can do with it… :smile:

3 Likes