Using symmetries to clean up master bias

Software Siril
1

With my camera (Canon 6D), using the optimal ISO 1600, I estimate the readout noise (7.8 ADU) to be significantly larger than the bias pattern noise (1.3 ADU), which necessitates taking hundreds of bias frames to push the noise down to acceptable levels. This wear off the shutter.

Instead, I looked into using the intrinsic symmetry of the bias patterns to dramatically reduce the noise in the master bias. What I did was:

  1. Created a master bias from 100 bias frame
  2. Loaded the fits file into Gimp editor (it properly handles fits files, keeping the 32 floating point bit depth)
  3. Averaged the initial noisy file vertically
  4. Averaged the initial noisy file horizontally
  5. Combined the two average files into one cleaned master bias, which has all the patterns, but very little noise.

To test the result, I subtracted the initial noisy file from the final cleaned one, and the resulting residual file had no patterns, just the random noise, as expected.

All the images are available in this thread: https://www.cloudynights.com/topic/951314-isogain-for-flats/#entry13915431

I am new to Siril - perhaps this procedure is already implemented? If not, can you please implement it? This can really save time and shutter life as we no longer need lots of bias images.

2

Hello, thank you for your message.
However, in Siril we have a slightly different approach that introduces no noise and assumes that bias is only applied to the flat, an image with a high signal-to-noise ratio. We have introduced the synthetic bias:

3

Yes, I saw those articles. For flats, it makes total sense, I plant to use synthetic bias. But my astrocamera doesn’t really need darks, so good bias is important. Somebody told me that what I described is already implemented in PixInsight (Superbias process). I hope this can be adapted in Siril as well, for people like myself who don’t need darks, just high quality bias.

4

I realized Gimp wasn’t doing the row/column averaging correctly, so I ended up installing AstroPy (which was very straightforward), and writing a short Python script for master bias image cleaning. The results are exactly as I hoped for: the cleaning process completely eliminated the readout noise from the master bias. Now I can use this as a perfect noise-free master bias for my flats. (I actually hope I do not need darks for my camera. If so, I will use this noise-free master bias as a master-dark, and then use flats shot at low ISO with a synthetic bias - see the discussion below.)

Here are the results.

First - the master bias file made in Siril from 100 bias frames (Canon 6d, ISO 1600, 1/4000s exposure); mean=2047.6, std=1.6

new_bias1600
new_bias16002106×1399 2.57 MB

Next: the cleaned version of the above file (processed with my script bias.py); mean=2047.4, std=0.6

new_bias1600_clean
new_bias1600_clean2106×1399 1.47 MB

To test if the cleaned bias is good, here is the difference between the original master bias, and the cleaned one (I made it using Pixel Math in Siril). As expected, all what is left is pure noise, plus whatever non horizontal and non vertical features the master bias image had; mean=0.0, std=1.5

new_bias1600_res
new_bias1600_res2106×1399 2.47 MB

Here is my script. It’s hardcoded to read the file named “bias_stacked.fit” in the same folder as the script. The cleaned file’s name is “bias_clean.fit”. The script has an optional feature to mask out bad (cold and hot) pixels before the cleaning procedure. You just need to change the value of Nsigma parameter to something like 3 (for the 3-sigma masking). I thought it might be useful, but my tests seem to suggest probably not.

Before using the script, you need to install AstroPy. I used the MiniForge method, described here. It was straightforward.

# Program to clean a master bias fits file (e.g. produced by Siril).
# Specifically, it averages the rows and columns of the master bias file, ignoring
# hot and cold pixels (beyond Nsigma from the global mean).
# It uses AstroPy package

 

Nsigma=30000000

import numpy as np
import numpy.ma as ma
import astropy
from astropy.io import fits

# Reading the master bias fits file:
hdu_list=astropy.io.fits.open('bias_stacked.fit')
hdu_list.info()
image_data = hdu_list[0].data
print('Input file:')
print('Min:', np.min(image_data))
print('Max:', np.max(image_data))
print('Mean:', np.mean(image_data))
print('Stdev:', np.std(image_data))

# Mean and std for the whole image:
mean=np.mean(image_data)
std=np.std(image_data)

# Masking out cold and hot pixels:
masked_image=ma.masked_outside(image_data, mean-Nsigma*std, mean+Nsigma*std)
print('Number of bad pixels:', ma.count_masked(masked_image))

# Averaging the columns (skipping masked pixels):
a=image_data.copy()
c=masked_image.mean(axis=0)
a[:,]=c[:]

# Averaging the rows (skipping masked pixels):
b0=image_data.copy()
bt=b0.transpose()
r=masked_image.mean(axis=1)
bt[:,]=r[:]
b=bt.transpose()

# Combining averaged rows and averaged columns images:
both=(a+b)-mean

 

# Writing the result:
outfile = 'bias_clean.fit'
hdu = fits.PrimaryHDU(both)
hdu.writeto(outfile, overwrite=True)
print('')
print('Output file:')
print('Min:', np.min(both))
print('Max:', np.max(both))
print('Mean:', np.mean(both))
print('Stdev:', np.std(both))

I did the same analysis for ISO 200, and it worked again perfectly. But the pattern noise I recovered (0.2 ADU) will be completely drowned by the shot noise for my flats (which I estimate at 55 ADU if averaging over 10 frames). I will be using the Synthetic bias of Siril instead, using the offset I derived (2047.4).