Problems with Fuji XT20 AF noise square fix

Alex180500 · August 25, 2020, 5:23pm

I’ve seen in this post here problems regarding bias and darks with Fuji cameras: Handling Fujifilm X-Trans Auto Focus Pixels - #13 by lock042

(all photos are with Siril 0.99.4 and just one histogram auto transformation in jpeg)

In my last night I photographed the milky way and noted the same thing with darks:
This is the stack of 15 darks 800 iso 10s exposure at about 15°C ambient

The square is subtle but is here

The square in Bias files is much dimmer:
Stack of 15 bias at 800 iso

When I try though preprocess the darks with the bias and then stack I get a much stronger AF square.
This is with the Fix X-Trans AF artifact tick on:

And this is with the tick off:

Basically they are the same and the problem seems even stronger, it doesn’t work or I’m doing something wrong?

lock042 · August 25, 2020, 5:32pm

Which DSLR?
Please, send me the log

Alex180500 · August 25, 2020, 5:42pm

The DSLR is a Fuji XT-20 like in the title.

Reading sequence file `Darks_.seq'.
free_image_data() called, clearing loaded image
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
log: Reading FITS: file Darks_00001.fit, 1 layer(s), 6032x4028 pixels
bitpix for the sequence is set as 20
log: Sequence loaded: Darks_ (1->15)
Purging previously saved reference frame data.
number of filtered-in images: 15
number of filtered-in images: 15
Saved the reference frame buffer for alignment preview.
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
log: Reading FITS: file #master-Bias.fit, 1 layer(s), 6032x4028 pixels
log: Preprocessing...
log: Preprocessing: processing...
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
log: Reading FITS: file Darks_00001.fit, 1 layer(s), 6032x4028 pixels
log: Saving FITS: file pp_Darks_00001.fit, 1 layer(s), 6032x4028 pixels
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
log: Reading FITS: file Darks_00007.fit, 1 layer(s), 6032x4028 pixels
log: Saving FITS: file pp_Darks_00007.fit, 1 layer(s), 6032x4028 pixels
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
log: Reading FITS: file Darks_00009.fit, 1 layer(s), 6032x4028 pixels
log: Saving FITS: file pp_Darks_00009.fit, 1 layer(s), 6032x4028 pixels
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
log: Reading FITS: file Darks_00003.fit, 1 layer(s), 6032x4028 pixels
log: Reading FITS: file Darks_00002.fit, 1 layer(s), 6032x4028 pixels
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
log: Reading FITS: file Darks_00008.fit, 1 layer(s), 6032x4028 pixels
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
log: Reading FITS: file Darks_00012.fit, 1 layer(s), 6032x4028 pixels
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
log: Reading FITS: file Darks_00015.fit, 1 layer(s), 6032x4028 pixels
log: Saving FITS: file pp_Darks_00003.fit, 1 layer(s), 6032x4028 pixels
log: Saving FITS: file pp_Darks_00002.fit, 1 layer(s), 6032x4028 pixels
log: Saving FITS: file pp_Darks_00008.fit, 1 layer(s), 6032x4028 pixels
log: Saving FITS: file pp_Darks_00015.fit, 1 layer(s), 6032x4028 pixels
log: Saving FITS: file pp_Darks_00012.fit, 1 layer(s), 6032x4028 pixels
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
log: Reading FITS: file Darks_00005.fit, 1 layer(s), 6032x4028 pixels
log: Reading FITS: file Darks_00014.fit, 1 layer(s), 6032x4028 pixels
log: Saving FITS: file pp_Darks_00005.fit, 1 layer(s), 6032x4028 pixels
log: Saving FITS: file pp_Darks_00014.fit, 1 layer(s), 6032x4028 pixels
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
log: Reading FITS: file Darks_00013.fit, 1 layer(s), 6032x4028 pixels
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
log: Reading FITS: file Darks_00011.fit, 1 layer(s), 6032x4028 pixels
log: Saving FITS: file pp_Darks_00013.fit, 1 layer(s), 6032x4028 pixels
log: Saving FITS: file pp_Darks_00011.fit, 1 layer(s), 6032x4028 pixels
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
log: Reading FITS: file Darks_00010.fit, 1 layer(s), 6032x4028 pixels
log: Saving FITS: file pp_Darks_00010.fit, 1 layer(s), 6032x4028 pixels
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
log: Reading FITS: file Darks_00004.fit, 1 layer(s), 6032x4028 pixels
log: Reading FITS: file Darks_00006.fit, 1 layer(s), 6032x4028 pixels
log: Saving FITS: file pp_Darks_00004.fit, 1 layer(s), 6032x4028 pixels
log: Saving FITS: file pp_Darks_00006.fit, 1 layer(s), 6032x4028 pixels
log: Sequence processing succeeded.
log: Execution time: 39.21 s.
seqfile 'Bias_.seq' already exists, not overwriting
seqfile 'Darks_.seq' already exists, not overwriting
seqfile 'Lights_.seq' already exists, not overwriting
seqfile 'MonoFlats_.seq' already exists, not overwriting
seqfile 'pp2_Darks_.seq' already exists, not overwriting
Writing sequence file pp_Darks_.seq
Sequence found: pp_Darks_ 1->15
seqfile 'pp_MonoFlats_.seq' already exists, not overwriting
Reading sequence file `Bias_.seq'.
Reading sequence file `Darks_.seq'.
Reading sequence file `Lights_.seq'.
Reading sequence file `MonoFlats_.seq'.
Reading sequence file `pp2_Darks_.seq'.
Reading sequence file `pp_Darks_.seq'.
Reading sequence file `pp_MonoFlats_.seq'.
Loaded 7 sequences
Reading sequence file `pp_Darks_.seq'.
free_image_data() called, clearing loaded image
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
log: Reading FITS: file pp_Darks_00001.fit, 1 layer(s), 6032x4028 pixels
bitpix for the sequence is set as -32
log: Sequence loaded: pp_Darks_ (1->15)
MODE: closing sequence
log: Closing sequence Darks_
Writing sequence file Darks_.seq
Purging previously saved reference frame data.
number of filtered-in images: 15
number of filtered-in images: 15
Saved the reference frame buffer for alignment preview.
log: Stacking will use registration data of layer 0 if some exist.
log: Stacking: processing...
log: Processing all images in the sequence (15)
log: Stacking result will be stored as a 32-bit image
log: Using 5120 MB memory maximum for stacking
Read from FITS header: pix size 3.91x3.91, binning 1x1, focal 18
image size: 6032x4028, 1 layers
cfitsio was compiled with multi-thread support, stacking will be executed by several cores
log: We have 24 parallel blocks of size 167 (+20) for stacking.
Block 0: channel 0, from 0 to 167 (h = 168)
Block 1: channel 0, from 168 to 335 (h = 168)
Block 2: channel 0, from 336 to 503 (h = 168)
Block 3: channel 0, from 504 to 671 (h = 168)
Block 4: channel 0, from 672 to 839 (h = 168)
Block 5: channel 0, from 840 to 1007 (h = 168)
Block 6: channel 0, from 1008 to 1175 (h = 168)
Block 7: channel 0, from 1176 to 1343 (h = 168)
Block 8: channel 0, from 1344 to 1511 (h = 168)
Block 9: channel 0, from 1512 to 1679 (h = 168)
Block 10: channel 0, from 1680 to 1847 (h = 168)
Block 11: channel 0, from 1848 to 2015 (h = 168)
Block 12: channel 0, from 2016 to 2183 (h = 168)
Block 13: channel 0, from 2184 to 2351 (h = 168)
Block 14: channel 0, from 2352 to 2519 (h = 168)
Block 15: channel 0, from 2520 to 2687 (h = 168)
Block 16: channel 0, from 2688 to 2855 (h = 168)
Block 17: channel 0, from 2856 to 3023 (h = 168)
Block 18: channel 0, from 3024 to 3191 (h = 168)
Block 19: channel 0, from 3192 to 3359 (h = 168)
Block 20: channel 0, from 3360 to 3526 (h = 167)
Block 21: channel 0, from 3527 to 3693 (h = 167)
Block 22: channel 0, from 3694 to 3860 (h = 167)
Block 23: channel 0, from 3861 to 4027 (h = 167)
allocating data for 12 threads (each 57 MB)
log: Starting stacking...
log: Pixel rejection in channel #0: 0.078% - 1.039%
free and close (0)
log: Rejection stacking complete. 15 images have been stacked.
Ending stacking idle function, retval=0
log: Integration of 15 images:
log: Pixel combination ......... average
log: Normalization ............. none
log: Pixel rejection ........... Winsorized sigma clipping
log: Rejection parameters ...... low=4.000 high=3.000
log: Background noise value (channel: #0): 5.223 (7.970e-05)
log: Saving FITS: file pp_Darks_stacked.fit, 1 layer(s), 6032x4028 pixels
number of filtered-in images: 15
number of filtered-in images: 15
log: Execution time: 2.41 s.

I’ve seen that in the preferences, Pre-processing there is a tab to insert AF pixels and it is empty, how can I locate mine and add them there?

lock042 · August 25, 2020, 5:48pm

If you load a master dark, with the square, and you type in siril console “fix_xtrans” what’s going on?

Ping @setnes

Alex180500 · August 25, 2020, 5:55pm

Seem to work, this is only the darks stacked:

Before:

After:

And this is for the darks preprocessed with masterBias:

Before:

After:

The log is pretty basic:

log: Reading FITS: file pp2_Darks_stacked.fit, 1 layer(s), 6032x4028 pixels
Loading image OK, now displaying
log: Running command: fix_xtrans
1598377780: running command fix_xtrans

lock042 · August 25, 2020, 5:58pm

Hum so the fix xtrans button should also work.
When did you check it?

Alex180500 · August 25, 2020, 6:02pm

I checked the button when preprocessing the darks sequence with biases.

lock042 · August 25, 2020, 6:08pm

Why do you do that? Dark optimization??

You need to check it when
1 - preprocessing the flats with master bias
2 - preprocessing lights with master darks

lock042 · August 25, 2020, 6:11pm

I’ve edited my post

setnes · August 25, 2020, 6:24pm

This actually makes some sense. We were always testing with a flow similar to what’s in the included DSLR scripts (now renamed). The biases are corrected when applied to the flats, and the darks are corrected when applied to the lights. When you use other processing flows that use darks corrected with biases, or flats corrected with darks, etc. It will take some thought to know when to apply the correction.

What’s not in the original post is a conversation @lock042 and I had about correcting light frames. Most of the time this shouldn’t be necessary, but I have noticed a situation using an OIII filter where the color cast caused the artifact to be visible. So… that caused me to rework the logic to only compute the correction based on green pixels. This will allow the logic to be used against light frames before demosaicing… but we haven’t really told anybody about that yet… as it shouldn’t be the norm.

I’ve said too much.

Alex180500 · August 25, 2020, 7:01pm

Thanks you all for the replies!!
Yeah I mostly do Dark optimization in my stacks.

I tested the Flats with the Fix xtrans button checked:

And without:

both using multiplicative average with rejection (winsorized sigma clipping)

I cannot find the difference though.
I took one light frame and preprocessed it with the normal master dark:
With button checked:

Without:

No differences here too, I really don’t know if the button is working or not at this point so I’ll just use the fix_xtrans command on the masters instead.

lock042 · August 25, 2020, 7:06pm

The button use the same function. So it should work.