Reference Image becomes "overexposed" when color calibrating while following Comet Tutorial.

Hi,
I have some data from when comet C2022 e3 came by, and I’m trying a second attempt at processing since the first was generating severe star trails. I ran thru the tutorial a few weeks back, and it worked great. It was my data; I didn’t realize the 97 images I had covered 3 hours. Anyway, I’m back at it, and this time once I reopen the sequence before any registration, my reference frame is white. This is after I have color-calibrated it. When I set the display mode back to linear, it’s a very light grey. The sliders below the image are set to 0 and 65535. I am using a slightly customized script as I have a master bias, mast flat, and master dark specific to my exposure duration, 60 seconds.

Is anyone else experiencing this issue? I have tried to step through the comet tutorial several times and still have the same result. Below is the script I’m using to generate per-image background extracted files for the comet tutorial. FYI - lights are in their own folder, and the master calibration frames are in the process folder. I can share images once I figure out how.

Any help is greatly appreciated.

############################################

Script for Siril 1.0

July 2020

(C) Cyril Richard

Preprocessing v1.0

########### PREPROCESSING SCRIPT ###########

Script for color camera preprocessing

Needs 4 sets of RAW images in the working

directory, within 4 directories:

biases/

flats/

darks/

lights/

############################################

requires 0.99.4

#Convert Bias Frames to .fit files
#cd biases
#convert bias -out=…/process
#cd …/process

#Stack Bias Frames to bias_stacked.fit
#stack bias rej 3 3 -nonorm
#cd

#Convert Flat Frames to .fit files
#cd flats
#convert flat -out=…/process
#cd …/process

#Pre-process Flat Frames
#preprocess flat -bias=bias_stacked

#Stack Flat Frames to pp_flat_stacked.fit
#stack pp_flat rej 3 3 -norm=mul
#cd

#Convert Dark Frames to .fit files
#cd darks
#convert dark -out=…/process
#cd …/process

#Stack Dark Frames to dark_stacked.fit
#stack dark rej 3 3 -nonorm
#cd

#Convert Light Frames to .fit files
cd lights
convert light -out=…/process
cd …/process

#Pre-process Light Frames
preprocess light -dark=60_dark_stacked -flat=pp_flat_stacked -cfa -equalize_cfa -debayer

#Extract linear gradient
seqsubsky pp_light 1

#Align lights
register bkg_pp_light

#Stack calibrated lights to result.fit
#stack r_bkg_pp_light rej 3 3 -norm=addscale -output_norm -out=…/result

cd …
close

#######################################################################

All images are show in AutoStretch display mode.

Image prior to color calibration, opened as a single file.

Image post color calibration, opened as a single file.

Image displayed as the reference image in the sequence.

Just to be clear: you color calibrated (with PCC?) a single image within a sequence? When opened, images are opened in linear mode with the saved low and high cut-off values, shown by the sliders below the image.
You may need to switch to min/max, or autostretch.
If it’s still white, you should restart your process in a new directory, with no old .seq file.

Thank you for your reply.

Yes, I used PCC. It was applied to one single image while that one image was the only item loaded. Then upon reselecting the sequence, after the PCC’d single file was saved, it would open as the third image shown above. That image is set to AutoStretch in the display mode and when I choose Linear, it’s a very washed out light grey. It’s as if the PCC is brightening the entire image. The cutoffs are set to 0 and 65535.

I will say I can open the saved file again, by itself, and it does NOT display this way, but when I open the sequence that one frame is completely blown out.

I will try again tonight. But I have already restarted the entire process twice with the same result.

Hi, I experience the same problem. I processed comet ZTF with a previous version of SiriL a while ago, and PCC caused no problems. Then I updated SiriL to 1.2.0 and have exactly the same issue. Before PCC background pixel values are about 0.1%, and after PCC they jump to over 20% now when I try to process another comet. I tried to do the same with comet ZTF which was OK before, and pixel values also jumped to 15%. So it looks like in 1.2.0 the picture is brightened many times.

If there is no quick solution to this, maybe it is possible to have two different versions of Siril installed at once? I’d use an older version just for color calibration then.

Could you share your image you are seeing white please?
I’m not sure there is a bug in PCC, but maybe in the sequence statistics. Try to remove the seq file.

I forgot to mention that after PCC I was trying to register the sequence, and registration process failed on many frames. I removed the seq file, and global star registration process went well. But I forgot to look how the reference frame looked in autostretch mode. It is still much brighter than all the other frames, that was not the case in previously processed comet. I don’t know if this will be a problem or not, I will see.

At least now, after registering, the reference frame looks normally in autostretch, even with 23% background.

Sorry I am probably not very consistent, but now I have to connect to the PC with all the data remotely, and it’s not very convenient. I think I will just redo everything tomorrow, and see how deleting the seq file changes everything.

Hi, as I wrote in the post I have to connect remotely now, so not very good for testing, but just in case I put two frames, one without PCC, and the other with PCC (frame 23):

https://drive.google.com/file/d/1d-snZ43Kt2x9SwRyU3XYlKOuXtzNFVuU/view?usp=share_link

Regards,
Donatas

Take a look to the statistics of your image before PCC.

08:11:17: Running command: stat
08:11:17: Red layer: Mean: 127.9, Median: 124.4, Sigma: 140.1, Min: -11105.3, Max: 37195.2, bgnoise: 29.1
08:11:17: Green layer: Mean: 129.7, Median: 125.3, Sigma: 160.0, Min: -11790.8, Max: 38366.5, bgnoise: 27.5
08:11:17: Blue layer: Mean: 128.2, Median: 124.8, Sigma: 137.9, Min: -11310.5, Max: 37054.2, bgnoise: 28.9

So first you need to understand why and try to fix that. This is clearly not normal.
Second, the issue comes from the statistics cached in the seq file. If you remove the seqfile it will be ok.

I am not very techy, but I assume that it is the negative values that cause the problem. I have no clue were they come from, maybe this has something to do with console warning messages while preprocessing, that GRGB pattern found instead of RGGB, something like that. Removing the seqfile does not solve it as far as I can tell, as after PCC the negative values are shifted to zero, and the reference frame becomes much brighter than all the other ones.

Anyway, I reinstalled version 1.0.6, and PCC went perfectly well, calibrating colors and leaving those negative values. Then I installed 1.2.0 again, because I need Starnet integration.

Even if I don’t find other workaround than switching to older version just for PCC (and it is not very likely I’ll be able to do that on my own), I still like SiriL very much so far :slight_smile:

Below are the stats of my image pre and post PCC. It should be noted that I have stepped through this process three times at this point and all results are the same. I end up with that overexposed white image.

To clarify, when you say remove the seq file, do you mean simply delete it? If so, how do you go about creating its replacement. The only way I have ever created Seq files is using a script. Can I create a replacement Seq file without running a script. Is that process outlined in a tutorial?

Here is Pre-PCC
Image 3-10-23 at 2.43 PM

This is post PCC, prior to reopening the Seq File.
Image 3-10-23 at 2.32 PM

You have awful stats before PCC. So you need to understand why your calibration create somepixels at -65535 !!
You have an issue at this stage.

I know nothing about stats and what is good vs. what is bad. Is there documentation you can recommend so I can educate myself? I see in the example you provide, the stats are far lower. Would you think my stats are worse due to several factors adding up, or could it be a processing error and my data is fine? I am truly ignorant here.

Additionally, I am following the individual background extraction workflow/tutorial. My original data set spanned 3 hours. So in an effort to reduce start trails I have broken the data up into thirds. I was going to run the first 30 images thru the individual background removal process and then the comment stacking process, then rinse and repeat for the other two sets of 30.

Is this a bad idea?

Hello, if calibrated image levels are bad, then it must be that calibration did not happen in the expected way. Make sure the image parameters match between lights and darks and between flats and bias. When some of the images are incorrect, it happens that the calibrated image have more negative pixels than expected, or in your case, a few pixels with very negative values.
Your images looks well corrected by the field otherwise, it’s strange.

To understand what the statistics mean, you can refer to Statistics — Siril 1.2.0 documentation
But the problem probably lies in the calibration frames.

Oh that’s great info. Thank you very much. I will read through the link and reshoot my calibration frames.