RawTherapee Dark-Frame auto-selection picks only 1 file

wincent · December 16, 2020, 10:23pm

Hello all,
First message here, I hope I follow the rules for posting.

I use RawTherapee for demosaicing astrophotos.
I have 5 dark-frames that I took right-after the star images : same ISO, same camera, same exposure duration and almost same time (less than a minute difference).
If I select one of them as a Dark-Frame, I can see at zoom level = 100% that it actually worsens the “noise”.
So I hoped that by using the Auto-selection tick box, RawTherapee would average my 5 dark-frames and subtract this average dark-frame to my star picture.
However, it seems to me that only 1 dark-frame is selected (see attached screenshot).

How can I force RawTherapee to average my 5 dark-frames ?

I know that I could create an average dark-frame (or “master dark”) with astro-programs like Siril, but they produce a .tiff file that cannot be subtracted to my raw photo in Rawtherapee. I have already tried, but no effect at all is visible. Perhaps RawTherapee detects the dimension, color space and/or CFA are different and as a result abort the dark-frame subtraction.

Waveluke · December 23, 2020, 10:12pm

Use imagemagick to convert from the SIRIL produced tiff to dng, using the command

magick stacked_dark.tiff stacked_dark.dng

Then use exiftool to copy the tags from an original file with the same exposure settings to the new dng to finish the process

exiftool -tagsFromFile direct_from_camera_raw.RAF -x BitsPerSample stacked_dark.dng

Then in the file browser for selecting dark frames, go to the bottom right corner and switch from “same as current photo(RAF)” to “All files”, and your fake raw dng should appear in the directory.

flint · February 5, 2021, 5:21pm

Hi wincent,

Did you ever find a solution to this problem? I have the same issue; that is, I put multiple dark frames from same camera, iso, shutter speed etc in the darks folder specified in preferences. RawTherapee seems to group them correctly and creates 1 template.

But as you reported, only 1 file from the group is selected when the Auto-selection box is checked.

ChasingShadows · February 5, 2021, 9:03pm

As far as I know only one file, the best matching one, is selected. If multiple files with same settings exist, only the newest one is considered. As I said, as far as I know …

heckflosse · February 5, 2021, 10:31pm

Sounds like a bug, as there is code to average dark frames

github.com

Beep6581/RawTherapee/blob/dev/rtengine/dfmanager.cc#L137

    
      
                  updateRawImage();
                  updateBadPixelList( ri );
              }
          
          
    return badPixels;
          }
          /* updateRawImage() load into ri the actual pixel data from pathname if there is a single shot
           * otherwise load each file from the pathNames list and extract a template from the media;
           * the first file is used also for reading all information other than pixels
           */
          void dfInfo::updateRawImage()
          {
              typedef unsigned int acc_t;
          
          
    if( !pathNames.empty() ) {
                  std::list<Glib::ustring>::iterator iName = pathNames.begin();
                  ri = new RawImage(*iName); // First file used also for extra pixels information (width,height, shutter, filters etc.. )
          
          
        if( ri->loadRaw(true)) {
                      delete ri;
                      ri = nullptr;

skygaze · February 17, 2022, 7:03pm

Still an issue… I will first have to learn how to properly report a bug on github…
As a test, I copied one dark frame into a directory, made 3 copies, changed the names, so I expect I have four identical files with only name different. In preferences, select this directory, it finds 4 files, unsure of why it has discovered a template if it has yet to do anything, I have not even selected a light frame. When I do, and go to darkframe, Auto-select then selects only one of the 4 identical files instead of averaging them. Or if it has, it doesn’t tell me, only 1 of the 4 files is listed. I rebooted, created new folder, moved different files just in case the system was remembering previous settings. Even if this never gets fixed, it would be nice to modify the manual to point at the workaround.

ChasingShadows · February 18, 2022, 3:29pm

And you are using a current development version? The last official release was two years ago…

skygaze · February 18, 2022, 4:46pm

Latest stable…Version 5.8 commit date 2020-02-04 … so you are about to tell me it’s “old”. I’m quite new to the nightly build concept, I’ve only done one github pull, so it might take a while for me to catch up.

The workaround (stack in Siril → imagemagick → exiftool → fake raw) is pretty simple and only a minor inconvenience, barely longer than waiting for SIril to do global star registration/alignment. At least I can now use RT’s aberration corrections!

“Chasing shadows” that’s what my mentors called Foucault mirror testing when I ground and polished my mirrors way back.

Jade_NL · February 18, 2022, 6:34pm

You don’t have to clone, pull, make and build yourself if you don’t want to (or don’t have time at the moment to figure it all out). Just grab the latest pre-build development version from here:

skygaze · February 19, 2022, 4:56am

Thank you for your patience with an “accomplished person lacking specific expertise”. I installed the latest (5.8-3073 = 20220213). Do I run the rawtherapee.exe or the rawtherapee-cli.exe? Nothing in the release notes. Anyway, the “dark frame pics only 1 file” remains. I even used 4 identical files with only the name changed and auto-select gives me only one). I will re-read the “how to report a bug” and submit it properly. I am also getting weird flat-field output but to be proper I will rerun with only one variable changing at a time .

Jade_NL · February 19, 2022, 4:52pm

In general you want to run rawtherapee.exe, which gives you a full blown GUI. The cli version can be useful, for example, when doing batch processing or if you have a series that can use the same sidecar when processing and converting to your desired output. It comes with a whole set of options: RawTherapee CLI

skygaze · February 19, 2022, 9:12pm

Ah! Thank you for the clarification and link. I can see the value in it (I enjoy time-lapses). I must have jumped that part when I skimmed the documentation as I was focused on just figuring out my basic needs.

skygaze · February 19, 2022, 9:47pm

I went to the GitHub for Rawtherapee and searched open issues (https://github.com/Beep6581/RawTherapee/issues). Yes, both the dark frame and flat field have open issues. Saved me from bug reporting. Regards.

Florian_BEAULIEU · September 11, 2024, 7:24pm

Hello, (sorry to re-up) I think I’am getting it !
I have explored the code and yes, Rawtherapee does the average but it takes the metadata of the first frame in the batch. This is why it is displaying one image name when checking the ‘auto select’ checkbox.
As an experiment :
Load a random black image from the set.
Crank up the exposure to see what is happening.
Go to Dark-Frame filter, check the “auto select” checkbox, see the modification in the graphic view and note the file name.
Then uncheck the “auto select” checkbox and choose in your dark frames folder the image that correspond to the name you noted before.
See the result is not the same proving it’s indeed averaging the images using the template (49 shots, 1 template).

Now the question is :
How to export this template and delete all the shot to make some space on the HDD ?(making a fake dng with Siril + ImageMagick + Exiftool is not working at all for me).

Edit :
I finally found a fast workflow for generating master bias/dark frames and flat fields [SOLVED] Workflow for creating master bias for Rawtherapee - #4 by Florian_BEAULIEU