When opening in the File Browser a directory which contains tiff or dng files of large size, those files are not listed among the thumbnails. Files of those types having smaller size are displayed fine.
The file sizes in question are about 2.5Gb and the resolution is around 23000x18000 pixels. These are scans of large format negatives.
Is there an explicit limit on the supported file size in RawTherapee and if there is, are there specific issues associated with lifting that limit (apart from the linear performance hit)?
As mentioned, the images are not even listed among the thumbnails. The available ram is 64Gb. Lightroom, as one example, on the same machine does handle these files reasonably well (although it has a limit of 512Mpix, which comes pretty close).
To make my question more specific, is there a coded limit on the file size in RT?
With more experimentation it turned out the issue is specific to dng files. Tiffs of size around 2.5Gb are displayed fine. Dng files of the same size are just skipped as there is no disk activity going on after navigating to a folder with a single large dng. Dngs of around 1.4Gb open fine, which may point to the use of a signed int to store the file size or offset (a popular issue these days with the scanned dngs).
RT seems to use a modified dcraw code for interpreting the files, and it turns out the original dcraw cannot open a 2.5Gb dng either, failing with Cannot decode file error. The RT dcraw code does not have that branch and may not have that issue at all, but as there is no apparent message, it is hard to tell at a glance.
In the meantime I noticed similar behavior with seemingly completely regular tiffs. One is only 20Mb but it does not appear in the File Browser. It’s dng counterpart (same scan, saved as raw dng) is listed and opens fine. Both files are scanned in a similar fashion using VueScan. I can’t tell whether the tiff was saved as a raw or plain though. It is opened by other programs without issues.
Will send the large one a bit later (maybe not today).
Just curious, why was the choice made in RT to not show the placeholders for the files in directory before the thumbnails are loaded, and then display the thumbnails as they get prepared? This would avoid all the jumping that currently happens while the thumbnails are being rendered.
Also, why not show the images of less typical proportions scaled down to fit the placeholder? As long as there is a proportional frame, establishing the placeholder boundaries, there will be no confusion. This is what I believe most general purpose image viewers do.
This could be argued to be less of a concern for RT. Being a raw processor, it is not likely to encounter such images often. Yet, although a scanner-produced raw may not be a raw in full sense, files like this take valid part in the raw workflow, being used for process calibration.
Sorry about all the noise, I see there is an open issue in that area.
@heckflosse Here is a link to a 2.3Gb 16bpc RGB dng. This is just a blank negative sheet, scanned with VueScan and saved as a raw dng. This file does not show up in the File Browser, but the same exact scan buffer saved as a regular tiff and having approximately the same size shows up and opens fine.
Please note I will delete this file after a couple of weeks, so please download.
@heckflosse There seems to be an issue with large tiffs saving as well. When the file is exported to TIFF 16bpc integer, and the outcome exceeds 4Gb, it gets truncated to about 7kb and contains mostly the metadata part. Saving of the same file to JPEG works, resulting in about 104Mb file which opens fine. Also saving to TIFF 8bpc works fine.
The same also happens in darktable, if that’s relevant.
Right… Well, how about the BigTIFF? I realize however this is a different thing and even if there is a way to implement a transparent support, it may not be trivial when saving. Moreover, BigTIFF is not a standard and its support may be limited. Or may be not, as libtiff supports it since 2009: Add a note that BigTIFF work is completed (e7ccb79a) · Commits · libtiff / libtiff · GitLab.
In any case it seems reasonable to expect an error message when the tiff being saved is going to breach the 4Gb boundary, and also not to output a corrupted file.
