Is there a file size limit in RawTherapee?

When opening in the File Browser a directory which contains tiff or dng files of large size, those files are not listed among the thumbnails. Files of those types having smaller size are displayed fine.

The file sizes in question are about 2.5Gb and the resolution is around 23000x18000 pixels. These are scans of large format negatives.

Is there an explicit limit on the supported file size in RawTherapee and if there is, are there specific issues associated with lifting that limit (apart from the linear performance hit)?

That’s an impressive amount of pixels for a scan :eyes:
In any case, the limit is usually just your RAM

As mentioned, the images are not even listed among the thumbnails. The available ram is 64Gb. Lightroom, as one example, on the same machine does handle these files reasonably well (although it has a limit of 512Mpix, which comes pretty close).

To make my question more specific, is there a coded limit on the file size in RT?

The file browser also has to generate a preview, right? In that case a 2.5gb scan may take sometim to have a preview generated.

There is also a list of extensions to consider located in the preferences. Make sure your file types are checked.

Here is a partial answer.

With more experimentation it turned out the issue is specific to dng files. Tiffs of size around 2.5Gb are displayed fine. Dng files of the same size are just skipped as there is no disk activity going on after navigating to a folder with a single large dng. Dngs of around 1.4Gb open fine, which may point to the use of a signed int to store the file size or offset (a popular issue these days with the scanned dngs).

RT seems to use a modified dcraw code for interpreting the files, and it turns out the original dcraw cannot open a 2.5Gb dng either, failing with Cannot decode file error. The RT dcraw code does not have that branch and may not have that issue at all, but as there is no apparent message, it is hard to tell at a glance.

1 Like

Can you please provide a problematic file for download? That would allow us to work on that issue.

2 Likes

I will provide the large file a bit later.

In the meantime I noticed similar behavior with seemingly completely regular tiffs. One is only 20Mb but it does not appear in the File Browser. It’s dng counterpart (same scan, saved as raw dng) is listed and opens fine. Both files are scanned in a similar fashion using VueScan. I can’t tell whether the tiff was saved as a raw or plain though. It is opened by other programs without issues.

I’m attaching the small tiff in question here.stouffer-wedge-1200-1s-2mm-dry.tif (20.0 MB)

Let me know if you also want its dng counterpart.

Is that a 4-channel tiff, 4th channel being alpha channel (infrared) for removing scratches?

It is a 3-channel 16bpc RGB tiff, no alpha channel.

Ok, for your last tiff: RT doesn’t show it because the ratio (heigh/width) is more than 10, to not mess up the file browser:

So we need also the very large tiff from op

Will send the large one a bit later (maybe not today).

Just curious, why was the choice made in RT to not show the placeholders for the files in directory before the thumbnails are loaded, and then display the thumbnails as they get prepared? This would avoid all the jumping that currently happens while the thumbnails are being rendered.

Also, why not show the images of less typical proportions scaled down to fit the placeholder? As long as there is a proportional frame, establishing the placeholder boundaries, there will be no confusion. This is what I believe most general purpose image viewers do.

This could be argued to be less of a concern for RT. Being a raw processor, it is not likely to encounter such images often. Yet, although a scanner-produced raw may not be a raw in full sense, files like this take valid part in the raw workflow, being used for process calibration.

Sorry about all the noise, I see there is an open issue in that area.

@heckflosse Here is a link to a 2.3Gb 16bpc RGB dng. This is just a blank negative sheet, scanned with VueScan and saved as a raw dng. This file does not show up in the File Browser, but the same exact scan buffer saved as a regular tiff and having approximately the same size shows up and opens fine.

Please note I will delete this file after a couple of weeks, so please download.

1 Like

@nicnilov Thanks for the DNG

In first post you wrote

Now you wrote

Which means tiff works, but dng does not. darktable shows and opens the dng, means I will investigate why RT does not…

Yes. A couple of days after the first post I wrote this

So indeed, at this point it’s only about the dng.

Sorry, somehow I missed that :frowning:

No worries, let me know if I can be of use.

@heckflosse There seems to be an issue with large tiffs saving as well. When the file is exported to TIFF 16bpc integer, and the outcome exceeds 4Gb, it gets truncated to about 7kb and contains mostly the metadata part. Saving of the same file to JPEG works, resulting in about 104Mb file which opens fine. Also saving to TIFF 8bpc works fine.

The same also happens in darktable, if that’s relevant.

Sounds like a 4GB output issue…
I will investigate, though with low priority!

1 Like

Hmm:
https://www.awaresystems.be/imaging/tiff/faq.html#q8

Right… Well, how about the BigTIFF? I realize however this is a different thing and even if there is a way to implement a transparent support, it may not be trivial when saving. Moreover, BigTIFF is not a standard and its support may be limited. Or may be not, as libtiff supports it since 2009: https://gitlab.com/libtiff/libtiff/-/commit/e7ccb79ae3b5dd68366c8a2f50b5a06fbcbe01e3.

In any case it seems reasonable to expect an error message when the tiff being saved is going to breach the 4Gb boundary, and also not to output a corrupted file.

1 Like