Here is a partial answer.
With more experimentation it turned out the issue is specific to dng files. Tiffs of size around 2.5Gb are displayed fine. Dng files of the same size are just skipped as there is no disk activity going on after navigating to a folder with a single large dng. Dngs of around 1.4Gb open fine, which may point to the use of a signed int to store the file size or offset (a popular issue these days with the scanned dngs).
RT seems to use a modified dcraw code for interpreting the files, and it turns out the original dcraw cannot open a 2.5Gb dng either, failing with Cannot decode file error. The RT dcraw code does not have that branch and may not have that issue at all, but as there is no apparent message, it is hard to tell at a glance.