I’ve just come across this same behaviour, and I think it’s normal. Annoying, but intended.
As I understand it, RT is developed with camera sensors in mind (raw images), so a 2:3 ratio is expected. In this scenario, there wouldn’t be many problems, and the Maximum thumbnail height setting in Preferences is all that is needed to control the size of thumbnails.
The problem is that, to my knowledge, nobody prevent us to process jpegs, pngs or tiffs with a wildly different aspect ratio (say 1:120), and then those thumbnails try to get the max height up to where a hardcoded minimum column width comes into play. And those thumbnails go even over 1:1 and start getting enlarged (say a 10x50px image and a max height of 350px).
Maybe a simple solution would be adding another optional setting called Maximum thumbnail width in a future release.
In the meantime, perhaps a workaround is putting those images in a separate folder…