If you insist on the thumbnail cache being temporarily you probably want to disable the secondary disc cache at all (This might decrease lighttable performance ) or manage your cache directory on a filesystem basis otherwise.
The latter might be something, metadata driven views of git-annex might be helpful for
https://git-annex.branchable.com/tips/metadata_driven_views/