Yeah, higher precision data would obviously be ideal, but in general, the GPU manufacturers are now setting things up such that if you choose int8, you get double throughput of FP16 or more - and it appears that increasing network complexity is far more beneficial than precision of the individual parts of the network.
1 Like
This depends on application and you can further quantize different layers differently - for images you only care that, the closer you get to output layers, you might want to be getting something better than int8… For example, some applications care more about range than precision, so bfloat16 was gaining traction in place of FP16. Now we’ll even have FP8…
3 Likes