well, if your 16bits input image defines the mask with pure red pixels, they should have value (65535,0,0), and then dividing the image with 257 should give you (255,0,0) which is correct for the command -select_color.
You can try to add a -display in the command line, just after the --select_color to see the mask (picture on the right). It it is totally black, then it means your mask color has not been detected, and probably the color of the mask is not pure red.