This is a great point. I experience this often when I miss the focus by a bit: it would not be too bad by itself, but some other part is super-sharp and draws the eye.
Some of my M43 primes are eerily sharp, so much that I have acquired the habit of unsharpening less relevant parts of the image with diffuse & sharpen and masking.