Exactly that’s the problem.
If you hold up your thumb 
and then close the right eye, open the left. Then then alternate opening the left closing the right you will see the thumb jumping and not staying at the same position. The same happens in your panorama. Fixing those error is really hard.
That’s the reason why there is a panorama head. It fixes this error and makes it easy to create a panorama if you have an object infront of you you want correctly displayed.
However I’m interested if someone can just get this fixed with hugin 