Color balance RGB has built-in gamut mapping to avoid creating crazy colors.
When you rotate hues, you do so at constant luminance and chroma. But nothing guarantees than the new hue at same chroma still lies within the working RGB space gamut. So it gets clipped.
To avoid this, you need to reduce the global linear chroma. But ultimately, this is not very suitable for large hue shifts. Hue is an angle, in 3D color spaces, and you will have an hard time to blend large angle shifts smoothly.