Ouch.
Not much wrong with the work flow. Maybe you need to invert your selection back to original. It would help knowing details of the image, a flattened jpg / png or multi-layer .xcf or psd …or…
Then the intent:
Mask out an area as an overlay to cover existing details: demo animation https://i.imgur.com/1EDaO0A.mp4 Use the rounded corners option in the rectangular selection tool.
or
A whole canvas sized layer with rounded corners: demo animation https://i.imgur.com/6bIwVEO.mp4 Use Select → Rounded Rectangle.
No need to invert the selection in either case.