512px images on a gpu with 6gb vram will take only seconds. (But longer if you use control net). Use sd 1.5 for that. This would probably cover most of your needs for inpainting/outpainting. If you want to generate a sky for sky replacement on say a 6000px image then just upscale after.
1024px images will need MUCH more vram if you want fast results. Sdxl is designed for that.