You don’t need raster brush for that, and I don’t know any brush with edge detection built-in. Use a regular polygon vector mask and increase the mask feathering by a lot.
You know you can manually discard bad edges from the horizontal/vertical lines detection just by painting on UI preview with right click ?
Sounds like a bad alpha merging but needs investigation. You know you can send me samples.