If this is the 2GB model, then yes, it is expected. Basically it costs more, at some points for your GPU, to split the image into tiles that fit into the GPU’s memory and process them than it does for the CPU to process the image.
If it is the 4GB model, that should be a bit better, especially if you’re processing <24mpix images, but it can still happen with resource hungry modules.
I highly depends on which modules you are using. I’ve been processing 16-Mpx-images on a 2GB GPU for a long time and it was working pretty well. Then came diffuse or sharpen which seems to require lots of memory, so thanks to tiling it was abysmally slow on the GPU (iirc, export time for images was measured in minutes). For some time I disabled OpenCL so that it was at least somewhat usable, but eventually upgraded my graphics card (now I have a second hand GTX1060 with 6 GB and no issues, even with the 24 Mpx images of my new camera).
In case you decide to upgrade the GPU and don’t want to spent too much money: Consider buying a used one, e.g. from ebay and then sell the old one. I upgraded from a GTX 760 to 1060 this way for <50 Euro (I was quite surprised that someone was still willing to pay 30 Euro for the 760…).