I think performance is key for a nice user experience, and at least on the same machine it should be no compromise. Having not too much experience with client-server architectures, I guess remaining on the GPU for display will be impossible with that architecture? (On a single machine of course)
However, the idea of having a more modern image processing library, separate from the UI, was the same I was having and asking about in this thread:
https://discuss.pixls.us/t/image-signal-processing-library
I think that would be really great to help development of more independent UIs based on different user needs.