I think the choice of D50 for XYZ has no mathematical consequence, unless we care about XYZ values.
When converting from one linear RGB colorspace to another linear RGB colorspace, every pixel might be multiplied by each matrix in turn. But for performance, the CMS will multiply the matrices together, resulting in a single 3x3 matrix, and that is used to multiply each pixel.
A chromatic adaptation matrix (CAM)depends on the two white points and the CAT (Bradford or whatever). It doesn’t depend on the primaries. So a CAM from D65 to D50 is the mathematical inverse of the CAM from D50 to D65 (provided the same CAT was used).
Chromatic adapatation from WP1 to a different WP2 and then to a third WP3 is mathematically the same transformation as from WP1 directly to WP3.