I'm aware of the dangers in general, but do you have any specific example for RawTherapee? I would especially be interested if the flag results in some sub-optimal output (e.g. causes some artifacts).
Also, I should clarify that I'm not recommending to build with -ffast-math by default, I'm only suggesting it as something worth trying to get some extra speed. Personally, I haven't experienced any issues with RT since I moved to 64 bit, and I'm using -ffast-math on both my machines. But maybe I'm just lucky