I know, mate.
I also think that the situation is stacked against the Brits, with genuine claims.
But on the other hand, if you were to live in one of the other EEA countries, you would have the same rights.
The problem for you would be that you would still a settlement visa for the Mrs once you did decide to return to the UK for good.
Vastly unfair.