I think the main reason a Family permit can be refused by a EU member state is if they believe the marriage is not genuine.