There's no major immigration problem in leaving UK for 6 months under your specific EEA Family member residence.
However, do be aware that absences over 6 months will mean you have broken the EEA residency rules and your re-entry back to UK will effectively 're-start' the 5 years' residency period towards Permanent Residency and British Citizenship.
Hope I've explained that OK for you![]()