Time to tell your husband to choose who he wants to share his life with.....
Looks like that you aren't having much support from him at all.....
I suggest you have a heart to heart with your husband, without the lodgers present, and explain how you feel and how bad they make you feel.
Continuing to suffer in silence will only make you feel worse and unfulfilled.