I would say this is ignorance. My mother has had the comments after 8 years trying to have bb. "Oh, you only have the one" or "You like the money too much, you are selfish" and then another 10 years after me.
They do not know your pain. The comments hurt, but do not despair. Only you know your feelings on this.