Stay with your family, the few extra days. Your husband will understand, you have a lifetime together.
Your family, will be really upset that you are leaving anyway, be that in a few days or whenever. It will be another happy memory, for you & your family to look back on.