no if you have ilr you will not have a problem, as long as your not out of the uk 2yrs or more in one trip your fine
did you make it clear in your visa app that you have been lving with your daughter 1yr and you and only you make the decisons that effect her life ..
i'm surprised that your visa was refused, thou i dont know what your daughter told the embassy, maybe they got the inpression he still is involved in her life..
