I understand your concerns Purple, why is he still friends with her? I find it difficult to stay friends with my ex's. These are natural questions to ask when in any relationship, especially a LDR where paranoia can often creep in. However, I have to stay reasonably friendly with my ex wife for the sake of our kids, we don't want them to see us falling out and hating each other. When I come to PI for 2 weeks in Jan, my ex wife has volunteered to look after my house and feed my cats every day. I haven't decided to take up her offer yet, but if I did, I do trust her to take good care of the cats and to not ransack my house.
I think you have answered the question yourself by saying he has visited you 5 times this year and he married you, he wouldnt do that unless he really loved you. Personally I don't think you have anything to worry about, but have a right to ask difficult questions