Interesting view. Relationships are about compromising. Putting off having your own children, because your partner has already had theirs is quite a big one, especially if his children are grown up. The most important thing is the happiness in my opinion, but if you feel it's compromising your happiness, then it may end up being a nagging thought in your head.