Why should it be a problem?
It may cause some comments from other people, but thats their problem, not yours.
Does it 'feel' right?
Do you notice the age gap when you're together?
Its the way YOU feel that matters. Don't even worry about what others might think - it just isn't worth it.