If an asylum seeker came into your shop and you asked if he spoke English and he just shrugged his shoulders and looked blankly at you, and you then spoke to him in English (what else could you do?) That wouldn't make you racist.
Bearing in mind most of us are married to "foreigners", how does it make you feel seeing all those illegal immigrants sneaking onto trucks in Calais?