If you read his biography, he was not a racist, and was a very intelligent man. He was also a man of his times, and if anything, his real mistake was to use the ancient government habit of "lying" and embellishing the truth, to score points about immigration - something which would turn into one of the great "must not says", but which he maybe didn't see the way this tide was turning.

The incident I relate to, was the supposed constituent of his who did not dare go out because of immigrants who had moved into that street.
He wouldn't name the constituent, but didn't think the media would then set out to painstakingly go through the electoral register and other sources to try track this constituent. No-one could be found who would fit the bill (older indigenous person in a street of many incomers). Yes it was probably an embellishment, and yes, Mr Powell made the speech just at the wrong time.

Racist though? Slander!