The beginning of the end started back in 1972 with all the Ugandan Asians, then Ted Heath drove in the final nail with EU membership.