Sir Keir Starmer has doubled down on his comment that Britain risks becoming an "island of strangers" despite comparisons being made to Enoch Powell.