From what I understand, no matter how you spin it, a lot of voters in the Leave camp are anti-immigrant.
While most sane people realize that even if the U.K. leaves the E.U. and builds a wall like Trump wants for the U.S. that the immigration situation would remain unchanged, I can still imagine this ultimatum making it explicit will make all the xenophobes foam at the mouth.
EU to Britain: No access to single market without migration • 2016 Jun 29 • Angela Charlton and Lorne Cook • Yahoo! Finance