In August 2018, he resigned the Labour whip citing anti-semitism in the party, and a "culture of intolerance, nastiness and intimidation" in parts of the party, including in his own constituency. Field lost a confidence vote in his constituency party a month before his resignation, after siding with the government in Brexit votes. His resignation of the whip also led to his departure from the wider membership of the Labour party, according to the Labour National Executive Committee, although Field disputes this.