Despite his efforts, Socialist Party practice and positions included segregated locals in the South and racist positions on Asian immigration. Harrison's position in the Party was also affected by his alignment with its left-wing and the IWW, who were engaged in factional struggle with its right-wing faction: many leftists exited after Haywood (who had been a member of the SPA's executive committee) was expelled from the Party in 1912. The Socialist Party of New York was led by Morris Hillquit, a prominent figure on the right, and party leaders in New York City began restricting Harrison's activities, including preventing his own branch from having him as a speaker. Harrison concluded that Socialist Party leaders, like organized labor, put the white "Race first and class after." After writing a note to the NYC executive committee telling it to "go chase itself", he was suspended from the Party for three months: he resigned from the Socialist Party in 1918, before his suspension was over, but was periodically referred to as a socialist by others for years afterwards. After resigning from the Socialist Party, Harrison increased his activism within the 1920s Single-Tax movement.