Phillips was originally a member of the Labour Party, and contested the 1970 General Election for Labour in Harwich. But he was expelled from the party in 1973 for publishing a letter in The Times deploring the party's policy of nationalising the top 100 companies and indemnifying trade unionists with criminal convictions, and joined the Liberals in the early 1970s. He later stood as a Liberal candidate in Saffron Walden in both a 1977 by-election and the 1979 General Election, and later stood as a Liberal in Gainsborough and Horncastle in 1983.  He also contested the first European Parliament elections as a Liberal in 1979, in North Essex.