In the lead up to the 2004 federal election, Brown put his name forward as a candidate for the Conservative Party nomination race for the riding of Barrie. Brown defeated Rod Jackson and Douglas Edwards to win the party nomination in what Jackson considered a nasty race. In the election, he lost to Liberal incumbent Aileen Carroll by 1,295 votes. Brown ran again in 2006, this time defeating Carroll by 1,523 votes. He was re-elected in the 2008 election by 15,295 votes over Liberal candidate Rick Jones.