In 1802 'Colonel Grant' was elected to the House of Commons for Elgin Burghs, a seat he held until 1806, and then represented Inverness Burghs from 1806 to 1807, Elginshire from 1807 to 1832 and Elginshire and Nairnshire from 1832 to 1840. In 1840 he succeeded his elder brother as sixth Earl of Seafield, and sat in the House of Lords as a Scottish Representative Peer from 1841 until his death in 1853. He therefore attended Parliament for a period of 50 years, voting against the Reform Act of 1832 while sitting in the Commons. Sir William Fraser reported: "In politics his Lordship was a Conservative, and during his long public career loyally supported his party. He was a warm supporter of Sir Robert Peel."