Palfrey was elected as a Whig to the Thirtieth Congress (March 4, 1847 – March 3, 1849). He was a "Conscience Whig" who opposed slavery, having freed sixteen slaves inherited from his father, who, like his two brothers, was a successful Louisiana plantation owner. In Washington, he was a member of a small group of anti-slavery congressmen, including Joshua Giddings, who met regularly. His anti-slavery views alienated him from more conservative members of his district, such as the "Cotton Whigs," and in 1848 he was unsuccessful in his campaign for re-election on the Free-Soil ticket. (The seat remained vacant for the 31st Congress despite multiple elections, in which no candidate achieved the needed 51% majority.) He was also the Free-Soil candidate for Governor of Massachusetts in 1851.