Benjamin scaled back his involvement in politics in the late 1840s, distracted by his plantation and law practice. His mother Rebecca, whom he had brought to New Orleans, died in 1847 during a yellow fever epidemic. In 1848, Benjamin was a Whig member of the Electoral College; he voted for fellow Louisiana planter, General Zachary Taylor, who was elected U.S. President. He and other Louisianans accompanied President-elect Taylor to Washington for his inauguration, and Benjamin attended a state dinner given by outgoing president James K. Polk. In 1850, Millard Fillmore, who succeeded Taylor after his death earlier that year, appointed Benjamin as judge of the United States District Court for the Northern District of California. He was confirmed by the Senate, but he declined the appointment as the salary of $3,500 was too small. The following year, Benjamin assisted the United States Attorney in New Orleans in prosecuting American adventurers who had tried to spark a rebellion against Spanish rule in Cuba, but two trials both ended in hung juries.