On April 23, 1844, Rathbun was involved in a physical confrontation on the House floor with former Speaker John White.
When Rathbun told White to be quiet, White confronted him and their disagreement lead to a fistfight between the two with dozens of their colleagues rushing to break it up.