Like most southerners at the time, Callaway grew up as a supporter of the Democratic Party. He switched parties out of frustration with the Democrats' more liberal policies regarding desegregation. In 1964, he ran as a "Goldwater Republican" for a seat in the House of Representatives from Georgia's 3rd congressional district. He won, having defeated the former lieutenant governor, Garland T. Byrd, 57 percent to 43 percent. Callaway thus became the first Republican elected to the US House from Georgia since the Reconstruction era.