After the war, Johnson returned to the House of Representatives. The state legislature appointed him to the Senate in 1819 to fill the seat vacated by John J. Crittenden. With his increasing prominence, Johnson was criticized for his interracial relationship with Julia Chinn, a mixed-race slave who was classified as octoroon (or seven-eighths white). Unlike other upper-class planters and leaders who had African-American mistresses or concubines, but never acknowledged them, Johnson treated Chinn as his common law wife. He acknowledged their two daughters as his children, giving them his surname, much to the consternation of some of his constituents. It is believed that because of this, the state legislature picked another candidate for the Senate in 1828, forcing Johnson to leave in 1829, but his Congressional district voted for him and returned him to the House in the next election.