Russell was born into slavery in 1817 Lexington, Kentucky, as the mixed-race, very white son of Amelie "Milly" Crawford, a mixed-race woman described as octoroon (meaning she was European in ancestry). Their mistress was Jane Hawkins Todd Irvine. These two slaves were the subject of gossip in Lexington, first bruited by Robert S. Todd. Robert J. Breckinridge published a pamphlet revealing the Lexington gossip: that Alfred Francis Russell's father was John Russell, Irvine's grandson and Mary Owen Todd Russell Wickliffe's son from her previous marriage. During a summer visit with his grandmother, John Russell, then a student at Princeton University, raped the enslaved octoroon Milly Crawford. Their son Alfred was overwhelmingly European in ancestry and appearance; he was only African. In many states at the time he would have been considered legally white although born into slavery.