Ashley was born in Allegheny County, Pennsylvania, to John Ashley, a bookbinder and Campbellite preacher who evangelized in Kentucky and West Virginia, and his wife Mary A. (Kilpatrick) Ashley of Kentucky. As a boy in the Ohio River valley, Ashley saw coffles of chained slaves being walked to the Deep South, boys his own age being sold, and even white men who refused to let their cattle drink from a stream in which his father had baptized slaves. He grew to hate the "peculiar institution" (which he considered a violation of Christian principles) and the oligarchy that supported it.