Ashley was mostly self-taught in elementary subjects, although his father wanted him to follow family tradition and become a Baptist minister. Rather than attend a seminary, the 14 year old ran away to become a cabin boy on Ohio and Mississippi River boats, and later worked as a clerk on those boats. He had begun helping slaves to escape as early as 1839, and late in his life Ashley relished telling stories of the families he had saved as a 17 year old.He told the story later in life, which came down through the family that, when he left the home at 14, the last words his father said to him as he went off was: "You're on the straight road to Hell, boy!" Twenty years later, when he was elected to the U.S. House of Representatives, his first act, when he sat down in his office in Washington, D.C., was to pen his father, to whom he had not spoken in twenty years, a letter on the Congressional stationary: "Dear Father, I have just arrived!"