Ford was an Illinois associate of Isaiah L. Potts and the Potts Hill Gang, highway robbers, of the infamous Potts Inn. James Ford also was an associate of John Hart Crenshaw, an illegal slave trader and a kidnapper of free African Americans, and may have taken part in the Illinois version of the Reverse Underground Railroad. At one point, the outlaws used "Cave-in-Rock" as their headquarters on the Illinois side of the lower Ohio River, approximately around 85 miles below Evansville, Indiana.