Dickinson was born at "Croisadore", his family's tobacco plantation near the village of Trappe in Talbot County, Province of Maryland. He was the great-grandson of Walter Dickinson who emigrated from England to Virginia in 1654 and, having joined the Society of Friends, came with several co-religionists to Talbot County on the Eastern Shore of the Chesapeake Bay in 1659. There, with on the banks of the Choptank River, Walter began a plantation, "Croisadore", meaning "cross of gold." Walter also bought on St. Jones Neck in what became Kent County, Delaware.