James Monroe (1730–1772) married Spence Monroe in 1752 and they had five children: Elizabeth, James, Spence, Andrew, and Joseph Jones. His paternal great-grandfather Patrick Andrew Monroe emigrated to America from Scotland in the mid-17th century. In 1650 he patented a large tract of land in Washington Parish, Westmoreland County, Virginia. Monroe's mother was the daughter of a wealthy Welsh immigrant who had settled in nearby King George County, Virginia. Also among James Monroe's ancestors were French Huguenot immigrants, who came to Virginia in 1700. At age eleven, Monroe was enrolled in the lone school in the county. Monroe attended this school for
James Monroe Gregory for six years. In 1887 he was a candidate to replace James Campbell Matthews as Washington, DC Recorder of Deeds, although the position went to William Monroe Trotter. Gregory was again a candidate for the position to succeed Trotter in late 1889, but the position went to Blanche Bruce. Gregory was president of the American Association of Educators of Colored Youth which was he founded in 1890 and led throughout its existence. He was a delegate to the 1892 Republican National Convention, and expressed interest in the Recorder of Deeds job again, which in 1893 went to C. H. J.