Gregory was born in 1638. His mother Janet was the daughter of Jean and David Anderson and his father was John Gregory, an Episcopalian Church of Scotland minister, James was youngest of their three children and he was born in the manse at Drumoak, Aberdeenshire, and was initially educated at home by his mother, Janet Anderson (~1600–1668). It was his mother who endowed Gregory with his appetite for geometry, her uncle – Alexander Anderson (1582–1619) – having been a pupil and editor of French mathematician Viète. After his father's death in 1651 his elder brother David took over responsibility for his education. He attended Aberdeen Grammar School, and then Marischal College from 1653–1657, graduating AM in 1657.