Gregory was born in Darjeeling, West Bengal, India. His father, Neville, was a tea planter. His mother, Marina, was a Burmese school teacher. He moved to London in the early 1960s. In 1984, he married Christine. Their daughter, Dominique, was born in 1997.