Gray was born in Sauquoit, New York, on November 18, 1810, to Moses Gray (b.February 26, 1786), then a tanner, and Roxanna Howard Gray (b. March 15, 1789). Born in the back of his father's tannery, Gray was the eldest of their eight children. Gray's paternal great-grandfather had arrived in Boston from Northern Ireland in 1718; Gray's Scotch-Irish Presbyterian ancestors had moved to New York from Massachusetts and Vermont after Shays' Rebellion. His parents married on July 30, 1809. Tanneries needed a lot of wood to burn, and the lumber supply in the area had been shrinking, so Gray's father used his profits to buy farms in the area, and in about 1823 sold the tannery and became a farmer.