Gilmore was born in Richmond, Virginia. His parents were Margaret Evelyn (née Kandle), a church secretary, and James Stuart Gilmore, Jr., a grocery store meat cutter. He graduated from John Randolph Tucker High School and received an undergraduate degree from the University of Virginia in 1971.