He was born in Poughkeepsie, New York. Gilman graduated from Middletown High School in Middletown, New York in 1941 and received a B.S. from the Wharton School Finance at the University of Pennsylvania in 1946. He also earned an LL.B. from New York Law School. Gilman served in the United States Army Air Corps from 1942 until 1945 during World War II.