Gilman first won election to Congress in 1972 to represent New York's 26th congressional district. New York’s congressional districts had been redrawn in reapportionment, and in the general election Gilman defeated incumbent congressman John G. Dow, a Democrat who had been serving in New York's 27th congressional district, (which included most of the territory and population of the new 26th district). Gilman subsequently served in Congress from January 3, 1973, until January 3, 2003. During his time in Congress, he was chair of the House Committee on International Relations (104th through 106th Congresses).