Gillom signed with the Browns in early 1947, making him the third black player to join the team after Bill Willis and Motley. Before the season started, he played in the College All-Star Game, a now-defunct matchup between the National Football League (NFL) champion and a selection of the best college players from around the country. Gillom played well in the game as the All-Stars beat the Chicago Bears, 16–0. With the Browns, Gillom was used mostly as a punter and a defensive end. He came in second in the AAFC in punting average in his rookie season at 44.6 yards as Cleveland finished the regular season with a 12–1–1 record and beat the New York Yankees to win the AAFC championship. "Gillom had such a powerful leg and kicked the ball so far; before that punters used to line up 10, 12 yards behind the center", running back Sherman Howard later said. "He started the 15-yard drop. And with Horace, he would kick it so high that by the time guys got down, the ball was coming down, so most guys had to fair catch."