In his first full season as the Broncos' coach in 1965, Speedie's team posted a 4–10 record. After two losses to begin the 1966 season, Speedie resigned and assistant Ray Malavasi took over. He said the move was in the best interest of the club. Speedie had a 6-19-1 record as the Broncos' coach. He then accepted a scouting position with the organization and was based out of his home in Laguna Hills, California. He held the post until his retirement in 1982.