From 1955 to 1957, he coached at Alabama, where he posted a 4–24–2 record, the worst record for a non-interim coach in school history. This included a winless 1955 season, Alabama's last winless season on the field to date, and a 14-game losing streak from 1955 to 1956. In his first year at Alabama, Whitworth was only allowed to hire two of his own coaches and forced to retain the rest of former coach Harold Drew's assistants.
From 1955 to 1957, he coached at Alabama, where he had a dismal record of 4-24-2, the worst for any non-interim coach in school history. This included a winless season in 1955, which was Alabama's last winless season on the field, and a 14-game losing streak from 1955 to 1956. In his first year at Alabama, Whitworth had limited control over hiring staff, only being allowed to bring two of his own coaches while retaining the rest of Harold Drew's assistants.