Hoyte made his senior debut as an 89th-minute substitute in Arsenal's 6–1 Premiership victory over Southampton in May 2003. He made one equally late league appearance in their 2003–04 FA Premier League-winning season, which was not enough to qualify him for a medal. Hoyte played in the 2004 FA Community Shield, again from the bench, as Arsenal beat Manchester United, and made his first senior start on 28 August 2004, away to Norwich City, as a last-minute replacement for centre-back Pascal Cygan who was injured in the warm-up. Although he conceded a penalty in Arsenal's 4–1 win, the "Independent" reported that he " have played well enough to forestall a proposed loan to Ipswich". The loan to the Championship club fell through after Arsenal failed to sign backup for Lauren at right back, so Hoyte remained at Arsenal and finished the season with 12 appearances in all competitions.