The 2013–14 season saw Tozer featured regularly in the first team for the first five matches at the start of the season. He soon had his playing time reduced following the arrivals of Kevin Amankwaah. With eleven appearances made, Tozer joined League One club Colchester United on an initial one-month loan on 28 November 2013 until 4 January 2014 to provide defensive cover over the Christmas period. He made his debut for the club on 14 December in a 4–0 home defeat to Notts County. He was replaced on 67 minutes by Tosin Olufemi while the score was 2–0. After making one appearance for Colchester United, Tozer was recalled by the club on 23 December 2013. Following his return to Northampton Town, Tozer made a return to the first team, which saw him awarded Northampton Player of the Month for January. Despite being on the substitute bench for nine matches towards the end of the season, Tozer would make 29 appearances for the club. He soon expressed his future in doubt following the arrival of Manager Chris Wilder, who arrived during the season.