In Backley's final year of competition in 2004, where he was hoping to add to his Olympic medal haul, he struggled to find form and at the 2004 Summer Olympics in Athens, Greece, in a wide-open field, he only made the final as the last non-automatic qualifier. In the final his third round throw of 84.13 metres pulled him up to fourth place but he could not improve on this and finished in that position unable to gain a medal at his fourth consecutive games.