After shelving his swimming career on his return to Australia for four years, Charlton again broke the Australian record in both the 440 and freestyle events at the 1932 New South Wales championships to gain selection for the 1932 Summer Olympics in Los Angeles, the oldest member of the team at 25 years of age. He contracted influenza a fortnight before the Games after arriving in the United States. Charlton raised hopes that he had recovered when he won his heat, but then only won third place in the semi-final, although he still progressed to the final of the 400 m freestyle. Charlton finished in a distant sixth, some ten seconds behind the winner. In the 1500 metres, Charlton finished second in his heat, before coming fifth in his semi-final, resulting in his elimination.