Man of the series Rilee Rossouw hit 122 from 118 balls in a 178-run partnership with JP Duminy to help the Proteas to a commanding 327-8 at Newlands.
Australia's David Warner struck a brilliant 173 from 136 balls as the visitors were dismissed for 296.
Australia last ODI whitewash was a 4-0 loss to England in 2012.
They had never lost a five-match series to nil.
South Africa had been struggling at 52-3 when Rossouw and Duminy came together and took advantage of the visitors' inexperienced bowling attack to score freely.
Duminy was bowled by Joe Mennie for 73, while Rossouw - having reached his century off 100 balls - eventually fell with five overs to go.
David Miller then marshalled the tail to take 46 off the final five overs and set Australia an imposing target.
The highest successful ODI chase at the ground is 258, but opener Warner's magnificent effort brought an improbable victory within reach.
Lacking meaningful support, the 29-year-old held his side together as wickets fell around him - hitting 24 fours in his innings - before being run out in the 48th over attempting a second run to Imran Tahir.
"I thought he was outstanding," Australia captain Steve Smith said of Warner. "The rest of us just didn't stand up. This has been a tough series. We have been outplayed."
South Africa condemned Australia to their first 5-0 one-day international series whitewash with a 31-run victory in the final match in Cape Town.