Dettori held on to score with the 11-1 chance, trained by Willie Mullins, as the challenge from last year's winner came too later under Ryan Moore.
Order Of St George had won his previous six races for trainer Aidan O'Brien.
Earlier, O'Brien was beaten into second as Intricately gave son Joseph, 23, his first Group One winner as a trainer.
The 25-1 chance, bred by Joseph's parents and ridden by his brother Donnacha, won the Moyglare Stud Stakes, from Hydrangea.
Their father, whose favourite Idaho was unseated in the  English Leger at Doncaster on Saturday, did have something to celebrate, with Churchill winning the Vincent O'Brien National Stakes.
In the big race, Dettori's decision to send his mount out from the front paid dividends for a trainer more associated with big jump racing wins rather than a flat racing Classic.
"It was a surprise, but a Classic win is always great. Willie is a top man, his horses are always in good nick and we tried something different and it worked out," said the 45-year-old Italian.
Mullins said the "horse and jockey were brave". He will now aim Wicklow Brave at the Melbourne Cup in November.
Wicklow Brave, ridden by Frankie Dettori, caused a big shock as he beat the 1-7 favourite Order Of St George to win the Irish St Leger at the Curragh.