Media playback is not supported on this device
Aaron Haire headed the minnows in front before Ports forward Mark McAllister was red-carded after an off-the-ball incident on 32 minutes.
Shea Conaty's superb strike made it 2-0 but the hosts fought back with goals from Marcio Soares and Sean Mackle.
Fitzpatrick netted the winner to set up a semi-final against Linfield.
Ronnie McFall resigned as Portadown manager just minutes after the final whistle, bringing his almost 30-year reign at Shamrock park to an end.
Haire and Conaty came back to haunt their former club with the former Ports pair scoring the opening two goals.
It was a spirited fightback from the Premiership side with Soares firing into the bottom corner before Mackle nodded in from close range to equalise.
Despite being a player down Portadown were now the dominant side but Celtic had the final say thanks to Fitzpatrick's spot-kick after Chris Casement tripped Haire.
It will be Lurgan Celtic's first semi-final appearance and they will take on the Blues at Windsor Park on Friday, 1 April.
Lurgan Celtic boss Colin Malone said: "It's unbelievable - it was a real rollercoaster ride and we're delighted.
"I'm so proud of the lads and delighted for everyone involved with the club.
"We look forward to facing Linfield and we'll go up there and give it our best."
Raymond Fitzpatrick's injury-time penalty gave Championship side Lurgan Celtic a shock Irish Cup quarter-final win over 10-man Portadown.