Media playback is not supported on this device
Declan O'Brien capitalised on a mistake by Loughgall keeper Gareth Buchanan to put Glenavon ahead before the break.
Loughgall defender Richard Copeland was dismissed for a last-man foul on 52 minutes and Andy McGrory made it 2-0 from a penalty after he was fouled.
Stuart Mallon smashed in for the visitors midway through the second half but Glenavon held firm to progress.
The Lurgan Blues will meet the Premiership leaders in a Windsor Park semi-final on Saturday, 2 April.
Glenavon goalscorer Andy McGrory said: "We had Loughgall watched so we knew it wasn't going to be easy.
"I didn't think we were very good - Loughgall took the game to us despite only having 10 players and at times we looked like the team with a man less.
"But we're thankful to make the semis and we're really looking forward to it.
"Crusaders are a great team, but we're also a great side - we will go to Windsor with no fear."
Glenavon edged Championship outfit Loughgall to earn an Irish Cup semi-final clash against Crusaders.