But Tunisian giants Esperance are out, after a 1-1 draw at home with continental debutants Mouloudia Bejaia.
Mazembe went into the second leg with a 1-0 lead but fell to a 2-1 defeat in Gabes, just avoiding a major upset.
Gabesien made it 1-0 on the day, and levelled the tie, when Youssef Fouzai scored a 38th-minute penalty.
Jonathan Bolingi's equaliser put Mazembe back in charge - 2-1 on aggregate - and although Ahmed Hosni restored Gabesien's second-leg lead, and Bolingi was sent off for violent play, the hosts failed to take further chances to score that would have made them victorious.
Mazembe had dropped into the second-tier continental club tournament after losing in the final qualifying round of the Champions League by Morocco's Wydad Casablanca.
And they had been considered heavy favourites to beat Gabesien, whoonly qualified because Tunisian Cup winners Etoile du Sahel secured a Champions League slot.
However, the signs were evident in the first leg, which was Gabesien's first outing in African club competition, that five-time African champions were facing a serious challenge.
They struggled to convert their chances at home, eventually scoring four minutes into injury time at the end of the first leg through Bolingi.
And they almost paid for their profligacy, escaping only because Gabesien were also guilty of missing chances at home on Tuesday.In Tuesday's other game Esperance, winners of every Caf club tournament except the Confederation Cup, exited the competition at the hands of novices Mouloudia Bejaia of Algeria.
A 1-1 draw in Rades coupled with a goalless draw in the first leg, puts the Algerians through on away goals.
Esperance took the lead through Ghailane Chaalali after 30 minutes, having earlier missed a penalty.
But Bejaia's Mohamed Ndoye equalised for the visitors just four minutes later and they held on for the remainder of the match to go through to the group phase - the draw for which is in Cairo on 24 May.
TP Mazembe edged into the Confederation Cup group stage by beating Stade Gabesien on the away goals rule after a 2-2 aggregate draw.