But the Apostles, who knew Jesus best, did set up a Church.  

Your rejection of "Upon this rock . . ." is a rejection of Christ's founding of the Papacy, and hence a denial of the legitimacy of the Church itself.