Poirot had been asked by John to remain to lunch, and was already seated at the table. 
Poirot was sitting at the table because John had asked him to lunch.
Poirot had left the diner before John could say anything.