There is a theory that this term originated from a condition that even regular customers looked like they were running out of patience when leaving a restaurant after paying a bill.
常連でも勘定を払うと愛想をつかしたように帰っていくさまから、という説がある。