I think by "we all," he meant the American people, not each and every one of us.  In fact, I'm sure he knows that the Trump did not even win the popular vote.  Some, by their failure to vote, allowed him to get in.