I would think that if everyone in the town wanted you out and your wife and kids already left then you should probably start thinking about making an exit.  The story about wanting to make sure the place is clean is most likely a crock.