since they all lived in the big white houses down along the harbor side , the wind suited them when it wasnt blowing out over their houses and wasnt blowing out over terments mansion on the island either .