from his first short visit , before the wreck of the savannah , he knew that the townspeople , since the mission village was mostly deserted , were taking the adobe brick from the wall for building projects of their own .