the town flanking the edges of the church was mostly clapboard shacks filled with masons and lumberjacks , and although many of the tree-killers filtered away when the logging business failed , the ones who stayed behind proved the worst of the lot .