he picked the place because the stately structure reminded him of the gatehouse to his family 's estate , where he 'd often found refuge with the gardeners and maids when the storms grew too fierce at the main house .