clayton , who had removed to his townhouse in upper brook street to be near whitney during the weeks preceding the wedding , read her brief note declining the rutherfords ' party with a faint frown of bewilderment .