stephen was to join them there later , and the foursome would then proceed to the rutherfords ' ball , where whitney , clayton , and the dowager duchess would lend their protection and influence to ensure that nothing went wrong during the season 's most important opening ball .