Storrow met Helen Osborne, daughter of a prominent and activist upstate New York family (whose brother also attended Harvard), while attempting to scale the Matterhorn in Switzerland.They married and had only one son, James Jackson Storrow III (1892-1977), although they had hoped for a large family.