Pillsbury was born December 3, 1842 in Warner, New Hampshire, first of three children born to George Alfred Pillsbury and Margaret Sprague Carleton. His sister Mary A. died in infancy. His brother Frederick Carleton Pillsbury was born in 1852. Pillsbury had a modest upbringing. He graduated from Dartmouth College in 1863, paying for his college education by teaching part-time. He then worked for six years as a clerk and partner in Montreal, Quebec, Canada at a mercantile enterprise. During this time, he met his future wife, Mary Ann Stinson, daughter of Capt. Charles Stinson and his wife Mary Ann Poore. The two married on September 12, 1866.