In1806, Key's sister, Anne Phoebe Charlton Key, married Roger B. Taney, who would later become Chief Justice of the United States. In 1846 one daughter, Alice, married U.S. Senator George H. Pendleton and another, Ellen Lloyd, married Simon F. Blunt. In1859, Key's son Philip Barton Key II, who also served as United States Attorney for the District of Columbia, was shot and killed by Daniel Sicklesa U.S.Representative from New York who would serve as a general in the American Civil Warafter he discovered that Philip Barton Key was having an affair with his wife. Sickles was acquitted in the first use of the temporary insanity defense. In1861, Key's grandson Francis Key Howard was imprisoned in Fort McHenry with the Mayor of Baltimore George William Brown and other locals deemed to be Confederate sympathizers.