One woman remembered for accusing him of breach of promise shot and wounded him in 1883 with a pistol inside his luxury Baldwin Hotel, built in 1876 on the northeast corner of Powell and Market St. The woman's name was Fannie Verona Baldwin, a native of Olympia in what was then Washington Territory. Verona herself stated it was not a case of seduction, however, but of rape. He also narrowly escaped death in a San Francisco courtroom on July 2, 1896. He was sued by Lillian Ashley for seduction. While she was on the witness stand, her sister Emma Ashley, walked up behind Baldwin and fired a pistol at him, grazing his skull.