Seabury died in 1865 and the remaining family lived at the home into old age. Gertrude, the youngest member of the immediate family, lived here alone for 24 years after the death of her sister Julia in 1909. As she grew older and more eccentric she became obsessed with holding on to the elegant home in a neighborhood that had become, by the early 20th century, a run-down, semi-industrial, and disreputable part of town. Burdened with severe financial hardship in her last years, she somehow managed to keep the beautiful home in nearly original condition, long after all the neighboring private homes had been demolished or converted into rooming houses, tenements, or commercial structures.