The house was inherited by D'Arcy's son and Toronto mayor, William Henry Boulton. When he died in 1874, the house passed to his widow, Hariette Boulton. She remarried the prominent scholar Goldwin Smith, and the couple lived in The Grange for the rest of their lives. Upon Goldwin's Smith's death in 1910, the couple bequeathed the building to the Art Museum of Toronto (now known as the Art Gallery of Ontario), and The Grange became the new home of the gallery. The building also served as the first home of the Ontario College of Art and Design, now OCAD University. Since the early 20th century, the Art Gallery of Ontario has been expanded a number of times, and the original manor makes up only a small part of the structure.