Smart then attended Durham School, where the Reverend Mr. Richard Dongworth was headmaster; it is not known whether he lived with his uncle, John Smart, or with a school master. He spent vacations at Raby Castle, which was owned by Henry Vane, 1st Earl of Darlington, the grandson of Christopher Vane. Henry Vane and his wife Grace, sister to William and Henrietta Fitzroy the Duke and Duchess of Cleveland, had four children, Henry, Frederick, Anne, and Mary. They were only a few years younger than Smart and became playmates, with Anne and Henry "pairing off" with Christopher and his sister Margaret respectively. Although nothing resulted from the match, Anne has been traditionally described as being his "first love". During his time with the Vane family, Smart dedicated many poems to Henrietta, the Duchess of Cleveland. It was his closeness with the Vane family along with his skill for learning that encouraged Henrietta to allow him a pension of 40 pounds yearly, continued by her husband after her death in 1742. This allowed Smart to attend Pembroke College, Cambridge.