While at Hall-Place, Smart was sent to the local Maidstone Grammar School where he was taught by Charles Walwyn, a scholar from Eton College who had received an MA from King's College, Cambridge in 1696. It was here that Smart received an intensive education in Latin and Greek. He did not complete his education at Maidstone however, as his father died on 3 February 1733, and his mother took Smart and his siblings to live near relatives in Durham after selling off a large portion of the estate to pay off Peter Smart's debts.