Pepper was born to upper-class parents, physician (and former Union cavalry officer) George Pepper and his wife, the former Mehitable ("Hitty") Markoe Wharton, on March 16, 1867. Each was descended from families prominent in the region since the colonial era: Pennsylvania Dutch on his father's side and Quakers and Episcopalians on his mother's. He was born in his paternal grandmother's house, in a fashionable neighborhood, 1215 Walnut Street. Their first child had died in infancy, and the family soon moved to quarters on Pine Street. Dr. Wharton died when George was seven and his only sister Frances a newborn, so the family moved to smaller quarters, 346 S. 16th Street in the Rittenhouse Square neighborhood with his grandmother. His mother home-schooled her then weak-eyed son, with the assistance of his uncle, Dr. William Pepper (whom he considered like a father after his own father's demise), and later a blind tutor, John F. Maher. His grandmother's summer estate up the Schuylkill River later became one of those consolidated into Laurel Hill Cemetery. He considered Willie Ryder his best friend, noting that he was "colored"; and also remembered informally competing at memory exercises with boys from a nearby prep school. In 1876, his mother remarried, to his father's friend and former classmate, lawyer Ernest Zantzinger.