He was the eldest son of Francis Yelverton of Kanturk, County Cork, and Elizabeth Barry, daughter of Jonas Barry of Kilbrin (now Ballyclogh, County Cork) His father died when Barry was only ten; his mother reached a great age, dying only a year before her son. He went to school in Charleville and Midleton College, and attended Trinity College Dublin, where he took a degree of Bachelor of Arts in 1757 and of Bachelor of Laws in 1761. His family lacked wealth and social position and he was for some years an assistant master under Andrew Buck in the Hibernian Academy. This menial occupation was later a source of great embarrassment to him, as his enemies loved to ridicule him as "Buck's usher".