Richard Graves was born at his father's rectory in Kilfannan, near Kilmallock, Co. Limerick, 1 October 1763, the youngest son of Rev. James Graves (1710–1783), "an accurate and well-read scholar and to a mind imbued with classical tastes and acquirements... a conversation enlivened by a natural vivacity and a pointed but inoffensive wit (who) added so much Christian affability and kindness as to render him a general favourite in his own rank of life, and (procuring) him the esteem and affection of his parishioners of every denomination (and whose) society was much prized and proportionately sought for". James Graves had one brother, Richard, High Sheriff of Counties Limerick and Waterford; and one sister, Abigail, who married firstly Edward Southwell (1703-1736), son William Southwell; and, after his death a grandson of Sir William Scroggs.