In 1841, Frank Brinkley was born at Parsonstown House, Co. Meath, the thirteenth and youngest child of Matthew Brinkley (1797–1855) J.P., of Parsonstown and his wife Harriet Graves (1800–1855). His paternal grandfather, John Brinkley, was the last Bishop of Cloyne and the first Royal Astronomer of Ireland, while his maternal grandfather, Richard Graves, was also a Senior Fellow of Trinity College and the Dean of Ardagh. One of Brinkley's sisters, Jane (Brinkley) Vernon of Clontarf Castle, was the grandmother of Cyril Connolly. Another sister, Anna, became the Dowager Countess of Kingston after the death of her first husband, James King, 5th Earl of Kingston and was the last person to live at Mitchelstown Castle. Through his mother's family Brinkley was related to Richard Francis Burton, a distinguished linguist who shared Brinkley's passion for foreign culture.