George Miles was the son of the Rev. Robert Henry William Miles (1818–1883), rector of the Church of St. Mary and All Angels, Bingham, Nottinghamshire, and his wife Mary Ellen (née Cleaver). He was the grandson of Philip John Miles (1773–1845) by his second marriage to Clarissa Peach (1790–1868). Philip John Miles was an English landowner, banker, merchant, politician and collector, who was elected MP for Bristol from 1835–1837 having earlier been elected for Westbury from 1820–26 and Corfe Castle from 1829–1832. Frank Miles was therefore brother of Charles Oswald Miles, cousin of Philip Napier Miles and half-cousin of Sir Philip Miles, 2nd Baronet.