Harley was born in Bow Street, London, in 1661, the eldest son of Sir Edward Harley, a prominent landowner in Herefordshire and his wife Abigail Stephens and the grandson of Sir Robert Harley and his third wife, the celebrated letter-writer Brilliana, Lady Harley. He was educated at Shilton, near Burford, in Oxfordshire, in a small school which produced at the same time a Lord High Treasurer (Harley himself), a Lord High Chancellor (Lord Harcourt) and a Lord Chief Justice of the Common Pleas (Lord Trevor). Harley then spent some time at Foubert's Academy, but disliked it. He entered the Inner Temple on 18 March 1682, but was never called to the bar.