Clay was born in Paris, the fourth of six children of James Clay (1804–1873) and his wife, Eliza Camilla, "née" Woolrych. James Clay was a Radical Member of Parliament and was also well known as a player of and authority on whist. Both parents were musical: Clay's mother was the daughter of a leading opera singer, and his father was an amateur composer. Clay was educated at home in London by private tutors, and studied piano and violin, and later composition under Bernhard Molique. Through the influence of Lord Palmerston, Clay secured a post in HM Treasury, and was for a time private secretary to Benjamin Disraeli, who presented him at a court levee in 1859. Under a later administration Clay undertook confidential missions on behalf of W.E.Gladstone.