Lord Dudley was born in London, Middlesex. His father was landowner and cricketer William Ward, 1st Earl of Dudley. His mother was Scottish noblewoman Georgina Ward, Countess of Dudley. He was married to Rachel Gurney until her death by drowning in 1920. The couple had seven children. In 1924, he married actress Gertie Millar.