Joel married a beautiful young actress named Ellen "Nellie" Ridley. While highly successful in business, in his personal life familial relationships were not always cordial. His dislike of his daughter Doris' choice of spouse continued until she divorced after four years, at which point he resumed normal relations. Joel also disapproved of one of his sons, Stanhope's, marriage for two years. His daughter Eileen became the first woman jockey to win an open race when she rode Hogier to victory in the Town Plate at Newmarket. His son Dudley Joel was elected the Conservative Party Member of Parliament for Dudley but was killed in action during World War II.