Goldsmith was born in London and is the youngest child of the late billionaire James Goldsmith, a member of the prominent Jewish Goldsmith family, and his third wife Lady Annabel Vane-Tempest-Stewart. He has an older sister, Jemima Goldsmith; an older brother, Zac Goldsmith; and several half-siblings. Zac has had a strong influence on him, and has a passion for the environment inherited from their father, who, towards the end of his life, was one of Europe's most prominent founders of green causes, including campaigns against genetically modified food. His uncle Teddy Goldsmith was a co-founder of the Green Party UK and also of "The Ecologist".