Milhaud's family were Jewish. He grew up in Aix-en-Provence. His father was an almond dealer. When he was very small he often listened to the songs that were being sung by the women who were sorting out the nuts. They were folksongs from the Provence. When he was a young man he liked to go for walks in the Provence country. This was an important part of his life.