Elfman was born in Los Angeles, California, to a Jewish family of Polish-Jewish and Russian-Jewish descent. He is the son of Blossom Elfman (née Bernstein), a writer and teacher, and Milton Elfman, a teacher who was in the Air Force, and the brother of Richard Elfman. Elfman was raised in a racially mixed affluent community in Baldwin Hills, California, where he spent much of his time at the local movie theater discovering classic sci-fi, fantasy and horror films and first noticed the music of such film composers as Bernard Herrmann and Franz Waxman.