Goodman was the son of poor Jewish immigrants in Chicago, Illinois. They lived in Chicago's Maxwell Street neighborhood. He learned to play clarinet in a boys' band run by a charity. He became a strong clarinet player at an early age, and began playing professionally in bands while still wearing 'in short pants'.