Alpert was born into a family of musicians. His father, although a tailor by trade, was also a talented mandolin player. His mother taught violin at a young age, and his older brother, David, was a talented young drummer. Herb began trumpet lessons at the age of eight and played at dances as a teenager. Acquiring an early wire recorder in high school, he experimented on this crude equipment. After graduating from Fairfax High School in 1952, he joined the United States Army and frequently performed at military ceremonies. After his service in the Army, Alpert tried his hand at acting, but eventually settled on pursuing a career in music.