Costa was born in Boston, Massachusetts to an Italian American family. As a child, he took a keen interest in learning the guitar, and he became a member of the CBS Radio Orchestra by the time he was in his teens. In the late 1940s, Costa moved to New York City to further his career by becoming a session musician. He played guitar with Bucky Pizzarelli on Vaughn Monroe's hit record "". It was around this time that Costa started experimenting with combinations of instruments, producing musical arrangements, and selling them to big bands.