Silver graduated from St. Mary's Grammar School in 1943. From ninth grade he played Lester Young-influenced tenor saxophone in the Norwalk High School band and orchestra. Silver played gigs locally on both piano and tenor saxophone while still at school. He was rejected for military service by a draft board examination that concluded that he had an excessively curved spine, which also interfered with his saxophone playing. Around 1946 he moved to Hartford, Connecticut to take up a regular job as pianist in a nightclub.