Silver was born on September 2, 1928, in Norwalk, Connecticut. His mother, Gertrude, was from Connecticut; his father, John Tavares Silver, was born on the island of Maio, Cape Verde, and emigrated to the United States as a young man. She was a maid and sang in a church choir; he worked for a tire company. Horace had a much older half-brother, Eugene Fletcher, from his mother's first marriage, and was the third child for his parents, after John, who lived to 6 months, and Maria, who was stillborn.