Carver was born in Clatskanie, Oregon, a mill town on the Columbia River, and grew up in Yakima, Washington. His father was a skilled sawmill worker from Arkansas. He was a fisherman and a heavy drinker. Carver's mother sometimes worked as a waitress and as a clerk in a shop. He had one brother, James Franklin Carver, was born in 1943.