In 1961, Kennedy appointed him as a Special Assistant to the President for Civil Rights. In the White House, he served as chairman of the Subcabinet Group on Civil Rights. Wofford was instrumental in the formation of the Peace Corps and served as the Peace Corps' special representative to Africa and director of operations in Ethiopia. He was appointed associate director of the Peace Corps in 1964 and held that position until 1966. He also participated in the Selma to Montgomery marches in 1965. Wofford's book "Of Kennedys and Kings: Making Sense of the Sixties" details his years in the civil rights movement and the creation of the Peace Corps.