He was born on May 7, 1936 in Collinsville, Mississippi. His brother was born 2 years later. He first started singing with the gospel group, the Dixie Nightingales along with his brother. He then became a singer as part of the Motown stable in 1961. He was mostly on sessions but he also recorded singles for Miracle label. Ruffin was then drafted for national service. After he left the Army in 1964, Ruffin returned to Motown and was offered the opportunity to join the Temptations to replace Elbridge Bryant. They instead hired Ruffin's brother David after hearing him. Jimmy decided to continue his solo career and recorded for the Soul label, but he didn't have much success.