When his father, James Samuel Harris Sr. was shot while gambling and his mother died of grieving, Harris was orphaned at age 3. He lived with foster families until he was 11 or 12 when he and his sister went to live with his grandparents in Saint Paul, Minnesota. He has been married four times. One of his daughters died. He is the father of Jimmy Jam (James Harris III) who produced records for Janet Jackson, Mariah Carey and Usher.