In Robert Shelton's biography of Dylan, Shelton said he was told by Phil Spector that the inspiration for the song came when Spector and Dylan heard the Ray Charles song, "Let's Go Get Stoned" on a jukebox in Los Angeles. Spector said "they were surprised to hear a song that free, that explicit", referring to its chorus of "getting stoned" as an invitation to indulge in alcohol or narcotics. (This anecdote may be questioned, because the Ray Charles song was released in April 1966, after "Rainy Day Women" was recorded.) (Both The Coasters and Ronnie Milsap released versions in 1965.)