The title stems from an identically titled track by Junior Wells and Earl Hooker, with the rhythmic similarity to "Shame, Shame, Shame" by Jimmy Reed, a song which was released the previous year. Reed was not credited for the song, however, Phil Spector was given co-credit with "Nanker Phelge" (a pseudonym for songs credited to the whole group).
"Little by Little" is a song by the Rolling Stones recorded on 4 February 1964. Decca Records released it as the B-side to their version of "Not Fade Away" on 21 February 1964.