"Surf's Up" was recorded between January and July 1971, with the exception of a few tracks. After the release of "Sunflower", band engineer Stephen Desper assembled a collection of songs consisting mostly of outtakes deemed suitable for a follow-up LP, which he labelled "Second Brother Album". Rieley later called these selections "forgettable" and said that he was "totally perplexed ... I met with executive Mo Ostin, a true Brian Wilson fan, at Warner Brothers, who listened to the songs, and he declared: 'No way.'"