sherman island was almost as close to the city of san quentin , where one of the most famous prisons in the world existed in shocking contrast to the picturesque shores of san francisco bay and the affluent area surrounding it .