Radziwill was born Caroline Lee Bouvier, in Southampton, Long Island, New York. She was the second daughter of wealthy stockbroker John Vernou Bouvier III and Janet Norton Lee. Her older sister was Jacqueline Lee Bouvier (1929-1994), future First Lady of the United States. She also had a step-sister and a step-brother, Janet (1945-1985) and James Auchincloss (b. 1947). They were the children of her mother by her second husband Hugh D. Auchincloss.