Robards was born in New York City, the son of actor Jason Robards, Jr. (1922–2000) and actress Lauren Bacall (1924–2014). He is the only child from their marriage, though he has seven-half siblings; five through his father (three elder, two younger), and through his mother's marriage to Humphrey Bogart, half-siblings Stephen Humphrey and Leslie Bogart. Robards was seven when his parents divorced, which Bacall later blamed on the elder Robards' alcoholism. Robards recalled he was devastated by the divorce, and said he was raised "basically alone." Afterwards, he resided with his mother in New York.