Stillman was born in 1952 in Washington, D.C., to Margaret Drinker (née Riley), from Philadelphia, Pennsylvania, and a Democratic politician, John Sterling Stillman, an assistant secretary of commerce under President John F. Kennedy (a classmate of Stillman's father at Harvard), from Washington, D.C. His great-grandfather was businessman James Stillman; his great-great-grandfather, Charles Stillman, founded Brownsville, Texas. Stillman grew up in Cornwall, New York, and experienced depression during puberty. "I was very depressed when I was 11 or 12," he told "The Wall Street Journal". "I was sent to the leading Freudian child psychologist in Washington, D.C. It was heck. The last thing I needed to talk about was guilt about sex." However, when his parents separated, he found that his depression ceased: "I actually felt healthier."