Anderson has been married twice. He met his first wife, Mihoko "Mickey" Anderson, while he was a Marine stationed with the Armed Forces Radio and Television Service in Japan. They had one daughter, Gabrielle Anderson (born 1976). They later divorced. In 1982, he married a Lebanese native from a Maronite Christian family, Madeleine Bassil; they had one daughter, Sulome Anderson, born in 1985, three months after he was taken hostage, who later became a freelance journalist based in New York City and Beirut. She gained publicity for a photo depicting her kissing her formerly Orthodox Jewish boyfriend with a placard stating "Jews and Arabs REFUSE to be ENEMIES."