In 1971, Olmos married Katija Keel. She is the daughter of actor Howard Keel. They had two children but divorced in 1992 and later married actress Lorraine Bracco in 1994. However, they filed for divorce in January 2002. He has four adopted children. In the same year of 1994, he married Puerto Rican actress Lymari Nadal. 