Tang's wife Frances Young Tang was a philanthropist who specialized in landmark preservation. In 1992, she died at the age of 53 from cancer. A gallery at the Metropolitan Museum of Art is named after her, as is the Tang Art Museum at Skidmore College, her alma mater. She was a stepdaughter of the Chinese diplomat Wellington Koo. In 2013 Tang married Dr. Agnes Hsu-Tang (née Hsin-Mei Hsu 徐心眉), a Penn-, Cambridge-, and Stanford-educated archaeologist and UNESCO advisor on the faculty at Columbia University. Hsu has been dubbed by the Chinese press as "China's Lara Croft" for her hosting of the award-winning documentary series "Mysteries of China". Chinese and Singaporean press has widely reported that Hsu is a descendant of the Ming dynasty Catholic imperial minister Xu Guangqi 徐光启 who was beatified by the Vatican on April 15, 2011 and regarded by many to be the founder of modern Shanghai. Xujiahui 徐家汇 (the District of the "Xu/Hsu Clan) is the historical center of Shanghai. Hsu's maternal great-grandfather was Ji Xiaolan 紀晓岚, the Qianlong Emperor's imperial minister of war and poet laureate. Tang's daughter Tracy Tang Limpe is a philanthropist and the chairman of the board at the Masters School in Dobbs Ferry, NY, her alma mater. Another daughter, Dana Tang, an architect, married Andrew Haid Darrell in 1998. Tang's brother was Jack Chi-chien Tang (唐驥千), a prominent business leader, educator, philanthropist in Hong Kong and one of the founders of the Asia Society Hong Kong Center in 1990. Tang's brother-in-law was the distinguished Princeton University professor, Dr. Wen Fong (方闻), who was attributed to have "helped to create and shape the academic field of East Asian art history as we know it today."