Chang met his wife, American-born Chinese Isabelle Chin Chang, in the library at Yale University, shortly after he moved to the United States. Chin assumed the role of the housewife in the pair's marriage, allowing Chang to delve into his work without domestic concerns. They have two daughters and a son together – Claudia Chang Tourtellotte, head of the anthropology department at Sweet Briar College; Pamela O'Malley Chang, an architect, civil engineer, and sustainable design consultant and Francis Hugh Chang, director of health centers in Boston, Massachusetts. and San Jose, California.