In 1963, Dr. Blayton's son, Oscar H. Blayton, became the first African American to attend the College of William and Mary as an undergraduate. He graduated from Yale Law School in 1977, and opened a law practice in Williamsburg. Dr. Blaytons's Daughter Betty Blayton-Taylor was a well known and influential African American artist and educator in New York City. His other children include Barbara Richardson and James Blayton Jr. His grandchildren include Lisa Jackson, David Richardson, Shaun Blayton, and Omar Blayton. His great-grandchildren include and Coleman and Cleo Blaine Jackson, Bocar, Brandon, and Miles Richardson,