Ashford died of a heart attack at age 65 in Marina del Rey, California. Upon his death, Bowie Kuhn issued a statement, saying, "As the first black umpire in the major leagues, his magnanimous nature was sternly tested, but he was unshaken and uncomplaining, remaining the colorful, lively personality he was all his life." At his funeral, Ashford was eulogized by Kuhn and former USC baseball coach Rod Dedeaux. Ashford was cremated, and his ashes were interred in Cooperstown, New York.