He played with the Newark Eagles (1938–42, 46-48), New York Giants (1949–55) and Chicago Cubs (1956). He grew up in New Jersey and was a standout football player at Lincoln University. Irvin left Lincoln to spend several seasons in Negro league baseball. His career was interrupted by military service from 1943 to 1945.