Fahey was born into a musical household in Washington, D.C. in 1939. Both his father, Aloysius John Fahey, and his mother, Jane (née Cooper), played the piano. In 1945, the family moved to the Washington suburb of Takoma Park, Maryland, where his father lived until his death in 1994. On weekends, the family attended performances of the top country and bluegrass acts of the day, but it was hearing Bill Monroe's version of Jimmie Rodgers' "Blue Yodel No. 7" on the radio that ignited the young Fahey's passion for music.