McDermott and his sister were raised by their maternal grandmother Avis in Waterbury. As a teenager, he began taking trips to visit his biological father, who owned the West Fourth Street Saloon in Greenwich Village, New York. The two would go to the movies and the younger McDermott would work in his father's bar, serving drinks and breaking up fights. He would also fast-talk his way into the Mudd Club and Studio 54. McDermott was uncomfortable with himself as a teenager, saying he had a "Dorothy Hamill hairdo." He began to imitate his acting heroes, such as Marlon Brando and Humphrey Bogart, to adopt their demeanor. In 1979, McDermott graduated from Holy Cross High School in Waterbury.