At age 9, Stiller made his acting debut as a guest on his mother's short-lived television series, Kate McShane. In the late 1970s, he performed with the New York City troupe NYC's First All Children's Theater, playing several roles, including the title role in Clever Jack and the Magic Beanstalk. After being inspired by the television show Second City Television while in high school, Stiller realized that he wanted to get involved with sketch comedy. During his high school years, he was also the drummer of the post-punk band Capital Punishment, which released the studio album Roadkill in 1982. The band's bassist, Peter Swann, went on to become (as of 2018) an Arizona Court of Appeals Judge. The band reunited in 2018 to release a new EP, titled This is Capital Punishment, for Record Store Day.