Molecular polaritons are hybrid light-matter states that emerge when a molecular transition strongly interacts with photons in a resonator. At optical frequencies, this interaction unlocks a way to explore and control new chemical phenomena at the nanoscale. Achieving such control at ultrafast timescales, however, is an outstanding challenge, as it requires a deep understanding of the dynamics of the collectively coupled molecular excitation and the light modes. Here, we investigate the dynamics of collective polariton states, realized by coupling molecular photoswitches to optically anisotropic plasmonic nanoantennas. Pump-probe experiments reveal an ultrafast collapse of polaritons to pure molecular transition triggered by femtosecond-pulse excitation at room temperature. Through a synergistic combination of experiments and quantum mechanical modelling, we show that the response of the system is governed by intramolecular dynamics, occurring one order of magnitude faster with respect to the uncoupled excited molecule relaxation to the ground state.