Learning When to Speak: Latency and Quality Trade-offs for Simultaneous Speech-to-Speech Translation with Offline Models

Recent work in <PRE_TAG>speech-to-speech translation (S2ST)</POST_TAG> has focused primarily on <PRE_TAG>offline settings</POST_TAG>, where the <PRE_TAG>full input utterance</POST_TAG> is available before any output is given. This, however, is not reasonable in many real-world scenarios. In latency-sensitive applications, rather than waiting for the full utterance, translations should be spoken as soon as the information in the input is present. In this work, we introduce a system for <PRE_TAG>simultaneous S2ST</POST_TAG> targeting real-world use cases. Our system supports translation from 57 languages to English with tunable parameters for dynamically adjusting the latency of the output -- including four policies for determining when to speak an output sequence. We show that these policies achieve <PRE_TAG>offline-level accuracy</POST_TAG> with minimal increases in latency over a Greedy (wait-k) baseline. We open-source our evaluation code and interactive test script to aid future SimulS2ST research and application development.