- One main weakness is in the baselines. There is no comparison to regular finetuned baselines for the ASR and spoken translation (ST) tasks, that make use of the 15 hours of transcribed and translated speech. While the main objective of this work is to promote multimodal in-context learning with text and speech, it would still be useful to see how existing pretrained ASR/ST models fare after finetuning with labeled Kalamang speech.
- For the human baseline, the human only closely follows a little more than 1 hour of speech recordings rather than the full 15 hours. This makes the human baseline considerably weaker than the model baseline, apart from all the other factors mentioned in Section 4.1 that make the human baseline weaker. In the abstract, the authors mention "already beating the 34.2% CER and 4.51 BLEU achieved by a human who learned Kalamang from the same resources" which is a bit misleading given the 1 hour vs. 15 hour disparity. The authors should consider softening this claim.
- For the human baseline, the human only closely follows a little more than 1 hour of speech recordings rather than the full 15 hours. This makes the human baseline considerably weaker than the model baseline, apart from all the other factors mentioned in Section 4.1 that make the human baseline weaker. In the abstract, the authors mention "already beating the 34.2% CER and 4.51 BLEU achieved by a human who learned Kalamang from the same resources" which is a bit misleading given the 1 hour vs.