In this task, you will be presented with a context from an academic paper and you have to write an answerable question based on the context. Your questions can be extractive, abstractive, or yes-no questions.

A maximum of three contributors will listen to any audio clip. If an $<$audio,transcript$>$ pair first receives two up-votes, then the clip is marked as valid. If instead the clip first receives two down-votes, then it is marked as invalid. A contributor may switch between recording and validation as they wish. Only clips marked as valid are included in the official training, development, and testing sets for each language. Clips which did not recieve enough votes to be validated or invalidated by the time of release are released as “other”. The train, test, and development sets are bucketed such that any given speaker may appear in only one. This ensures that contributors seen at train time are not seen at test time, which would skew results. Additionally, repetitions of text sentences are removed from the train, test, and development sets of the corpus.
How is validation of the data performed?

To account for this, we compute the balanced accuracy, i.e., the average of the three accuracies on each class.
Is model explanation output evaluated, what metric was used?

For evaluation metrics, we used BLEU BIBREF8 and RIBES BIBREF9 to measure translation accuracy, and token-level delay to measure latency. We used Kytea BIBREF10 as a tokenize method for evaluations of Japanese translation accuracy.

Which metrics do they use to evaluate simultaneous translation?

