1. The paper raises two hypotheses in lines 078-086 about multilinguality and country/language-specific bias. While I don't think the hypotheses are phrased optimally (could they be tested as given?), their underlying ideas are valuable. However, the paper actually does not really study these hypotheses (nor are they even mentioned/discussed again). I found this not only misleading, but I would have also liked the paper to go deeper into the respective topics, at least to some extent. 2. It seemed a little disappointing to me that the 212 new pairs have _not_ been translated to English (if I'm not mistaken). To really make this dataset a bilingual resource, it would be good to have all pairs in both languages. In the given way, it seems that ultimately only the French version was of interest to the study - unlike it is claimed initially.
3. Almost no information about the reliability of the translations and the annotations is given (except for the result of the translation checking in line 285), which seems unsatisfying to me. To assess the translations, more information about the language/translation expertise of the authors would be helpful (I don't think this violates anonymity). For the annotations, I would expect some measure of inter-annotator agreement.
4. The metrics in Tables 4 and 5 need explanation, in order to make the paper self-contained. Without going to the original paper on CrowS-pairs, the values are barely understandable. Also, information on the values ranges should be given as well as whether higher or lower values are better.
- 066: social contexts >> I find this term misleading here, since the text seems to be about countries/language regions.
- 121: Deviding 1508 into 16*90 = 1440 cases cannot be fully correct. What about the remaining 68 cases?
- 241: It would also be good to state the maximum number of tasks done by any annotator.
- Table 3: Right-align the numeric columns.
- Table 4 (1): Always use the same number of decimal places, for example 61.90 instead of 61.9 to match the other values. This would increase readability. - Table 4 (2): The table exceeds the page width; that needs to be fixed.
- Tables 4+5 (1): While I undersand the layout problem, the different approaches would be much easier to compare if tables and columns were flipped (usually, one approach per row, one metric per column). - Tables 4+5 (2): What's the idea of showing the run-time? I didn't see for what this is helpful.
- 305/310: Marie/Mary >> I think these should be written the same.
- 357: The text speaks of "53", but I believe the value "52.9" from Table 4 is meant. In my view, such rounding makes understanding harder rather than helping.
- 575/577: "1/" and "2/" >> Maybe better use "(1)" and "(2)"; confused me first.
1. The paper raises two hypotheses in lines 078-086 about multilinguality and country/language-specific bias. While I don't think the hypotheses are phrased optimally (could they be tested as given?), their underlying ideas are valuable. However, the paper actually does not really study these hypotheses (nor are they even mentioned/discussed again). I found this not only misleading, but I would have also liked the paper to go deeper into the respective topics, at least to some extent.