- My main criticism is that the "mismatched" image caption dataset is artificial and may not capture the kind of misinformation that is posted on platforms like Twitter. For instance, someone posting a fake image of a lockdown at a particular place may not just be about a mismatch between the image and the caption, but may rather require fact-checking, etc. Moreover, the in-the-wild datasets on which a complementary evaluation is conducted are also more about mismatched image-caption and not the real misinformation (lines 142-143). Therefore, the extent to which this dataset can be used for misinformation detection is limited. I would have liked to see this distinction between misinformation and mismatched image captions being clear in the paper.
- Also, since the dataset is artificially created, the dataset itself might have a lot of noise. For instance, the collected "pristine" set of tweets may not be pristine enough and might instead contain misinformation as well as out-of-context images. I would have liked to see more analysis around the quality of the collected dataset and the amount of noise it potentially has. - Since this is a new dataset, I would have liked to see evaluation of more models (other than just CLIP). But given that it is only a short paper, it is probably not critical (the paper makes enough contributions otherwise)
- Table 4 and Table 5: Are the differences statistically significant? ( especially important because the hNews and Twitter datasets are really small) - Lines 229-240: Are the differences between the topics statistically significant?
- Also, since the dataset is artificially created, the dataset itself might have a lot of noise. For instance, the collected "pristine" set of tweets may not be pristine enough and might instead contain misinformation as well as out-of-context images. I would have liked to see more analysis around the quality of the collected dataset and the amount of noise it potentially has.