1) The main weakness for me is the statement of the specific hypothesis, within the general research line, that the paper is probing: I found it very confusing. As a result, it is also hard to make sense of the kind of feedback that the results give to the initial hypothesis, especially because there are a lot of them and they don't all point in the same direction.
The paper says: "This paper pursues the hypothesis that an accurate model of referential word meaning does not need to fully integrate visual and lexical knowledge (e.g. as expressed in a distributional vector space), but at the same time, has to go beyond treating words as independent labels."
The first part of the hypothesis I don't understand: What is it to fully integrate (or not to fully integrate) visual and lexical knowledge? Is the goal simply to show that using generic distributional representation yields worse results than using specific, word-adapted classifiers trained on the dataset?
If so, then the authors should explicitly discuss the bounds of what they are showing: Specifically, word classifiers must be trained on the dataset itself and only word classifiers with a sufficient amount of items in the dataset can be obtained, whereas word vectors are available for many other words and are obtained from an independent source (even if the cross-modal mapping itself is trained on the dataset); moreover, they use the simplest Ridge Regression, instead of the best method from Lazaridou et al. 2014, so any conclusion as to which method is better should be taken with a grain of salt. However, I'm hoping that the research goal is both more constructive and broader. Please clarify. 2) The paper uses three previously developed methods on a previously available dataset. The problem itself has been defined before (in Schlangen et al.). In this sense, the originality of the paper is not high. 3) As the paper itself also points out, the authors select a very limited subset of the ReferIt dataset, with quite a small vocabulary (159 words). I'm not even sure why they limited it this way (see detailed comments below).
4) Some aspects could have been clearer (see detailed comments).
5) The paper contains many empirical results and analyses, and it makes a concerted effort to put them together; but I still found it difficult to get the whole picture: What is it exactly that the experiments in the paper tell us about the underlying research question in general, and the specific hypothesis tested in particular? How do the different pieces of the puzzle that they present fit together?
- General Discussion: [Added after author response] Despite the weaknesses, I find the topic of the paper very relevant and also novel enough, with an interesting use of current techniques to address an "old" problem, REG and reference more generally, in a way that allows aspects to be explored that have not received enough attention. The experiments and analyses are a substantial contribution, even though, as mentioned above, I'd like the paper to present a more coherent overall picture of how the many experiments and analyses fit together and address the question pursued.
- Detailed comments: Section 2 is missing the following work in computational semantic approaches to reference: Abhijeet Gupta, Gemma Boleda, Marco Baroni, and Sebastian Pado. 2015. Distributional vectors encode referential attributes.
Proceedings of EMNLP, 12-21 Aurelie Herbelot and Eva Maria Vecchi. 2015.
Building a shared world: mapping distributional to model-theoretic semantic spaces. Proceedings of EMNLP, 22–32.
142 how does Roy's work go beyond early REG work?
155 focusses links 184 flat "hit @k metric": "flat"?
Section 3: please put the numbers related to the dataset in a table, specifying the image regions, number of REs, overall number of words, and number of object names in the original ReferIt dataset and in the version you use. By the way, will you release your data? I put a "3" for data because in the reviewing form you marked "Yes" for data, but I can't find the information in the paper.
229 "cannot be considered to be names" ==> "image object names" 230 what is "the semantically annotated portion" of ReferIt?
247 why don't you just keep "girl" in this example, and more generally the head nouns of non-relational REs? More generally, could you motivate your choices a bit more so we understand why you ended up with such a restricted subset of ReferIt?
258 which 7 features? ( list) How did you extract them?
383 "suggest that lexical or at least distributional knowledge is detrimental when learning what a word refers to in the world": How does this follow from the results of Frome et al. 2013 and Norouzi et al. 2013? Why should cross-modal projection give better results? It's a very different type of task/setup than object labeling.
394-395 these numbers belong in the data section Table 1: Are the differences between the methods statistically significant?
They are really numerically so small that any other conclusion to "the methods perform similarly" seems unwarranted to me. Especially the "This suggests..." part (407). Table 1: Also, the sim-wap method has the highest accuracy for hit @5 (almost identical to wac); this is counter-intuitive given the @1 and @2 results. Any idea of what's going on?
Section 5.2: Why did you define your ensemble classifier by hand instead of learning it? Also, your method amounts to majority voting, right? Table 2: the order of the models is not the same as in the other tables + text.
Table 3: you report cosine distances but discuss the results in terms of similarity. It would be clearer (and more in accordance with standard practice in CL imo) if you reported cosine similarities.
Table 3: you don't comment on the results reported in the right columns. I found it very curious that the gold-top k data similarities are higher for transfer+sim-wap, whereas the results on the task are the same. I think that you could squeeze more information wrt the phenomenon and the models out of these results.
496 format of "wac" Section 6 I like the idea of the task a lot, but I was very confused as to how you did and why: I don't understand lines 550-553. What is the task exactly? An example would help. 558 "Testsets" 574ff Why not mix in the train set examples with hypernyms and non-hypernyms?
697 "more even": more wrt what?
774ff "Previous cross-modal mapping models ... force...": I don't understand this claim.
792 "larger test sets": I think that you could even exploit ReferIt more (using more of its data) before moving on to other datasets.
5) The paper contains many empirical results and analyses, and it makes a concerted effort to put them together; but I still found it difficult to get the whole picture: What is it exactly that the experiments in the paper tell us about the underlying research question in general, and the specific hypothesis tested in particular? How do the different pieces of the puzzle that they present fit together?