In this task, you are given a paper review. Based on the review, your job is to identify language and generate "en" if the review is in English or generate "es" if the review is in Spanish. Note that URLs in the text have been replaced with [Link].

The paper introduces a simple and effective method for morphological paradigm completion in low-resource settings. The method uses a character-based seq2seq model trained on a mix of examples in two languages: a resource-poor language and a closely-related resource-rich language; each training example is annotated with a paradigm properties and a language ID. Thus, the model enables transfer learning across languages when the two languages share common characters and common paradigms. While the proposed multi-lingual solution is not novel (similar architectures have been explored in syntax, language modeling, and MT), the novelty of this paper is to apply the approach to morphology. Experimental results show substantial improvements over monolingual baselines, and include a very thorough analysis of the impact of language similarities on the quality of results. The paper is interesting, very clearly written, I think it’ll be a nice contribution to the conference program.   Detailed comments:   — My main question is why the proposed general multilingual methodology was limited to pairs of languages, rather than to sets of similar languages? For example, all Romance languages could be included in the training to improve Spanish paradigm completion, and all Slavic languages with Cyrillic script could be mixed to improve Ukrainian. It would be interesting to see the extension of the models from bi-lingual to multilingual settings.   — I think Arabic is not a fair (and fairly meaningless) baseline, given how different is its script and morphology from the target languages. A more interesting baseline would be, e.g., a language with a partially shared alphabet but a different typology. For example, a Slavic language with Latin script could be used as a baseline language for Romance languages. If Arabic is excluded, and if we consider a most distant language in the same the same family as a baseline, experimental results are still strong.   — A half-page discussion of contribution of Arabic as a regularizer also adds little to the paper; I’d just remove Arabic from all the experiments and would add a regularizer (which, according to footnote 5, works even better than adding Arabic as a transfer language).                — Related work is missing a line of work on “language-universal” RNN models that use basically the same approach: they learn shared parameters for inputs in multiple languages, and add a language tag to the input to mediate between languages. Related studies include a multilingual parser (Ammar et al., 2016), language models (Tsvetkov et al., 2016), and machine translation (Johnson et al., 2016 )  Minor:  — I don’t think that the claim is correct in line 144 that POS tags are easy to transfer across languages. Transfer of POS annotations is also a challenging task.    References:   Waleed              Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. "Many languages, one parser.” TACL 2016.   Yulia Tsvetkov, Sunayana Sitaram, Manaal Faruqui, Guillaume Lample, Patrick Littell, David Mortensen, Alan W. Black, Lori Levin, and Chris Dyer. "Polyglot neural language models: A case study in cross-lingual phonetic representation learning.” NAACL 2016.  Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat et al. "Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation." arXiv preprint arXiv:1611.04558 2016.  -- Response to author response:   Thanks for your response & I'm looking forward to reading the final version!
en

This paper delves into the mathematical properties of the skip-gram model, explaining the reason for its success on the analogy task and for the general superiority of additive composition models. It also establishes a link between skip-gram and Sufficient Dimensionality Reduction.  I liked the focus of this paper on explaining the properties of skip-gram, and generally found it inspiring to read. I very much appreciate the effort to understand the assumptions of the model, and the way it affects (or is affected by) the composition operations that it is used to perform. In that respect, I think it is a very worthwhile read for the community.  My main criticism is however that the paper is linguistically rather naive. The authors' use of 'compositionality' (as an operation that takes a set of words and returns another with the same meaning) is extremely strange. Two words can of course be composed and produce a vector that is a) far away from both; b) does not correspond to any other concept in the space; c) still has meaning (productivity wouldn't exist otherwise!) Compositionality in linguistic terms simply refers to the process of combining linguistic constituents to produce higher-level constructs. It does not assume any further constraint, apart from some vague (and debatable) notion of semantic transparency. The paper's implication (l254) that composition takes place over sets is also wrong: ordering matters hugely (e.g. 'sugar cane' is not 'cane sugar'). This is a well-known shortcoming of additive composition.   Another important aspect is that there are pragmatic factors that make humans prefer certain phrases to single words in particular contexts (and the opposite), naturally changing the underlying distribution of words in a large corpus. For instance, talking of a 'male royalty' rather than a 'king' or 'prince' usually has implications with regard to the intent of the speaker (here, perhaps highlighting a gender difference). This means that the equation in l258 (or for that matter the KL-divergence modification) does not hold, not because of noise in the data, but because of fundamental linguistic processes. This point may be addressed by the section on SDR, but I am not completely sure (see my comments below).  In a nutshell, I think the way that the authors present composition is flawed, but the paper convinces me that this is indeed what happens in skip-gram, and I think this is an interesting contribution.   The part about Sufficient Dimensionality Reduction seems a little disconnected from the previous argument as it stands. I'm afraid I wasn't able to fully follow the argument, and I would be grateful for some clarification in the authors' response. If I understand it well, the argument is that skip-gram produces a model where a word's neighbours follow some exponential parametrisation of a categorical distribution, but it is unclear whether this actually reflects the distribution of the corpus (as opposed to what happens in, say, a pure count-based model). The fact that skip-gram performs well despite not reflecting the data is that it implements some form of SDR, which does not need to make any assumption about the underlying form of the data. But then, is it fair to say that the resulting representations are optimised for tasks where geometrical regularities are important, regardless of the actual pattern of the data? I.e. there some kind of denoising going on?  Minor comments:  - The abstract is unusually long and could, I think, be shortened.  - para starting l71: I think it would be misconstrued to see circularity here. Firth observed that co-occurrence effects were correlated with similarity judgements, but those judgements are the very cognitive processes that we are trying to model with statistical methods. Co-occurrence effects and vector space word representations are in some sense 'the same thing', modelling an underlying linguistic process we do not have direct observations for. So pair-wise similarity is not there to break any circularity, it is there because it better models the kind of judgements humans known to make.  - l296: I think 'paraphrase' would be a better word than 'synonym' here, given that we are comparing a set of words with a unique lexical item.  - para starting l322: this is interesting, and actually, a lot of the zipfian distribution (the long tail) is fairly uniform.  - l336: it is probably worth pointing out that the analogy relation does not hold so well in practice and requires to 'ignore' the first returned neighbour of the analogy computation (which is usually one of the observed terms).  - para starting l343: I don't find it so intuitive to say that 'man' would be a synonym/paraphrase of anything involving 'woman'. The subtraction involved in the analogy computation is precisely not a straightforward composition operation, as it involves an implicit negation.   - A last, tiny general comment. It is usual to write p(w|c) to mean the probability of a word given a context, but in the paper 'w' is actually the context and 'c' the target word. It makes reading a little bit harder... Perhaps change the notation?  Literature:  The claim that Arora (2016) is the only work to try and understand vector composition is a bit strong. For instance, see the work by Paperno & Baroni on explaining the success of addition as a composition method over PMI-weighted vectors:  D. Paperno and M. Baroni. 2016. When the whole is less than the sum of its parts: How composition affects PMI values in distributional semantic vectors. Computational Linguistics 42(2): 345-350.  *** I thank the authors for their response and hope to see this paper accepted.
en

The paper introduces a new pruning method for neural networks based on the second-order Taylor expansion and compares the results against a first-order method and brute-force pruning. It performs experiments of the three methods on several toy examples - including a two-layer network on MNIST - and shows that the second-order method behaves much worse then the brute-force baseline. In addition, from the success of the brute-force pruning the authors conclude that the hypothesis of Mozer et al - that neurons either contribute to performance or cancel out the effect of other neurons - is probably correct.  The authors put in considerable effort to explain all details of the paper clearly and at length, so the content of the paper is accessible even to people novel to pruning methods. Additionally, the authors have very carefully answered all questions that were coming up through the pre-review and have been very responsive.  My major criticism is that the paper lacks focus, does not have a concrete conclusion and does not explain what it adds to the literature. To make this apparent, I here summarise each paragraph of the conclusion section:  Paragraph 1: We do not benchmark / Pruning methods do not fare well against brute-force baseline / Some evidence for hypothesis of Mozer & Smolensky, but further investigation needed  Paragraph 2: Introduced 2nd order Taylor method / Does not fare well against baseline  Paragraph 3: Re-training may help but is not fair  Paragraph 4: Brute-force can prune 40-70% in shallow networks  Paragraph 5: Brute-force less effective in deep networks  Paragraph 6: Not all neurons contribute equally to performance of network  The title of the paper and answers of the authors to the pre-review questions seemed to strongly suggest that the paper is not about the new second-order method, is not about benchmarking pruning algorithms but is instead about the learnt representations. But only two or three sentences in the conclusion, and no sentence in the part on results in the abstract, even refers to neural representations. In an answer to the pre-review questions the authors stated:  > Furthermore, we do not have to accept the conclusion that re-training is a necessary part of pruning because a brute force search reveals that neurons can in fact be  > pruned from trained networks in a piecemeal fashion with no retraining and minimal adverse effect on the overall performance of the network. This would be  > impossible if neurons did not belong to the distinct classes we describe."  But this can already be concluded from the 2nd order method, which has a similar characteristic and is based on other 2nd order methods (not shown here). What is the motivation to introduce a new 2nd order method here?  In addition, some other minor conclusions about representations - in particular the cancellation effect - might be based on side-effects of the greedy serial pruning method. Optimally, one would need to consider all the different ways of pruning (which, of course, scales exponentially with the number of neurons and is computationally infeasible). Notably, the authors do consider this limitation in the context of conventional pruning methods in the conclusions: "Third, we assumed that pruning could be done in a serial fashion [...]. We found that all of these assumptions are deeply flawed in the sense that the true relevance of a neuron can only be partially approximated [...] at certain stages of the pruning process". But the brute-force pruning process is also serial - why is that not a problem?  All in all it is unclear to me what the paper adds: there are little conclusions regarding the learnt representations nor is there sufficient benchmarking against state-of-the-art pruning methods. I would suggest to focus the paper in the following way: first, use a state-of-the-art pruning method from the literature (that works without re-training) or do not use any other pruning methods besides brute-force (depending on whether you want to compare pruning methods against brute-force, or want to learn something about the learnt representations). In this way you need to write little about this second-order tuning methods, and readers are not so easily confused about the purpose of this paper (plus it will be considerably shorter!). Then concentrate on 2-layer MNIST and a deeper CIFAR10 network. Further focus the paper by adding an itemised list of the exact contributions that you make, and streamline the paper accordingly. These measures could strongly boost the impact of your work but will require a major revision.  PS: I think the confusion starts with the following sentence in the abstract: "In this work we set out to test several long-held hypothesis about neural network learning representations and numerical approaches to pruning." Both aspects are pretty orthogonal, but are completely mixed up in the paper.

en

