In this task, you are given a paper review. Based on the review, your job is to identify language and generate "en" if the review is in English or generate "es" if the review is in Spanish. Note that URLs in the text have been replaced with [Link].

Q: The primary point made by this paper is that given certain architectural characteristics of multi-GPU systems, namely the use of bi-directional PCI-E for communication and the integration of two independent DMA engines on recent GPU devices (providing support for simultaneous independent communications), and given the characteristics of the communications patterns required by synchronous SGD trainers for deep neural networks, namely that the messages are large, dense, and have a fixed length, it makes sense to design communication collectives such as broadcast, reduce, and allreduce specifically for the use case of synchronous SGD training on a multi-GPU system.  The paper describes the implementation of these three collectives (broadcast, reduce, and allreduce) using a linear pipelining (LP) scheme on a (logical) ring topology.  The paper compares the LP collectives to two alternatives:  collectives based on a minimal spanning tree (MST) topology and collectives based on bidirectional exchange (BE).  First, a theoretical comparison is made using a standard cost model used in the high performance computing community.  When assumptions based on multi-GPU system architecture (very low latency for messages) and on the communication characteristics of synchronous SGD training (very large messages) are integrated into the model, the paper finds that the LP collectives should be less costly than BE collectives by a factor of 2 and less costly than MST collectives by a factor of log(p), where p is the number of GPUs being used.  Second, an empirical comparison is performed in which (1) the time required to perform each of the different collectives on a 4-device (k40m) system is measured as a function of message size and (2) the time required to perform each of the different collectives with a 200 MB message length is measured as a function of the number of devices in the system.  These measurements show that the LP-based collectives are consistently the fastest.  Third, DNN training experiments with AlexNet and GoogLeNet are performed on a 4-device system using three different synchronous SGD algorithms with the different implementations of the collectives (a total of 6 different algorithms in all).  Measurements of the communication and computation costs show that the LP collectives reduce communication costs without affecting computation costs (as expected).  Measurements of the convergence of the training loss as a function of time for the two DNN architectures show that use of the LP collectives leads to faster training.  While the theory says that the costs of LP collectives should be invariant to the number of devices in a multi-GPU system, the empirical work shows that in practice this does not hold going from 4 to 5 devices (in the tested configuration) because in a 5-device system messages must traverse the QPI.  Are there other practical considerations that the authors are aware of that affect the scaling of the LP collectives?  If so, these should be mentioned in the paper.  In the sentence "Worringen (2003) proposed a pipeline collective model in shared memory environment for CPU data, but communications of different MPI processes sharing the same CPU memory bus within the same CPU socket." I really can't figure out what the words after "but communications of different MPI processes" are trying to convey.  This sentence is not comprehensible.  "Please note the latency term is log pα, which is the smallest among algorithms in Table.1. Therefore, MST only suits for high frequent short messages."  The claim that MST collectives are only suitable for high-frequency, short messages does not follow from the statement that MST collectives have the smallest latency term.  You also need to consider the way the cost scales with message size (the bandwidth term).  If the MST collectives had a better bandwidth term than the other collectives, then they would also be superior for large messages.  "Let’s take an appropriate block size b to ensure n/b ≪ α."  This looks wrong, since n > b.  Should it be b/n ≪ α?  "However, the parameters are not consistent after several iterations due to the precision issues of float multiplications in Gradient Update."  Are you sure the inconsistency in weight estimates across devices is due to multiplication?  I would expect that it would be due to gradients being accumulated in different orders; that is, because floating point addition is not commutative.  I recommend replacing the term "sub-gradients" in this paper with "partial gradients."  In the optimization literature, the term "sub-gradient" has a very specific meaning that differs from this paper's use of the term (see

A: en
****
Q: The paper introduces a new pruning method for neural networks based on the second-order Taylor expansion and compares the results against a first-order method and brute-force pruning. It performs experiments of the three methods on several toy examples - including a two-layer network on MNIST - and shows that the second-order method behaves much worse then the brute-force baseline. In addition, from the success of the brute-force pruning the authors conclude that the hypothesis of Mozer et al - that neurons either contribute to performance or cancel out the effect of other neurons - is probably correct.  The authors put in considerable effort to explain all details of the paper clearly and at length, so the content of the paper is accessible even to people novel to pruning methods. Additionally, the authors have very carefully answered all questions that were coming up through the pre-review and have been very responsive.  My major criticism is that the paper lacks focus, does not have a concrete conclusion and does not explain what it adds to the literature. To make this apparent, I here summarise each paragraph of the conclusion section:  Paragraph 1: We do not benchmark / Pruning methods do not fare well against brute-force baseline / Some evidence for hypothesis of Mozer & Smolensky, but further investigation needed  Paragraph 2: Introduced 2nd order Taylor method / Does not fare well against baseline  Paragraph 3: Re-training may help but is not fair  Paragraph 4: Brute-force can prune 40-70% in shallow networks  Paragraph 5: Brute-force less effective in deep networks  Paragraph 6: Not all neurons contribute equally to performance of network  The title of the paper and answers of the authors to the pre-review questions seemed to strongly suggest that the paper is not about the new second-order method, is not about benchmarking pruning algorithms but is instead about the learnt representations. But only two or three sentences in the conclusion, and no sentence in the part on results in the abstract, even refers to neural representations. In an answer to the pre-review questions the authors stated:  > Furthermore, we do not have to accept the conclusion that re-training is a necessary part of pruning because a brute force search reveals that neurons can in fact be  > pruned from trained networks in a piecemeal fashion with no retraining and minimal adverse effect on the overall performance of the network. This would be  > impossible if neurons did not belong to the distinct classes we describe."  But this can already be concluded from the 2nd order method, which has a similar characteristic and is based on other 2nd order methods (not shown here). What is the motivation to introduce a new 2nd order method here?  In addition, some other minor conclusions about representations - in particular the cancellation effect - might be based on side-effects of the greedy serial pruning method. Optimally, one would need to consider all the different ways of pruning (which, of course, scales exponentially with the number of neurons and is computationally infeasible). Notably, the authors do consider this limitation in the context of conventional pruning methods in the conclusions: "Third, we assumed that pruning could be done in a serial fashion [...]. We found that all of these assumptions are deeply flawed in the sense that the true relevance of a neuron can only be partially approximated [...] at certain stages of the pruning process". But the brute-force pruning process is also serial - why is that not a problem?  All in all it is unclear to me what the paper adds: there are little conclusions regarding the learnt representations nor is there sufficient benchmarking against state-of-the-art pruning methods. I would suggest to focus the paper in the following way: first, use a state-of-the-art pruning method from the literature (that works without re-training) or do not use any other pruning methods besides brute-force (depending on whether you want to compare pruning methods against brute-force, or want to learn something about the learnt representations). In this way you need to write little about this second-order tuning methods, and readers are not so easily confused about the purpose of this paper (plus it will be considerably shorter!). Then concentrate on 2-layer MNIST and a deeper CIFAR10 network. Further focus the paper by adding an itemised list of the exact contributions that you make, and streamline the paper accordingly. These measures could strongly boost the impact of your work but will require a major revision.  PS: I think the confusion starts with the following sentence in the abstract: "In this work we set out to test several long-held hypothesis about neural network learning representations and numerical approaches to pruning." Both aspects are pretty orthogonal, but are completely mixed up in the paper.

A: en
****
Q: This paper proposes new initialization for particular architectures and a correction trick to batch normalization to correct variance introduced by dropout. While authors state interesting observations, the claims are not supported with convincing results.  I guess Figure 1 is only for mnist and for only two values of p with one particular network architecture, the dataset and empirical setup is not clear.  The convergence is demonstrated only for three dropout values in Figure 2 which may cause an unfair comparison. For instance how does the convergence compare for the best dropout rate after cross-validation (three figures each figure has three results for one method with different dropouts [bests cv result for each one])? Also how is the corresponding validation error and test iterations?  Also only mnist does not have to generalize to other benchmarks.  Figure 3 gives closer results for Adam optimizer, learning rate is not selected with random search or bayesian optimization, learning decay iterations fixed and regularization coefficient is set to a small value without tuning. A slightly better tuning of parameters may close the current gap. Also Nesterov based competitor gives unreasonably worse accuracy compared to recent results which may indicate that this experiment should not be taken into account.   In Table 2, there is no significant improvement on CIFAR10. The CIFAR100 difference is not significant without including batch normalization variance re-estimation. However there is no result for 'original with BN update' therefore it is not clear whether the BN update helps in general or not. SVHN also does not have result for original with BN update.  There should be baselines with batch normalizations for Figure 1,2 3 to support the claims convincingly.  The main criticism about batch normalization is additional computational cost by giving (Mishkin et al, 2016 ) as reference however this should not be a reason to not to compare the initialization to batch-normalization.  In fact, (Mishkin et al, 2016) performs comparison to batch normalization and also with and without data augmentation with recent state of art architectures.  None of the empirical results have data augmentation. It is not clear if the initialization or  batch normalization update will help or make it worse for that case.  Recent state of art methods methods like Res Net variant and Dense Net scale to many depths and report result for ImageNet. Although the authors claim that this can be extended to residual network variants, it is not clear if there is going to be any empirical gain for that architectures.    This work requires a comprehensive and fair comparison. Otherwise the contribution is not significant.

A:
en
****
