TL;DR I appreciate the efforts and observations / merits found by the authors. However, this paper poorly presents the methodology (both details and its key advantage), and it’s hard to validate the conclusion with such little hyperparameter analysis. I would love to see more detailed results, but I could not accept this version to be accepted as an EMNLP paper.
1 There are too many missing details when presenting the methodology: e.g., what will be the effect if I remove one or two losses presented by the authors? Though their motivations are clear, they do not validate the hypothesis clearly.
2 A lot of equations look like placeholders, such as equations (1, 2, 3, 5, 6).
3 Some of the pieces are simply using existing methods, such as equation (12), the presentation of these methods are also vague (can only be understood after checking the original paper).
4 The pipeline misses a lot of details. For example, how long does it take to pre-train each module? How adding pre-training will benefit the performance? How to schedule the training of the discriminator and the main module? Not mentioning the detail design of the RNN network used.
5 Why do we need to focus on the four aspects? They are just listed there. Also, some of the results presentation does not seem to be thorough and valid. For example, in table 2, the Quora datasets have the highest perturbation ratio, but the downgraded performance is the least among the three. Is it really because the adversarial samples are effective instead of the task variance or dataset variance? Also, we didn’t see the attack performance of other comparison methods. And how is the test set generated? What is the size of the adversarial test set and why is that a good benchmark?
6 In table 4, it’s actually hard to say which is better, A^3 or A^2T, if you count the number of winners for each row and column.
7 In table 5, is the computation time also considered the pre-training stage? If not, why? Does the pre-training stage can serve as a unified step which is agnostic to the dataset and tasks?
8 I don’t quite understand the point of section 4.6, and its relationship to the effectiveness of A^3. This influence of /rho seems to be really obvious. I would rather be more interested in changing the six hyperparameters mentioned in line 444 and test their effectiveness.
9 The related work section is also not very well-written. I couldn’t understand what is the key difference and key advantage of A^3 compared to the other methods.
3 Some of the pieces are simply using existing methods, such as equation (12), the presentation of these methods are also vague (can only be understood after checking the original paper).