As always, more could be done in the experiments section to strengthen the case for chunk-based models. For example, Table 3 indicates good results for Model 2 and Model 3 compared to previous papers, but a careful reader will wonder whether these improvements come from switching from LSTMs to GRUs. In other words, it would be good to see the GRU tree-to-sequence result to verify that the chunk-based approach is still best.
Another important aspect is the lack of ensembling results. The authors put a lot of emphasis is claiming that this is the best single NMT model ever published. While this is probably true, in the end the best WAT system for Eng-Jap is at 38.20 (if I'm reading the table correctly) - it's an ensemble of 3. If the authors were able to report that their 3-way chunk-based ensemble comes top of the table, then this paper could have a much stronger impact.
Finally, Table 3 would be more interesting if it included decoding times. The authors mention briefly that the character-based model is less time-consuming (presumably based on Eriguchi et al.'16), but no cite is provided, and no numbers from chunk-based decoding are reported either. Is the chunk-based model faster or slower than word-based? Similar? Who know... Adding a column to Table 3 with decoding times would give more value to the paper.
- General Discussion: Overall I think the paper is interesting and worth publishing. I have minor comments and suggestions to the authors about how to improve their presentation (in my opinion, of course). - I think they should clearly state early on that the chunks are supplied externally - in other words, that the model does not learn how to chunk. This only became apparent to me when reading about CaboCha on page 6 - I don't think it's mentioned earlier, and it is important.
- I don't see why the authors contrast against the char-based baseline so often in the text (at least a couple of times they boast a +4.68 BLEU gain). I don't think readers are bothered... Readers are interested in gains over the best baseline.
- It would be good to add a bit more detail about the way UNKs are being handled by the neural decoder, or at least add a citation to the dictionary-based replacement strategy being used here.
- The sentence in line 212 ("We train a GRU that encodes a source sentence into a single vector") is not strictly correct. The correct way would be to say that you do a bidirectional encoder that encodes the source sentence into a set of vectors... at least, that's what I see in Figure 2.
- The motivating example of lines 69-87 is a bit weird. Does "you" depend on "bite"? Or does it depend on the source side? Because if it doesn't depend on "bite", then the argument that this is a long-dependency problem doesn't really apply.
- The sentence in line 212 ("We train a GRU that encodes a source sentence into a single vector") is not strictly correct. The correct way would be to say that you do a bidirectional encoder that encodes the source sentence into a set of vectors... at least, that's what I see in Figure 2.