- Some important points about the method and the experiments are left unclear (see also questions below). - The writing could be improved (see also Typos & Additional Questions below) - Multiple runs and significance tests are missing. This makes it hard to judge the improvements (Table 2 & 3).
Most Important Questions - Line 156: What is q_ij^k here exactly? I thought q_ij was a state flag, such as “2” or “0”. But you tokenize it and encode it, so it sounds more like it is something like “Copy(snow)”? (If it is the latter, then what is the meaning of tokenizing and encoding something like “Len(9)”?) - 192: What exactly is storyline and what do you need it for? - The baseline takes the predicate logic constraints as input: How does T6 know what to do with these inputs? Was the model trained on this but without the NRETM module? Can you give an example of what the input looks likes? How do these inputs guide which sentences should be generated? Looking at the datsset, it feels like one would need at least the first 2 sentences or so to know how to continue. Maybe this information is now in your constraints but it would be important to understand what they look like and how they were created. Is there no other suitable baseline for this experiment? - What is the overhead of your method compared to standard decoding approaches? (you mention GBS can only be used with T5-Base, so your method is more efficient? That would be important to point out) - What happens if the decoding process cannot find a sequence that satisfies all constraint? - Document-level MT: How do you know at test time whether the system translates a particular sentence or not? - How many sentences are misaligned by Doc-mBART25? What are the s-BLEU and d-BLEU values on the subset that NRETM aligns correctly and Doc does not? - Why was NEUROLOGIC not used as a comparison baseline? - What is dynamic vs static strategy? In which experiment did you show that dynamic works better than static (from conclusion)?
Typos & Additional Questions - Line 40: you could mention here that the examples will be translated into logic forms in the next section. - Paragraph starting at line 53: Why did you choose these datasets? How will they help evaluate the proposed approach? - Line 75: a and b should be bold faced? - 83: “that used” -> “that are used” - 83: “details” -> “for details” - Paragraph at line 86: At this point, the state matrix is unclear. What are the initial values? How can the state matrix be used to understand if a constraint is satisfied or not? - 98: “take[s]” & “generate[s]” - 108: “be all” -> “all be” - Paragraph at line 101: What is dynamic vs static strategy? - Paragraph at line 109: The state flag explanation would greatly benefit from an example. Does q_i refer to whether a particular U_i is satisfied? - Eq 2: What is the meaning of N? Can it change depending on the definition of U_k? Does it mean this constraint is not relevant for x_i? - 133: Figure 1 should be Figure 2 - Figure 2: What exactly do the “&” rows track? - Figure 2: Is the state flag matrix equal to the state matrix? If not, how do you go from one to the other? - Line 146: What does the inf in the superscript signify? - 177: What is the symbolic operator? - Paragraph at line 194: Without understanding what a storyline is, it is not clear what the constraints are. An example might be helpful here. - Line 204: what is the ROUGH-L metric? Do you mean ROUGE-L? - Line 223: How do you obtain the morphological inflections for the concepts? - 237: @necessity [of] integrating” - 3.3: How exactly is the document-level MT done? Is the entire input document the input to T5? - 293: “because” typo - 3.4 where/how exactly is the sentence index used?
The paper's broader impact section discusses general potential benefits and issues of text generation (from large language models). It could maybe be tailored a bit better by discussing what effect this proposed work would have on the potential benefits and issues.
- Line 223: How do you obtain the morphological inflections for the concepts?