Unfortunately, this paper has many weaknesses. First, and importantly, the main contribution of this paper (section 5.3) is not well motivated and the explanation of the contribution itself is really handwavy. Second, explanation of the different baseline approaches is also too handwavy, making the overall paper difficult to understand and not self-contained.
On the motivations: the authors argue that previous approaches cannot be applied to structured prediction. I strongly disagree with this.
l 227, "the output distributiion of global prediction is often intractable": in the non-projective dependency parsing case, the distribution can be computed using the Matrix Tree Theorem [1, 2, 3]. If we restrict the problem to projective trees, this can be computed via dynamic programming [4, 5]. For the transition based model, approximated approaches have been explored in the litterature [6]
l. 222, "the instantiations of KD are usually based on a loss function computing the cross-entropy of output distributions [...] however existing methods are no directly applicable to structure prediction tasks": I don't understand why. First, the next sentence is false (see previous point), but also KL divergence between structured distributions has been studied in the litterature. For non-projective trees, see [7, Section 6.3], for methods based on dynamic programming see [8]
l. 263, "furthermore, it is unclear how to adapt existing sampling methods to solutions that are not formalized as sequence generation": Recent work has considered sampling from the non-projective tree distributions, including sampling without replacement [9, 10]. Moreover, previous work has also considered perturb-and-MAP approaches [11, 12, 13]. Finally, in the case of dynamic programming algorithms, it is well known that it is possible to sample from the associated exponential familly distributions, see e.g. [14]
Related work
As suggested by the comment above, the literature is not properly explored or cited by the authors. There are similar problems in the introduction and related work section. For example:
l. 54: authors cite Dozan and Manning (2017) for graph-based parsers, whereas the correct citations is more than 10 years older [15]
l. 21: for transition-based parsers, they cite Ma et al. (2018), better citations would be [16, 17]
[1] Structured Prediction Models via the Matrix-Tree Theorem (Koo et al.)
[2] Probabilistic Models of Nonprojective Dependency Trees (Smith and Smith)
[3] On the complexity of non-projective data-driven dependency parsing (McDonald and Satta)
[4] Semiring parsing (Goodman)
[5] Differentiable Dynamic Programming for Structured Prediction and Attention (Mensch and Blondel)
[6] Globally Normalized Transition-Based Neural Networks (Andor et al.)
[7] Efficient Computation of Expectations under Spanning Tree Distributions (Zmigrod et al.)
[8] First- and Second-Order Expectation Semirings with Applications to Minimum-Risk Training on Translation Forests (Li and Eisner)
[9] Efficient Sampling of Dependency Structures (Zmigrod et al.)
[10] Unbiased and Efficient Sampling of Dependency Trees (Stanojevi)
[11] Perturb-and-MAP random fields: Using discrete optimization to learn and sample from energy models (Papandreou and Yuille)
[12] Differentiable Perturb-and-Parse: Semi-Supervised Parsing with a Structured Variational Autoencoder (Corro and Titov)
[13] Learning Latent Trees with Stochastic Perturbations and Differentiable Dynamic Programming (Corro and Titov)
[14] See section 17.4.5 of Machine Learning: A Probabilistic Perspective (Muprhy) for the idea and Latent Template Induction with Gumbel-CRFs (Fu et al.) for application to CRF-like distribution
[15] Non-projective Dependency Parsing using Spanning Tree Algorithms (McDonald et al.)
[16] A Classifier-Based Parser with Linear Run-Time Complexity (Sagae, Lavie)
[17] Algorithms for Deterministic Incremental Dependency Parsing (Nivre)
54: authors cite Dozan and Manning (2017) for graph-based parsers, whereas the correct citations is more than 10 years older [15] l.