1. The experiments are held on a private datasets and the exact setup is impossible to reproduce.
2. A minor point would be that few-shot would be a more realistic setup for that task, as domain-specific TODOs are easy to acquire, however I agree that the current setup is adequate as well.
3. More error analysis could be useful, especially on the public dataset, as their data could be included without any restrictions, e.g., error types/examples? patterns? Examples when non-contextualized embeddings outperform contextualized ones, or even LITE?
I urge the authors to release at least some part of the dataset to the wider public, or under some end user-agreement.
Comments: 1. I suggest the authors to focus their comparison on word2vec baselines (currently in appendix), instead of Sentence-BERT, as the latter does not show good performance on the short texts. It seems that non-contextualized embeddings are more suitable for the task.
2. Maybe it makes more sense to try out models pre-trained on conversations, e.g., text from Twitter or natural language conversations.
2. A minor point would be that few-shot would be a more realistic setup for that task, as domain-specific TODOs are easy to acquire, however I agree that the current setup is adequate as well.