- There's over reliance on the LLM to trust the automated scoring with the knowledge that LLMs have their complex biases and sensitivity to prompts (and order).
- It is not clear how the method will perform on long conversations (the dialog datasets used for prompt and demonstration selection seem to contain short conversations)
- The paper can be simplified in writing - the abstract is too long and does not convey the findings well.
- Fig. 1 can also be drawn better to show the processing pipeline (prompt generation and manual check, demonstration selection with ground truth scores, and automatic scoring alongwith showing where model training is being used to optimize the selection modules.
- Fig. 1 can also be drawn better to show the processing pipeline (prompt generation and manual check, demonstration selection with ground truth scores, and automatic scoring alongwith showing where model training is being used to optimize the selection modules.