1. In this study, three distinct LLMs named Galactica, BioMedLM, and ChatGPT have been selected by the authors. The differences between these LLMs are outlined in the related work section of the paper. Despite their notable distinctions in training data and size, the evaluation of all these LLMs follows a uniform approach. A more effective approach would involve assessing the outputs of each LLM separately, as this could provide valuable insights into their relative performance in generating medical systematic reviews. It is plausible that certain LLMs might exhibit higher risk factors, while others could excel in generating coherent systematic reviews. In essence, this study would benefit significantly from a comprehensive comparative analysis between the LLMs, allowing for a more nuanced understanding of their respective capabilities, limitations, and potential benefits.
2. The number of samples presented to each domain expert appears to be relative inadequate to draw definitive conclusions about the abilities and constraints of LLMs in generating systematic reviews. Additionally, during the expert interviews, the inclusion of human-written systematic reviews, without indicating their human origin, could offer valuable insights. This approach would allow observation of how domain experts react to these reviews, shedding light on the deficiencies of LLM-generated systematic reviews and thereby allowing a more comprehensive understanding of the lacking of the LLM generated review.
3. The prompting technique used in this study is very basic and fail to leverage the full potentials of LLMs. Carefully curated prompts can gain better results in generating better systematic reviews.
3. The prompting technique used in this study is very basic and fail to leverage the full potentials of LLMs. Carefully curated prompts can gain better results in generating better systematic reviews.