Discussion
fits, confusion
Our results showed that using Chatprogress, a chatbot system, potentially improved students’ results on an academic test. The study also showed that medical students not only appreciated the chatbots but more importantly used it as a pedagogical tool. In fact, the intensity of the use of the platform shows a trend to correlate with students’ results: the more games were played or finished, and the more rounds were finished, the better the students performed on their pulmonology test when the subject was covered by Chatprogress.Chatprogress was created to ease the process of learning, using a tool that fits right into the students’ schedules, and pockets [Most of the learning on the platform is done through trial and error, a well-known and efficient learning method [Students were found to really appreciate this mean of teaching. This was concluded not only from the answers of the survey but was also proven by the students’ voluntary GOs to receive extra explanations. This suggests that students were not merely guessing their answers but were genuinely thinking the questions through. It also justifies students’ request to add explanations even when they got a question right to make sure their clinical reasoning held up. Furthermore, part of the games’ strength relies on its ability to teach medical reasoning by repetition, a powerful learning technique [We made sure, to the best of our ability, to limit the cross-over between groups, by first only meeting up with the students that were randomized in the trial group. Personal identification was also distributed, limiting the access of students in the control group to the tool. We also made sure the group contamination was limited by asking the question in the survey, which pointed out that cross-over was limited among those who answered the survey. However, we cannot eliminate a non-response bias, remaining ignorant of the sharing of the platform by those who did not fill out the survey. We also cannot argue that some of the knowledge acquired by using the chatbot could have been shared between peers.Our trial however faces an important limitation: the relatively weak adherence of students to the trial and the survey. First, a selection bias could result from students not showing up the meeting, moving them to the control group in our per-protocol analysis. Students who thus finally constituted the Gamers’ group were more likely to be more assiduous than those in the control group. Similarly, only 26% of users completed the survey. Thus, we can’t rule out that those who answered were those who were particularly fond of the concept. Likewise, grades were not statistically higher in Users when the test revolved around a subject covered by Chatprogress probably because of the small number of students of that trimester (n = 38).An explanation to students’ weak adherence to the trial could be that, in Paris Descartes, the PCC course is taught during their first clinical year. Students during that year learn to juggle between hospital obligations, lectures, study groups and self-education. That could have hindered the introduction of yet another learning tool. Students’ poor response rate to the survey can be explained partly by the timing of when it was sent out. The email was sent at the end of the year, so 3 to 6 months after the completion of the PCC exam for two thirds of the students. It was sent out in July and students had 2 months to fill it out. It was therefore sent during the summer break, when students are usually the least responsive. This is also why a second blast, a few weeks later, was not sent, although a pre-notification or a subsequent reminder after the initial survey would have increased the response rate [Our study was also a single centre one. It was performed on a single class with a limited number of students, targeted a single course, with a limited number of clinical scenarios, that all revolved around only one of the modules, pulmonology. Another limitation could be the use of only multiple-choice questions, which do not reflect how real-life conversations with patients are held. This is a disadvantage that our chatbot has compared to a scenario on a mannequin. We thus wish to develop games in the future with open questions. In addition, we understand that students’ ability to pass an exam is multifactorial, going from students’ schedule, to participation rate, to interest in the field or simply luck on exam day. We tried to minimize the confusion biais by performing a randomized controlled trial, with however a remaining attrition bias, that is hard to take into consideration considering the limited number of students per trimester, and evaluation bias on the evaluation of the primary outcome with students of different trimesters being evaluated on different topics.Similarly, another miss-match with reality would be our chosen outcome, grades on a standardized test, rather than students’ performance in the hospital, the ultimate goal of medical education. However, it was shown that clinical competency assessments, such as the pulmonology test at Paris Descartes that is built around a clinical case, are strong predictors of internship performance [