Discussion
tumor, tissue damage
TUMOR
This cohort study is the first study to demonstrate unintended surgical skill acquisition with both positive and negative consequences following an AI-enhanced curriculum, to our knowledge. In the competency-based framework of postgraduate medical training, AI provides a tool to identify and teach quantifiable metrics of expertise. Harnessing this power in neurosurgical simulation led our team to design and test the first curriculum with AI-selected competencies. The previous randomized trial involving medical students compared the efficacy of AI tutoring with remote expert instruction using performance outcomes that included only the 4 metrics taught by the VOA.Metrics affected by this AI-enhanced curriculum fall into 1 of 3 categories. The first group includes the intrinsic metrics taught by the intelligent tutoring system. Because the feedback provided by the tutor was focused on these preselected learning objectives, observing the expected outcome provides evidence for feedback efficacy. The second group involves implicit metrics, such as instrument divergence or changes in bipolar force, that despite receiving no direct feedback demonstrated significant change by virtue of their close association with the intrinsic learning objectives. The last group are extrinsic metrics, such as the rate of healthy tissue removal, that have a more complex relationship with the intended learning objectives that cannot be easily implied.Among extrinsic metrics, changes in dominant hand movement were an interesting observation because the students were instructed by the tutor to monitor the acceleration of their nondominant hand and to keep their instruments close together. Nondominant hand training was previously demonstrated to be an effective mode of surgical skill acquisition because of its close association with expertise.It is possible that the modality of feedback presentation contributes to the extended effects observed in this study. As part of the VOA feedback, participants were exposed to four 60-second videos of both competent and novice performance covering each learning objective. Although instructions in each video were specific to 1 metric, participants could potentially gain further information to improve their performance by seeing the expert demonstrations. However, this fails to explain the divergence from benchmarks in some efficiency and movement metrics.The extended effects of this AI curriculum show both productive and counterproductive changes to skill acquisition. Although VOA’s unintentional learning outcomes resulted in students scoring within expert safety benchmarks for this procedure, it diverged some efficiency metrics away from the expert’s benchmarks. In general, VOA participants demonstrated a safer approach with more focused and steady control of the instruments that also resulted in less healthy tissue damage. However, they became significantly slower in movements of their dominant hand and were less efficient in removing the tumor. Such unintended effects must be considered carefully and require a cost-benefit evaluation by experts to determine the acceptability of an intelligent tutoring system. Furthermore, given the extent of unintended outcomes observed in this study, future studies may benefit from measuring the slope of each metric’s learning curve to investigate the learning rate elicited by an AI-enhanced curriculum using extrinsic performance outcomes.One method to evaluate the gains and losses of skill due to unintended effects is to refer to participants’ previously reported expert-rated Objective Structured Assessment of Technical Skills. These results demonstrated that the VOA group achieved a significantly higher overall performance score and a higher economy of movement compared with the control group.