Evaluating ChatGPT-4 Vision on Brazil's National Undergraduate Computer Science Exam

Nabor C. Mendonça

Post-Graduate Program in Applied Informatics, University of Fortaleza
Av. Washington Soares, 1321, Edson Queiroz

The recent integration of visual capabilities into Large Language Models (LLMs) has the potential to play a pivotal role in science and technology education, where visual elements such as diagrams, charts, and tables are commonly used to improve the learning experience. This study investigates the performance of ChatGPT-4 Vision, OpenAI's most advanced visual model at the time the study was conducted, on the Bachelor in Computer Science section of Brazil's 2021 National Undergraduate Exam (ENADE). By presenting the model with the exam's open and multiple-choice questions in their original image format and allowing for reassessment in response to differing answer keys, we were able to evaluate the model's reasoning and self-reflecting capabilities in a large-scale academic assessment involving textual and visual content. ChatGPT-4 Vision significantly outperformed the average exam participant, positioning itself within the top 10 best score percentile. While it excelled in questions that incorporated visual elements, it also encountered challenges with question interpretation, logical reasoning, and visual acuity. A positive correlation between the model's performance in multiple-choice questions and the performance distribution of the human participants suggests multimodal LLMs can provide a useful tool for question testing and refinement.
However, the involvement of an independent expert panel to review cases of disagreement between the model and the answer key revealed some poorly constructed questions containing vague or ambiguous statements, calling attention to the critical need for improved question design in future exams. Our findings suggest that while ChatGPT-4 Vision shows promise in multimodal academic evaluations, human oversight remains crucial for verifying the model's accuracy and ensuring the fairness of high-stakes educational exams.
The paper's research materials are publicly available at <https://github.com/nabormendonca/gpt-4v-enade-cs-2021>.

<concept_desc>Applied computing E-learning</concept_desc>
<concept_desc>Computing methodologies Natural language processing</concept_desc>
<concept_desc>Computing methodologies Computer vision</concept_desc>

[500]Applied computing E-learning
[500]Computing methodologies Natural language processing
[500]Computing methodologies Computer vision

§ INTRODUCTION

The advent of large language models (LLMs), such as OpenAI's ChatGPT <cit.> and Google's Gemini <cit.>, has ushered in a new era in natural language processing. These models exhibit human-like performance in tasks requiring complex reasoning, including abstract thinking, problem-solving across diverse knowledge domains, coding, and comprehending human emotions and motives <cit.>. The recent integration of visual capabilities into these models <cit.> has the potential to play a pivotal role in advancing their use in educational settings, particularly in the science and technology fields, where visual elements such as diagrams, charts, and tables are commonly used to offer a more comprehensive and engaging learning experience <cit.>.

Using LLMs like ChatGPT and Gemini in education presents novel opportunities and challenges for students, educators, and standardized exam creators <cit.>. These models offer students innovative tools for exam preparation, potentially reshaping study habits and learning methodologies <cit.>. For educators, LLMs serve as a resource to enrich teaching strategies and help tailor exam content to better suit learning outcomes <cit.>. Meanwhile, standardized exam creators could utilize these AI tools to refine their questions, ensuring clarity, fairness, and appropriate difficulty levels. However, the use of these models also introduces concerns around accuracy, reliability, and potential for misuse, highlighting the need for careful evaluation of visual LLMs in academic settings <cit.>.

This study evaluates the accuracy of ChatGPT-4 Vision <cit.>, OpenAI's most advanced visual model at the time the study was conducted, in answering the Bachelor in Computer Science (BCS) section of Brazil's 2021 National Undergraduate Exam (ENADE) <cit.>. Unlike most prior evaluations of ChatGPT and other LLMs in academic assessments in Computer Science (e.g., <cit.>) and other fields (e.g., <cit.>), we presented all ENADE BCS exam questions to ChatGPT-4 Vision in their original image formats as seen by human participants. This method allowed a comprehensive evaluation of the model's multimodal reasoning abilities, challenging its linguistic and visual interpretation capabilities in a particular education domain.

Additionally, we explored the model's capacity for self-reflection by prompting it to reassess its initial responses when presented with differing answer keys. When ChatGPT-4 Vision's answers diverged from the answer keys after reassessment, we assembled an independent panel of computer science researchers from various Brazilian universities to evaluate the accuracy of the model's responses. This additional layer of expert review provided insights into the model's reasoning accuracy and the exam questions' quality regarding clarity and objectivity.

We further correlated the model's performance with the difficulty and discrimination levels of the multiple-choice questions, as reported by ENADE organizers. This analysis aimed to provide insights into how the model's proficiency compared to that of the human participants, offering a more nuanced understanding of the model's capabilities and limitations in a real-world academic assessment context.

Our study aims to answer the following quantitative research questions:

RQ1: How well does ChatGPT-4 Vision perform in answering ENADE 2021 BCS open questions?

RQ2: How well does ChatGPT-4 Vision perform in answering ENADE 2021 BCS multiple-choice questions?

RQ3: How often did ChatGPT-4 change or maintain its response when confronted with a disagreeing official answer?

RQ4: What is ChatGPT-4 Vision's ENADE 2021 BCS exam final score, and how does it compare with the exam participants?

In addition, we examined the reasoning, accuracy, and reflective analysis of ChatGPT-4 Vision, and the independent expert assessments to answer the following qualitative research question:

RQ5: What are the main challenges ChatGPT-4 Vision faced in answering ENADE 2021 BCS open and multiple-choice questions?

In summary, our contributions are as follows:

* We have evaluated ChatGPT-4 Vision's multimodal reasoning abilities on the ENADE 2021 BCS exam, a large-scale computer science academic assessment test involving textual and visual content, providing all questions in image format.

* We have explored ChatGPT-4 Vision's capacity for self-reflection by allowing it to reassess its initial responses when confronted with differing answer keys.

* We have correlated ChatGPT-4 Vision's performance with the difficulty and discrimination levels of the exam's multiple-choice questions, offering insights into how the model's proficiency compares to that of the human participants.

* We have involved an independent panel of computer science researchers from various Brazilian universities to evaluate the accuracy of ChatGPT-4 Vision's responses that conflicted with the official answer, providing further insights into the model's limitations and the exam questions' design quality.

Overall, our results reveal ChatGPT-4 Vision's impressive yet limited performance in reasoning about multimodal content in computer science academic assessments. In particular, ChatGPT-4 Vision significantly outperformed the average exam participant, achieving the top 10 best score percentile. However, the model faced several challenges with question interpretation, logical reasoning, and visual acuity. These findings underscore the continued need for human oversight in verifying LLMs's accuracy in solving complex reasoning tasks. Moreover, some disagreements between the model, the expert panel, and the official answer raise concerns about the quality of some exam questions. Given the significant impact of ENADE on Brazil's undergraduate academic rankings, our findings call attention to the critical need for improved question design and quality control in future ENADE exams to ensure a fair educational assessment process.

To facilitate the replication and extension of our study, we have made our research materials publicly available at <https://github.com/nabormendonca/gpt-4v-enade-cs-2021>. This repository contains the exam questions, response standards, answer keys, ChatGPT-4's full conversations, and experts' evaluations, in Portuguese and English.

The remainder of the paper is organized as follows. Section <ref> discusses related work in LLM evaluations in logical reasoning tasks and educational assessments in computer science and other domains. Section <ref> provides background information on ENADE and the 2021 BCS exam. Section <ref> details the methodology used in our study, including our evaluation procedure and experimental setup. Section <ref> presents the results of our study, and Section <ref> discusses its implications, insights, and limitations. Finally, Section <ref> concludes the paper with a summary of the main contributions and suggestions for future research.

§ RELATED WORK

Our work touches on several areas of related research, including LLM evaluations in general logical reasoning tasks <cit.>, using LLMs in educational assessments in computer science <cit.> and other domains <cit.>, and recent advances in Vision-Language Models (VLMs) <cit.>. Also of interest are related works that evaluated LLMs' effectiveness in answering Brazil's ENEM <cit.>—a national multidisciplinary high-school-level exam widely used as an admission test in Brazilian universities—and ENADE <cit.>. We briefly discuss these studies below to provide a comprehensive backdrop against which the contributions of our work can be contextualized and compared.

§.§ LLMs Evaluation in Logical Reasoning

<cit.> showed that GPT-4 exhibited significant logical reasoning capabilities in various domains, including professional and academic benchmarks, where it performs at the human level. The study also highlighted the model's struggles with planning and creative problem-solving due to its reliance on a sequential prediction paradigm. Similarly, <cit.> focused on logical reasoning in reading comprehension and natural language inference, finding that despite strong performances on known datasets, GPT-4's effectiveness diminishes with new, out-of-distribution data. This observation points to a crucial gap in GPT-4's capacity to handle unfamiliar data types and logical inferences beyond its training data.
<cit.> provided a more holistic view by surveying assessments across various capabilities of ChatGPT and GPT-4, indicating strong language understanding and generation skills but also limitations in tasks requiring multi-step reasoning and domain-specific knowledge.
Finally, <cit.> compared reasoning abilities across LLMs including GPT-3.5, GPT-4, and BARD, finding that while GPT-4 generally outperforms its predecessors and competitors, all models show limitations in inductive, mathematical, and multi-step reasoning tasks.

Building on these foundational studies, our study further explores the capabilities of ChatGPT-4 Vision by evaluating its performance on the ENADE 2021 BCS exam, which includes complex reasoning tasks in a multimodal context. This extends the current understanding of LLMs' capabilities by testing them on more challenging, real-world academic assessments.

§.§ LLMs Evaluation in Computer Science Assessments

<cit.> evaluated the performance of ChatGPT-3.5 across a variety of computer science exercises, including true/false, multiple-choice, short and long answers, and coding-related questions. The study demonstrated ChatGPT-3.5's inconsistent accuracy across subjects and question types, particularly in numerical-based problems. Similarly, <cit.> provide an in-depth analysis of ChatGPT-3.5's performance in a university computer science curriculum. The study involved a blind marking exercise in which the ChatGPT-generated answers were directly compared with those of human students. The findings showed that barring two outliers, every ChatGPT-3.5 answer achieved at least a passing grade. The study also highlights that while ChatGPT-3.5 is capable of producing adequate solutions in most cases, its performance varies across subjects and question types, with ChatGPT-3.5's effectiveness diminishing as the complexity of the questions increases.
<cit.> explored ChatGPT-3.5's performance in answering 31 questions from a popular software testing textbook. The study found that ChatGPT-3.5 provided correct or partially correct answers and explanations for over half of the questions tested, showcasing the model's potential as an educational tool. However, the study also noted ChatGPT's inconsistent accuracy and poor self-assessment, especially in questions that involved both coding and conceptual understanding.
<cit.> investigated ChatGPT-3.5 and 4's performance in an undergraduate computer science exam focusing on algorithms and data structures. ChatGPT-3.5 narrowly passed the exam with 20.5 out of 40 points, a result that was slightly below the performance of the average student. ChatGPT-4 scored about 17% higher, reaching the average student's level. The study underlined the model's challenges with exercises requiring a detailed understanding of algorithmic processes, particularly involving structured output that was not in pseudo-code format.
Finally, <cit.> investigated the capabilities of ChatGPT-4 (no vision) on the UK A-Level and Irish Leaving Certificate computer science examinations. They found that ChatGPT-4 achieved high marks on both exams but struggled with questions involving symbols or images and cascading questions where initial inaccuracies led to further errors. This work underscores the strengths and weaknesses of generative AI in educational assessments and has implications for exam design and marking schemes.

Our research builds on these findings by not only evaluating ChatGPT-4 Vision on a comprehensive academic assessment in Computer Science but also examining its performance relative to question difficulty and discrimination levels. This provides a more nuanced understanding of the model's capabilities and limitations compared to the human participants, which is extremely important in educational contexts.

§.§ LLMs Evaluation in Other Education Domains

<cit.> presents a comprehensive evaluation of ChatGPT-4's performance on the United States Medical Licensing Examination (USMLE) and the MultiMedQA medical benchmark. The study found that GPT-4, without any specialized prompt crafting, surpassed the passing score on the USMLE by over 20 points and outperformed previous models like ChatGPT-3.5 and specialized medical models. The study also explored ChatGPT-4's calibration, crucial in high-stakes domains like medicine, and found it to be significantly better than its predecessor. Similarly, <cit.> investigated the performance of ChatGPT-3.5, ChatGPT-4, and Bard on the official board examination of the Japan Radiology Society. The study found that ChatGPT-4 significantly outperformed ChatGPT-3.5 and Bard by over 20 percentage points, achieving an accuracy of 65% in the exam's multiple-choice questions. Both studies underscore the potential utility of AI models in assisting professionals in high-stakes domains like medicine, while also emphasizing the need for cautious application due to the models' limitations in accuracy and safety-critical tasks.

In the field of engineering, <cit.>  examined ChatGPT-3.5 and 4's performance on the United States Fundamentals of Engineering (FE) Mechanical Engineering Exam. The study found that ChatGPT-4 significantly outperforms ChatGPT-3.5, achieving 76% correct answers compared to 51% by GPT-3.5. A similar study was conducted by <cit.> where they evaluated GPT-3.5-Legacy, GPT-3.5-Turbo, and GPT-4's performance across different sections of the FE Environmental Exam. The study also found that GPT-4 outperformed GPT-3.5-Legacy and GPT-3.5-Turbo by a significant margin, achieving 75% accuracy with slightly refined prompts. However, both studies confirmed previous findings about ChatGPT's error types and pitfalls, noting its inconsistency across question types and subjects and its tendency to confidently produce incorrect answers.

Our study contributes to this body of work by evaluating ChatGPT-4 Vision's performance on a high-stakes academic exam (ENADE), involving both textual and visual content. By allowing the model to reassess its responses with differing answer keys, we explore its self-reflective capabilities, a novel aspect not covered in previous education-centered studies.

§.§ Recent Advances in VLMs

<cit.> conducted a comprehensive evaluation of state-of-the-art VLMs, including GPT-4 Vision, on tasks requiring visual deductive reasoning using Raven's Progressive Matrices (RPMs). Their findings revealed significant challenges in models' abilities to perform multi-hop relational and deductive reasoning based solely on visual cues, highlighting perception issues and difficulties in understanding abstract patterns. <cit.> proposed improvements to GPT-4 Vision's performance on visual reasoning tasks through enhanced prompt engineering techniques. By incorporating specific instructions in prompts, such as paying attention to details and describing items with positional specificity, Mayne demonstrated substantial improvements in the model's accuracy on visual puzzles. <cit.> introduced the Visualization-of-Thought prompting technique, which aims to elicit spatial reasoning in LLMs by visualizing their reasoning traces. This technique enhances spatial reasoning tasks, such as natural language navigation, visual navigation, and visual tiling, by guiding subsequent reasoning steps through mental image visualization. Finally, <cit.> presented MathVerse, a specialized benchmark designed to evaluate the visual mathematical reasoning skills of Multi-modal Large Language Models (MLLMs). MathVerse includes a diverse set of 2,612 visual math problems across three primary areas: plane geometry, solid geometry, and functions. The study found that most existing MLLMs struggle to understand math diagrams and often rely more on textual questions than visual inputs, emphasizing the need for better visual encoding capabilities.

Our work differs from these related studies in several ways but also stands to benefit significantly from their insights and methodologies. While <cit.> provide a comprehensive evaluation of VLMs on visual deductive reasoning tasks, our focus is on assessing ChatGPT-4 Vision's performance in the context of the ENADE exam, which involves a broader range of multimodal reasoning tasks beyond visual deductive reasoning. The visual-enhanced prompt engineering techniques proposed by <cit.> and <cit.> could be particularly beneficial in improving ChatGPT-4 Vision's handling of spatial reasoning tasks within the ENADE exam, guiding the model through intermediate steps via mental image visualization. Furthermore, our work could benefit from the development of a MathVerse-like fine-grained multimodal benchmark based on ENADE questions with visual content to better evaluate and understand ChatGPT-4 Vision's capabilities in real-world educational assessments.

§.§ LLMs Evaluation in Brazil's ENEM and ENADE

<cit.> studied GPT-3.5 and GPT-4's performance on answering multiple-choice ENEM questions, using different prompt strategies, including Chain-of-Thought (CoT). GPT-4, especially with CoT, achieved up to 87% accuracy on the 2022 exam, significantly outperforming GPT-3.5.
<cit.> also examined the performance of ChatGPT-3.5 and 4, the latter with and without Vision capabilities, on the 2022 and 2023 editions of ENEM. The study showed a clear advancement in performance from ChatGPT-3.5 to ChatGPT-4, with GPT-4 models significantly outperforming their predecessors. Specifically, ChatGPT-4 (no Vision) and ChatGPT-4 Vision achieved accuracies of about 73% and 79% in the 2022 exam, and 80% and 86% in the 2023 exam, respectively. Interestingly, when provided with human-generated captions transcribing the questions' visual content, ChatGPT-4 with no Vision capabilities improved its accuracies to 81% and 91% in the 2022 and 2023 exams, respectively, outperforming ChatGPT-4 Vision with images and no captions.
<cit.> investigated the performance of quantized open-source LLaMA-based models on a custom-built benchmark containing over one thousand ENEM questions, showing that the best models achieved an accuracy of 46% for the original texts of the Portuguese questions and 49% on their English translations. Despite the modest computational requirements of these open-source models, allowing them to run on low-end hardware, their limited accuracy compared to that of closed models indicates significant room for improvement.
Finally, in a recent study, <cit.> introduced Sabiá-2, a new family of LLMs specifically trained on Portuguese texts and evaluated its performance on a diverse range of exams, including a multidisciplinary ENADE benchmark containing only multiple-choice questions with no images from the 2022 and 2023 exams. The study revealed that Sabiá-2, particularly the Sabiá-2 Medium model, matched or surpassed GPT-4 Turbo's performance in several ENADE disciplines and significantly outperformed GPT-3.5 in all but one discipline. The model's lowest scores were achieved in domains related to engineering and economics, with a particularly lower score compared to GPT-4 Turbo in Computer Engineering.

Our work extends this area of research by evaluating ChatGPT-4 Vision's performance on the ENADE BCS exam, which includes both open and multiple-choice multimodal questions. Additionally, by involving an independent panel of computer science researchers, we provide a more rigorous assessment of the model's accuracy and limitations in answering ENADE questions, in addition to the questions' overall quality.

§ BACKGROUND

§.§ ENADE

Established in 2004 by the Brazilian Ministry of Education, the Exame Nacional de Desempenho dos Estudantes (ENADE) is a federal government exam that assesses the academic performance of undergraduate students in Brazil, fostering an environment of continuous improvement in educational standards <cit.>. ENADE conducts annual assessments targeting students who are nearing the completion of their academic programs. The exam committee defines the knowledge areas to be evaluated each year, with individual areas having their graduating students assessed every three years.

The exam covers various disciplines—from computer science and mathematics to history and philosophy—and encompasses bachelor, teaching, and technology degrees. The exam consists of two primary components: the first, the general education component, common across various disciplines, assesses competencies and skills developed by students and their general knowledge relating to Brazilian and global realities. The second component, the specific knowledge component, is tailored to each discipline, evaluating the mastery of knowledge and skills expected for the discipline's professional profile <cit.>. Our study focuses on the specific knowledge component of the ENADE 2021 BCS exam <cit.>.

§.§ 2021 Bachelor in Computer Science Exam

ENADE's BCS exam evaluates core computer science principles and concepts such as algorithms, programming, operating systems, software engineering, artificial intelligence, and distributed systems. At the time of writing, the BCS exam had last been conducted in November 2021 with 10,119 participants from over 300 institutions across all five regions of Brazil <cit.>.

§.§.§ Questions and Statistics

The 2021 BCS exam comprises three open questions and 27 multiple-choice questions. Tables <ref> and <ref> present the number, subject, modality (text or visual), and reasoning strategy (requiring direct or indirect answers) for the open and multiple-choice questions, respectively. Additionally, Table <ref> lists the mean, median, and standard deviation scores (out of 100) for the open questions. Table <ref> includes additional statistics for the multiple-choice questions, such as the percentage of correct answers, facility level, point-biserial correlation, and discrimination level. Two questions, 29 and 33 (highlighted in red in Table <ref>), were invalidated after the exam, possibly due to poor construction. Note that questions 01 and 02, as well as questions 06–08, are not listed in these tables as they are general knowledge questions not related to computer science.

The facility level classifies the percentage of correct answers for each multiple-choice question into a hierarchy of levels, from Very Easy ($\geq 0.86$) to Very Hard ($\leq 0.15$), as outlined in Table <ref>. The discrimination level classifies the point-biserial correlation <cit.> between participants' performance on each question and their overall performance in the exam into another hierarchy of levels, from Very Good ($\geq 0.40$) to Weak ($\leq 0.19$), as shown in Table <ref>. The point-biserial correlation is commonly used in academic exams to measure the relationship between a student's overall exam performance (a continuous variable) and their response to a specific question (a dichotomous variable, correct or incorrect) <cit.>. A higher point-biserial correlation for a question indicates effective discrimination between students who score high overall and those who score low. Questions with very low or negative point-biserial correlation values may be seen as excessively difficult or potentially flawed, as they do not align with the general performance trends of the examinees and could be considered for exclusion from scoring.

ENADE 2021 BCS open questions' features and scores (out of 100).

Strategy     2*[c]Mean
Score     2*[c]Median
Score     2*[c]Standard

No.     1cSubject     Modality

03     Theory of Computing         Visual       Direct     29.3     25.0     34.4

04     Computer Architecture       Visual       Direct      9.8      0.0     22.8

05     Algorithms                  Visual       Direct      6.6      0.0     17.7

4lAgregated score     15.2     8.3     25.0

ENADE 2021 BCS multiple-choice questions' features and statistics. Questions invalidated by the ENADE committee after the exam are highlighted in light red. Questions excluded from the final score due to statistical discrepancies are highlighted in light yellow.

Strategy     2*[c]% Correct
Answers     2*[c]Facility
Level     2*[c]Point-Biserial
Correlation     2*[c]Discrim.

No.     1cSubject     Modality

09     Operating Systems           Text     Direct     0.41     Medium     0.24     Medium

10     Programming                 Text     Direct     0.40      Hard     0.40     Very Good

11     Artificial Intelligence     Visual     Indirect     0.60      Easy     0.47     Very Good

12     Comp. in Society            Text     Indirect     0.14      Very Hard     0.04     Weak

13     Software Engineering        Text     Direct     0.20      Hard     -0.04     Weak

14     Computer Architecture       Text     Indirect     0.25     Hard     0.26     Medium

15     Software Engineering        Text     Indirect     0.51     Medium     0.29     Medium

16     Computer Architecture       Visual     Direct     0.63     Easy     0.37     Good

17     Operating Systems           Visual     Indirect     0.12     Very Hard     -0.01     Weak

18     Artificial Intelligence     Text     Indirect     0.42     Medium     0.31     Good

19     Human-Computer Inter.       Text     Indirect     0.38     Hard     0.26     Medium

20     Programming                 Text     Indirect     0.35     Hard     0.53     Very Good

21     Computer Networks           Visual     Direct     0.10     Very Hard     -0.08     Weak

22     Information Systems         Visual     Direct     0.53     Medium     0.38     Good

23     Programming                 Visual     Direct     0.45     Medium     0.28     Medium

24     Computer Security           Text     Indirect     0.28     Hard     0.33     Good

25     Distributed Systems         Text     Indirect     0.19     Hard     -0.05     Weak

26     Web Development             Text     Indirect     0.15     Very Hard     0.23     Medium

27     Performance Analysis        Visual     Direct     0.51     Medium     0.45     Very Good

28     Computer Architecture       Visual     Indirect     0.34     Hard     0.34     Good

29     Image Processing            Visual     Indirect

30     Compilers                   Text     Indirect     0.45     Medium     0.37     Good

31     Theory of Computing         Visual     Indirect     0.23     Hard     0.37     Good

32     Algorithms                  Text     Indirect     0.17     Hard     0.02     Weak

33     Programming                 Text     Direct

34     Graph Theory         Visual     Direct     0.23     Hard     0.20     Medium

35     Distributed Systems         Visual     Direct     0.51     Medium     0.48     Very Good

ENADE 2021 BCS classification of multiple-choice questions into facility levels.

% Correct Answers     Facility Level

$\geq 0.86$     Very Easy

$0.61$ to $0.85$     Easy

$0.41$ to $0.60$     Medium

$0.16$ to $0.40$     Hard

$\leq 0.15$     Very hard

ENADE 2021 BCS classification of multiple-choice questions into discrimination levels.

Point-Biserial Correlation     Discrimination Level

$\geq 0.40$     Very Good

$0.30$ to $0.39$     Good

$0.20$ to $0.29$     Medium

$\leq 0.19$     Weak

The ENADE 2021 committee utilized the calculated facility and discrimination levels to exclude six multiple-choice questions from the final score, namely questions 12, 13, 17, 21, 25, and 32 (highlighted in light yellow in Table <ref>) <cit.>. These questions were classified as Hard or Very Hard at the facility level and Weak at the discrimination level.

Appendix <ref> presents the image and English transcription of the three open questions and a sample of six multiple-choice questions of the ENADE 2021 BCS exam.
[In the PDF version of the paper, all question numbers in Tables <ref> and <ref> are hyperlinked to their corresponding page in the paper's companion repository, where readers can view the questions' original images, ChatGPT-4 prompts and responses, and the independent expert evaluations (when available) in both English and Portuguese.]

§.§.§ Scores

ENADE 2021 BCS specific knowledge component scores.

Mean     2*[c]Exam
Median     2*[c]Standard

Section     Weight

Open questions                0.15     15.2      8.3     19.5

Multiple-choice questions     0.85     40.5     42.1     16.5

Final  (weighed) score                            36.7     35.8     15.8

ENADE 2021 calculates the final score of the specific knowledge component by combining the scores of the open and multiple-choice questions, with the former weighting 15% of the final score and the latter weighting 85% <cit.>. The open question score is calculated as the average score of the three open questions. The multiple-choice question score is calculated as the proportion of correct responses to the remaining 19 out of the initial 27 multiple-choice questions. The final score is then converted to a scale of 0 to 100, with 0 being the lowest possible score and 100 being the highest.

Table <ref> presents the numbers for the open questions, multiple-choice questions, and final scores for the 2021 BCS exam. The mean and median scores for the open questions were 15.2 and 8.3, respectively, with a standard deviation of 19.5. The mean and median scores for the multiple-choice questions were 40.5 and 42.1, respectively, with a standard deviation of 16.5. Lastly, the mean and median scores for the final score were 36.7 and 35.8, respectively, with a standard deviation of 15.8. These numbers indicate that the exam's open questions were particularly challenging for the participants. In contrast, the multiple-choice questions were relatively more accessible, even though the average score in this section was still below the 50% threshold. This result aligns with the observation that nearly half of the 19 scored multiple-choice questions were classified as Hard or Very Hard, as shown in Table <ref>.

§ STUDY METHOD

Our study evaluates ChatGPT-4 Vision's performance on the specific knowledge component of the ENADE 2021 BCS exam. To this end, we provided the model with the exam's questions in image format by taking a high-resolution screen capture of each question's page from the exam's official PDF document. Since the exam's questions and response standard are in Portuguese, we conducted the evaluation experiment in Portuguese, including the model's prompts and responses, and the expert assessments of conflicting answers. We translated all materials into English for this paper. Although we have not replicated the entire experiment using the English translations of the questions, our experience with ChatGPT-Vision and its successors is that these OpenAI models are as good at answering questions in Portuguese as they are in English.

§.§ ChatGPT Versions Used in the Study

We used ChaGPT-4 Vision <cit.> to answer ENADE's 2021 BCS exam questions and ChatGPT-4 Turbo <cit.>, to reflect on ChaGPT-4 Vision's performance on each question. The reason we have not used ChatGPT-4 Turbo from the start is that it was only released after we completed the question-answering phase of the research using ChatGPT-4 Vision.

ChatGPT-4 Vision, released in September 2023, expands upon the text-based capabilities of ChatGPT-4 by incorporating image analysis and understanding <cit.>. It features a context window of 32k tokens and an updated knowledge base until January 2022. Since the ENADE 2021 exam was administered on November 14th, 2021, and the exam key and response standard were made publicly available on March 18th, 2022, it is reasonable to assume ChatGPT-4 Vision's knowledge base did not include any information from the exam.

ChatGPT-4 Turbo, which was launched in November 2023, features a significantly larger context window (128k tokens) compared to GPT-4, allowing it to handle complex and lengthy prompts with greater accuracy and coherence <cit.>. Additionally, ChatGPT-4 Turbo incorporates Retrieval Augmented Generation (RAG) <cit.>, which integrates external information to enhance the model's responses. Furthermore, ChatGPT-4 Turbo boasts DALL-E 3 integration for text-to-image generation, text-to-speech capabilities with multiple voice options, and an updated knowledge base extending to April 2023.

§.§ Evaluation Procedure

Question-answering evaluation procedure.

Our evaluation procedure for each exam question involved a multi-step process, as illustrated in the activity diagram shown in Figure <ref>. Initially, we captured an image of the question from the exam's PDF file. We then provided this image along with a simple (zero-shot) Chain-of-Thought (CoT) prompt <cit.> to ChatGPT-4 Vision, asking the model to answer the question depicted in the image and to detail its reasoning step by step. This prompting approach was chosen because it enables the model to demonstrate its ability to understand and solve complex problems without prior specific training on the exam's content, relying instead on its general pre-trained knowledge base. Moreover, by incorporating CoT prompts, the evaluation process gains transparency and insight into the model's reasoning, allowing the assessment not just of the correctness of the final answers but also of the logical processes used to arrive at them. This is especially valuable in an educational context, where understanding the steps of problem-solving is as important as the solution itself. Appendix <ref> shows our study's zero-shot CoT prompt templates for the ENADE 2021 BCS open and multiple-choice exam questions.

After providing ChatGPT-4 Vision with the appropriate prompt, we compared the model's generated response against the exam's official answer standard (for open questions) or answer key (for multiple-choice questions). When the model's response did not align with the official answer, we prompted the model to reassess its response in light of the official answer and self-reflect on its possible reasoning mistakes. Essentially, our reassessment prompts inform the model that its initial answer is inconsistent with the official answer and instruct it to ponder whether to maintain or change its initial answer given the new information. Appendix <ref> shows our reassessment prompt templates for open and multiple-choice questions. We interactively analyzed the model's subsequent responses and prompted it for further clarification when the model's decision on the correct answer was unclear or incoherent.

Following the clarification stage, if discrepancies persisted between ChatGPT-4 Vision's responses and the official answer, the conflicting answers were escalated to an independent expert panel for review. The panel comprised two or more computer science researchers from multiple Brazilian universities with known expertise on the question's subject. We provided each panel member with the question, official answer, and ChatGPT-4 Vision's responses. They were individually asked to assess the correctness of the model's responses and provide their reasoning for their decisions without knowing the identity and opinions of their peers. The panel members were also asked to indicate if the question was unclear or had multiple or no possible correct answers. We then compiled and classified the panel's assessments as conclusive or inconclusive. Conclusive assessments were those where all panel members agreed on the question's correct answer. Inconclusive assessments were those where the panel members disagreed on the correctness of the question's answer or indicated that the question was unclear or had no possible answer.

In the final evaluation step, we provided ChatGPT-4 Turbo with the question image, ChatGPT-4 Vision's prompts and responses, and the expert assessments (when available) for each question in which the model did not initially agree with the official response and prompted it to reflect on the entire question-answering process. In particular, we asked the model to identify the possible challenges the AI faced in answering the question, the cause of its errors, and whether these errors could be mitigated by using more sophisticated prompt engineering techniques or would be inherent to the limitations of current LLMs. This final reflective assessment aimed to provide further insights into the AI's reasoning and decision-making processes, contributing to a deeper understanding of the model's multimodal capabilities and limitations.

§.§ Question Scoring

We scored ChatGPT-4 Vision's responses to the three open questions using the exam's official answer standard as a reference. We assigned scores from 0 to 100 to each response, where 0 indicated an utterly incorrect response, and 100 indicated an entirely correct response. We scored partially correct responses proportionally, based on the number of correct items or assertions they contained. For example, a response with two correct assertions out of three would be scored 66.6.

We scored ChatGPT-4 Vision's responses to the multiple-choice questions proportionally to the number of responses that agreed with the exam's answer key. For reference, we scored all 25 valid multiple-choice questions, including the six questions the exam committee excluded from the final score. However, we only consider the 19 valid non-excluded questions for computing the model's final score.

We only scored the model's initial response to each question and ignored its subsequent reassessments or the expert panel's reviews in determining the score value. For example, if the model initially answered a question incorrectly but later changed its response after reassessment, we only considered the initial incorrect response for scoring purposes.

§.§ Experimental Setup

During the question-answering sessions, we manually interacted with both ChatGPT-4 Vision and ChatGPT-4 Turbo using their respective web interfaces. We did not use an automated script for this task because ChatGPT-4 Vision's model was not available via API at the time the experiment was conducted. We used the models' default settings with no parameter fine-tuning. We saved all the model's prompts and responses and the expert assessments as text files in their original Portuguese version. We translated them into English using the GPT-4 Turbo API via a customized Python script with minor manual revisions and improvements. We also translated the exam's official answer standard for open questions into English using the same script.

§ RESULTS

We now present the results for each of the five research questions outlined in Section <ref>. We first analyze ChatGPT-4 Vision's performance in answering the ENADE 2021 BCS exam questions, focusing on the open and multiple-choice questions. We then examine the model's response changes when confronted with disagreeing official answers and compare its final score with the exam participants. Finally, we discuss the main challenges ChatGPT-4 Vision faced in answering the exam questions and reflect on its reasoning and decision-making processes.

§.§ RQ1: Performance in Open Questions

Human participants and ChatGPT-4 Vision's scores for open questions.

3cHuman Participants     3cChatGPT-4 Vision

[0.5pt](r0.5em)4-6 [0.5pt](l0.5em)7-9
Strategy     2*[c]Mean
Score     2*[c]Median
Score     2*[c]Standard
Deviation         2*[c]Imp./
Mean     2*[c]Top

No.     Modality                     Score

03     Visual       Direct     29.3     25.0     34.4     62.5     2.1     22

04     Visual       Direct      9.8      0.0     22.8     50.0     5.1     17

05     Visual       Direct      6.6      0.0     17.7     50.0     7.6     11

2rAgregated score         15.2     8.3     15.8     54.2     3.6     8

Table <ref> shows ChatGPT-4 Vision's score (out of 100), improvement over mean, and top percentile relative to the participants' mean, median, and standard deviation scores for the three open questions. The improvement over mean metric for ChatGPT-4 Vision is calculated by comparing the model's score to the mean score of the participants, specifically by dividing the model's score by the mean participant score. The table shows that ChatGPT-4 Vision failed to provide entirely correct answers to all three questions, with an average score of 54.2. Nevertheless, the model still performed much better than most participants in all three questions, significantly more so for questions 04 and 05, where the model's score was five to seven times higher than the participants' mean scores. These results place the model's scores within the top 10-20 best score percentiles for all three open questions, with its average score being within the top 8 best average score percentile.

§.§ RQ2: Performance in Multiple-Choice Questions

ChatGPT-4 Vision's results for multiple-choice questions.

4cChatGPT-4 Vision Performance

Strategy     2*[c]Facility
Level     2*[c]Disc.
Level                     2*[c]Score
(% Corr.)

Scope     Modality                 Total     Corr.     Incorr.     Confl.

questions     all     all     all      all     19     13     5     1     68.4

Text     all     all     all     11     7     3     1     63.6

Visual     all     all     all     8     6     2     0     75.0

all     Direct     all     all     8     6     2     0     75.0

all     Indirect     all     all     11     7     3     1     63.6

all     all     Easy     all     2     2     0     0     100.0

all     all     Medium     all     8     6     1     1     75.0

all     all     Hard     all     8     5     3     0     62.5

all     all     Very Hard     all     1     0     1     0     0.0

all     all     all     Very Good     5     5     0     0     100.0

all     all     all     Good     7     5     1     1     71.4

all     all     all     Medium     7     3     4     0     42.9

questions     all     all     all     Weak     6     2     2     2     33.3

Text     all     all     Weak     4     1     1     2     25.0

Visual     all     all     Weak     2     1     1     0     50.0

all     Direct     all     Weak     2     0     1     1     0.0

all     Indirect     all     Weak     4     2     1     1     50.0

all     all     Hard     Weak     3     1     0     2     33.3

all     all     Very Hard     Weak     3     1     2     0     33.0

Table <ref> shows ChatGPT-4 Vision's performance for all valid (scored and excluded) multiple-choice questions. The model's results are shown for all question types of each category and also grouped row-wise by modality, reasoning strategy, facility level, and discrimination level.

ChatGPT-4 Vision's overall score, considering all 19 scored questions, was 68.4, with 13 correct answers, five incorrect answers, and one conflicting answer where the experts disagreed on the correct answer. The model answered correctly 75.0% of questions with visual elements and 63.6% of text-only questions. The model showed a similar scoring pattern for questions requiring direct and indirect answers, with 75.0% and 63.6% correct answers, respectively. These results indicate that the model performed slightly better in questions with visual elements and direct answers than in text-only questions or those requiring multi-step answers.

The model's performance degraded as the questions' facility level increased from Easy to Very Hard, with 100.0%, 75.0%, 62.5%, and 0.0% correct answers, respectively. We observed a similar degradation pattern in the model's performance when grouping questions by discrimination level, where it answered correctly 100.0%, 71.4%, and 42.9% of questions classified as Very Good, Good, and Medium, respectively.

The model's performance in the six excluded questions was significantly lower than in the scored questions, with only 33.3% correct answers compared to 68.4% in the first category. Even when grouping the excluded questions by modality, reasoning strategy, and facility and discrimination levels, the model did not get more than half of the questions correct in either group. These results suggest that ChatGPT-4 Vision's performance in the multiple-choice questions is generally aligned with the overall performance of the exam's participants, with the model scoring higher in questions classified as Medium to Easy at the facility level and Medium to Very Good at the discrimination level.

We applied Spearman's rank correlation coefficient <cit.> to test this hypothesis by calculating the correlation between the AI model's performance and each question's facility and discrimination levels. This method was chosen for its robustness in handling non-parametric data and its effectiveness in measuring the strength and direction of association between two ranked variables. The AI model's answers were numerically coded to facilitate the analysis: correct answers as 1 and 0 otherwise. The facility level and the discrimination level were numerically represented by the percentage of correct answers and the Point-Biserial correlation values, respectively.

Our Spearman's rank correlation analysis indicated a coefficient of 0.555 ($p = 0.004$) between the AI model's performance and the facility level. The analysis also yielded a Spearman's correlation coefficient of 0.584 ($p = 0.0022$) when comparing the AI model's performance with the Point-Biserial Correlation values. These results reveal statistically significant positive correlations ($p < 0.05$) between the AI model's performance and the exam questions' facility and discrimination levels, as evidenced by the $p$-values reported. Thus, the correlation results support our claim that ChatpGPT-4 Vision is more adept at answering questions that are both easier for students and more discriminative regarding student performance.

§.§ RQ3: Performance in Questions with Disagreeing Answers

ChatGPT-4 Vision's responses and expert panel assessments for multiple-choice questions with initially disagreeing responses. Questions excluded from the final score due to a weak point-biserial correlation are highlighted in light yellow. Questions where the expert panel disagreed on the correct answer are highlighted in boldface.

2cChatGPT-4 Vision     3cIndependent Expert Panel

[0.5pt](r)3-4 [0.5pt]5-7
Key     2*[c]Initial
Response     2*[c]Revised
Response                 2*[c]Final

No.                 Expert  #1     Expert #2     Expert #3

09     E     C     C     E     E         Incorrect

12     B     D     B                 Incorrect

13     E     D     D     D     Invalid     E     Inconclusive

14     C     B     B     C     C     C     Incorrect

18     E     D     D     Invalid     A     E     Inconclusive

21     E     Invalid     E                 Incorrect

25     B     E     D     Invalid     B     Invalid     Inconclusive

26     A     E     A                 Incorrect

31     E     D     Invalid     E     E         Incorrect

34     C     Invalid     C                 Incorrect

Table <ref> shows ChatGPT-4 Vision's initial and revised responses and the expert panel's assessments (when available) for the ten multiple-choice questions where the model initially disagreed with the official answer. This table reveals that the model initially disagreed and subsequently agreed with the official answer key after reassessment in four questions (12, 21, 26, and 34). These four questions were classified as Hard (21) or Very Hard (12, 26, and 34), with two (12 and 21) being excluded from the final score due to a Weak discrimination level (see Table <ref>). Notably, for questions 21 and 34, ChatGPT-4 Vision initially suggested invalidation but revised its stance to align with the official answer upon reassessment.

Out of the six questions where the model's disagreement with the answer key persisted after reassessment, the expert panel unanimously agreed with the answer key in three questions (09, 14, and 31) and did not agree on the correct answer for the remaining three questions (13, 18, 25). The three questions where the expert panel agreed on the correct answer were classified from Medium (09) to Hard (14 and 31) at the facility level and from Medium (09 and 14) to Good (31) at the discrimination level. On the other hand, two of the three questions where the expert panel disagreed on the correct answer (13 and 25) were classified as Hard with a Weak discrimination level, thus being among the six excluded from the final score. Question 18 is the exception from this group since it was classified as Medium at the facility level and Good at the discrimination level. This discrepancy means that the AI model and the expert panel struggled to provide a definitive answer to a question that a significant fraction of the exam's participants did not find particularly difficult.

The results in Table <ref> also reinforce the claim that the model's performance tends to reflect that of the human participants in the multiple-choice section of the exam, with both the model and the expert panel struggling to provide a definitive answer to some questions classified as Hard or Very Hard with a Weak discrimination level.

§.§ RQ4: Final Score and Relative Performance

ChatGPT-4 Vision's partial and final scores for the open and multiple-choice questions of the ENADE 2021 BCS exam.

3cChatGPT-4 Vision

Mean     2*[c]Exam
Median     2*[c]Standard
Deviation         2*[c]Imp./
Mean     2*[c]Top

Scope     Weight                 Score

Open questions                0.15     15.2      8.3     19.5     54.2     3.6     8

Multiple-choice questions     0.85     40.5     42.1     16.5     68.4     1.7

Final (weighed) score                            36.7     35.8     15.8     66.3     1.8     10

Table <ref> shows ChatGPT-4 Vision's partial and final scores for the specific knowledge component of the ENADE 2021 BCS exam. Considering both open and multiple-choice questions, the model's final (weighed) score was 66.3. This score, representing an 80% improvement over the participants' mean final score of 36.7, places ChatGPT-4 Vision within the top 10 percentile of exam scores <cit.>. This result indicates that ChatGPT-4 Vision performed significantly better than the average human participant in the ENADE 2021 BCS exam.

§.§ RQ5: Challenges in Answering Open and Multiple-Choice Questions

Challenges ChatGPT-4 Vision faced in answering ENADE 2021 BCS questions.

1cChallenge     1cError Type     1cExample     1cQuestions with Error

Question Interpretation     Inconsistent Responses     The AI model produces inconsistent responses due to vague or ambiguous statements in the question.     13, 18, 25

Non-Compliance with Guidelines     The AI model ignores specific directives or constraints, resulting in an incorrect response that does not align with the question's guidelines.      31

Logical Reasoning     Evaluation Leniency     The AI model identifies yet overlooks “minor” flaws, validating a false or partly correct assertion as true.     12, 14, 21

Insufficient Domain Knowledge     The AI model applies an incorrect or incomplete premise, causing a false or partly correct assertion to be misidentified as true.     09, 13, 14, 26

Incorrect Algorithmic Reasoning     The AI model fails to simulate the execution of an algorithm accurately.     05, 31

Incorrect Multi-Step Reasoning     The AI model correctly evaluates the validity of individual assertions but then chooses a “true” answer that includes or depends upon one or more false assertions.     03, 12, 18, 25, 26

Visual  Acuity     Misidentification of Visual Elements     The AI model misidentifies one or more labels in a table or diagram.     03, 31, 34

Lack of Domain-Specific Visual Output     The AI model cannot generate a response using the required visual formalism.     04

Based on our analysis of ChatGPT-4 Vision's initially incorrect responses, the expert assessments (when available), and ChatGPT-4 Turbo's responses, we identified three challenges categories: Question Interpretation, Logical Reasoning, and Visual Acuity. We further classified the most common errors generated by the model when facing each of these challenges into eight error types, as described and illustrated in Table <ref>. Question interpretation errors are caused by the model's inability to correctly interpret the questions' statements. Logical reasoning errors arise from the model's flawed reasoning processes when answering the assertions in the questions. Finally, visual acuity errors are caused by the model's inability to correctly identify and process the questions' visual elements. These challenge categories and error types are not mutually exclusive, as the manifestation of errors related to one challenge may lead to the manifestation of other errors related to another challenge. For example, the model's misidentification of a visual element in a question may trickle down to an incorrect multi-step reasoning process.

The last column of Table <ref> reveals that the most common error types generated by ChatGPT-4 Vision in answering the ENADE 2021 BCS questions were related to logical reasoning, particularly incorrect multi-step reasoning (in five questions), insufficient domain knowledge (in four questions), and evaluation leniency (in three questions). Inconsistent responses and misidentification of visual elements (in three questions each) were also common error types. Other less frequent error types included incorrect algorithmic reasoning and non-compliance with guidelines (in two questions), and lack of domain-specific visual output (in one question). Appendix <ref> shows the model's conversations for the three open and six multiple-choice questions illustrating most of these challenges categories and error types. The model's full conversations for all the questions are available in the paper's companion repository.

These results provide several insights into ChatGPT-4 Vision's multimodal reasoning capabilities. For example, the model's frequent incorrect multi-step reasoning and insufficient domain knowledge suggest its ability to reason about complex, multi-step problems in computer science is still limited. Moreover, the model's multiple evaluation leniency issues indicate its ability to evaluate the validity of assertions is not yet as strict as that of the human experts. Finally, the model's tendency to misidentify visual elements and lack of domain-specific visual output show that its visual acuity is not yet as developed as its text-based reasoning capabilities. This limitation is particularly critical given the frequent use of visual data in academic assessments in computer science and other science and engineering fields.

Another relevant result was the model's struggle with vague or ambiguous statements, particularly in questions where the expert panel did not agree on the correct answer, such as in questions 13, 18, and 25. This result suggests that some of the exam's questions may have been poorly constructed, leading to confusion and disagreement among the model and the human experts. Indeed, some experts explicitly mentioned the low quality of some ENADE questions, with one expert being even more forthright when assessing the quality of one particular question: “I'm sad that such a poorly formulated question was used to evaluate computer science students.”

It is worth noting that these challenges did not manifest uniformly across the open and multiple-choice questions. For example, the model's issues with incorrect multi-step reasoning were more frequent in the multiple-choice questions with indirect answers. At the same time, the lack of domain-specific visual output is only an issue for open questions requiring visual responses. Moreover, some challenges were not uniformly distributed across questions with similar characteristics. For example, the model's issues with misidentifying visual elements did not occur in all questions containing figures or diagrams. These results suggest that the model's challenges are highly context-dependent and may require tailored solutions to address them effectively.

§ DISCUSSION

Given the results presented in the previous section, we discuss the implications of ChatGPT-4 Vision's performance in the ENADE 2021 BCS exam and the insights gained from the model's challenges in answering the exam's questions. We also discuss the limitations of our study and future research directions.

§.§ Implications of ChatGPT-4 Vision's Performance

The performance of ChatGPT-4 Vision on the ENADE 2021 BCS exam highlights significant advancements in AI's ability to engage with complex, multimodal academic content. The AI's overall score, comparable to that of the top-scoring human participants, particularly its superior performance in open questions and its competitive edge in multiple-choice questions with visual elements, underscores the potential of multimodal AI in educational settings. These results indicate the evolving capabilities of AI systems to process and analyze information in a manner previously thought to be exclusive to human cognition <cit.>. Such capabilities could be harnessed to support various educational tasks, such as developing advanced tutoring systems, aiding educational assessment, and providing personalized learning experiences <cit.>. The model's ability to reason about complex, multimodal problems in computer science is particularly promising, given the growing demand for AI systems that can understand and reason about multimodal data in this education domain <cit.>.

§.§ Insights from ChatGPT-4 Vision's Challenges

The challenges faced by ChatGPT-4 Vision, particularly in logical reasoning, question interpretation, and visual acuity, provide valuable insights into the current limitations of AI in fully understanding and responding to complex multimodal academic problems. The model's difficulties with incorrect multi-step reasoning and insufficient domain knowledge reinforce similar findings reported by prior studies (e.g., <cit.>). Nevertheless, these insights provide valuable guidance for future improvements to the model's reasoning capabilities, perhaps through more sophisticated training datasets or advanced reasoning algorithms <cit.>. The model's challenges with visual acuity also highlight the importance of developing AI systems that can accurately interpret and integrate multi-modal inputs <cit.>. Furthermore, the model and expert panel's struggle with vague or ambiguous statements could inform educational assessment design, helping improve the clarity and precision of questions during exam elaboration. Finally, the finding that the model's performance is correlated with the difficulty and discriminative power of the exam's multiple-choice questions suggests that ChatGPT-4 Vision's performance could be used as a proxy for question quality and complexity, thus providing a valuable tool for exam creators to test and refine their questions before administering the exam.

§.§ Study Limitations and Future Directions

Our study has certain limitations that are important to acknowledge. First, the reliance on zero-shot CoT prompts may have limited the AI's ability to fully engage with the multi-step reasoning required by some exam questions, potentially underestimating its true reasoning capabilities. Future studies should experiment with advanced prompting techniques like Self-Consistency <cit.>, Tree-of-Thoughts <cit.>, and Visualization-of-Thought <cit.>, which may enable the AI to navigate complex reasoning sequences more effectively, potentially revealing deeper insights into its problem-solving abilities.
The fact that ChatGPT-4 Vision was able to reassess and correct its initial response to several questions during our experiment suggests that new prompting techniques tailored for the exam's format and style have the potential to further enhance the model's accuracy.
Second, our evaluation protocol involved asking the AI model to answer each question only once. This design choice limits our ability to analyze the consistency of the AI's performance or its variability in responding to the same question across multiple attempts. A more comprehensive evaluation could involve multiple attempts to answer each question, allowing for a more nuanced understanding of the AI's performance and the potential for improvement over time <cit.>.
Third, our study did not compare ChaGPT-4 Vision's ENADE 2021 BCS exam performance with other multimodal generative AI models like Google's recently released Gemini family of models <cit.>. A systematic comparative analysis involving other multimodal generative AI models could provide valuable insights into the unique capabilities and limitations of each model in handling educational content. This approach could also help identify areas for cross-model improvements and innovations <cit.>.
Finally, our study did not incorporate traditional exam validation methods, such as pre-assessment moderation, where human experts review and validate exam questions to ensure their quality, fairness, and alignment with educational objectives <cit.>. Incorporating such traditional methods could provide a valuable benchmark against which the AI's performance can be measured. Future research could explore this comparison to better understand the potential and limitations of AI in educational assessments.

§ CONCLUSION

ChatGPT-4 Vision's performance on Brazil's ENADE 2021 Bachelor in Computer Science exam offers a compelling glimpse into the future of generative AI technologies in educational assessment. ChatGPT-4 Vision not only matched but, in some cases, exceeded the top human scores, particularly in handling questions with visual components and complex problem-solving scenarios. While showcasing the model's impressive multimodal reasoning capabilities, the results particularly underline the need for improvements in the AI's handling of complex visual data interpretations and its ability to apply domain-specific knowledge accurately. Future research should focus on refining multimodal generative AI models to enhance their interpretive, reasoning, and vision capabilities. This includes exploring adaptive prompt techniques that dynamically adjust based on the AI's response quality, aiming to steer AI models more effectively through complex reasoning pathways. Additionally, employing AI models in simulated testing environments can help identify biases, ambiguities, or inconsistencies in exam questions, thereby enhancing their fairness and effectiveness. Successfully addressing these challenges will not only advance AI's capabilities in educational assessments but also pave the way for broader applications in personalized learning, automated tutoring systems, and interactive educational content.

§ ACKNOWLEDGMENTS

The author is grateful to the following computer science researchers in Brazil who kindly agreed to participate in this study: Alfredo Goldman (University of São Paulo), Aline Marins Paes Carvalho (Federal Fluminense University), Altigran Soares da Silva (Federal University of Amazonas), Avelino Francisco Zorzo (Pontifical Catholic University of Rio Grande do Sul), Carla Taciana Lima Lourenco Silva Schuenemann (Federal University of Pernambuco), Carlos Alberto Kamienski (Federal University of ABC), Celso Alberto Saibel Santos (Federal University of Espírito Santo), Francisco Vilar Brasileiro (Federal University of Campina Grande), Itana Maria de Souza Gimenes (State University of Maringá), João Fernando Lima Alcântara (Federal University of Ceará), Leopoldo Motta Teixeira (Federal University of Pernambuco), Luiz Fernando Bittencourt (State University of Campinas), Nelson Souto Rosa (Federal University of Pernambuco), Ricardo Marcondes Marcacini (University of São Paulo), Raimundo da Silva Barreto (Federal University of Amazonas), and Tayana Uchôa Conte (Federal University of Amazonas).

The generative AI tools ChatGPT-4, GitHub Copilot, and Gemini Advanced were used to prepare this paper, including revising and improving several sections of the manuscript, creating and formatting tables and figures, and generating and executing code for data analysis.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Almeida et al(2024)]
authorpersonThales Sales Almeida, personHugo
Abonizio, personRodrigo Nogueira, and personRamon
Pires. year2024.
Sabiá-2: A New Generation of Portuguese Large
Language Models.
journalarXiv preprint arXiv:2403.09887

[Bloxham et al(2016)]
authorpersonSue Bloxham, personClair
Hughes, and personLenore Adie.
What's the point of moderation? A discussion of
the purposes achieved through contemporary moderation practices.
journalAssessment & Evaluation in Higher
Education volume41, number4 (year2016),

authorpersonDouglas G Bonett.
Point-biserial correlation: Interval estimation,
hypothesis testing, meta-analysis, and sample size determination.
journalBrit. J. Math. Statist. Psych.
volume73 (year2020), pages113–144.

[Bordt and von Luxburg(2023)]
authorpersonSebastian Bordt and
personUlrike von Luxburg. year2023.
ChatGPT Participates in a Computer Science Exam.
journalarXiv preprint arXiv:2303.09461

[Bubeck et al(2023)]
authorpersonSébastien Bubeck,
personVarun Chandrasekaran, personRonen Eldan,
personJohannes Gehrke, personEric Horvitz,
personEce Kamar, personPeter Lee,
personYin Tat Lee, personYuanzhi Li,
personScott Lundberg, et al
Sparks of Artificial General Intelligence: Early
experiments with GPT-4.
journalarXiv preprint arXiv:2303.12712

[Chen et al(2023)]
authorpersonLingjiao Chen, personMatei
Zaharia, and personJames Zou.
How is ChatGPT's behavior changing over time?
journalarXiv preprint arXiv:2307.09009

[Dai et al(2023)]
authorpersonWei Dai, personJionghao Lin,
personHua Jin, personTongguang Li,
personYi-Shan Tsai, personDragan Gašević,
and personGuanliang Chen. year2023.
Can Large Language Models Provide Feedback to
Students? A Case Study on ChatGPT. In booktitle2023 IEEE
International Conference on Advanced Learning Technologies (ICALT). IEEE,

[Espejel et al(2023)]
authorpersonJessica López Espejel,
personEl Hassane Ettifouri, personMahaman
Sanoussi Yahaya Alassan, personEl Mehdi Chouham, and
personWalid Dahhane. year2023.
GPT-3.5, GPT-4, or BARD? Evaluating LLMs reasoning
ability in zero-shot setting and performance boosting through prompts.
journalNatural Language Processing Journal
volume5 (year2023), pages100032.

[Evagorou et al(2015)]
authorpersonMaria Evagorou, personSibel
Erduran, and personTerhi Mäntylä.
The role of visual representations in scientific
practices: from conceptual understanding and knowledge generation to 'seeing'
how science works.
journalInternational journal of STEM education
volume2, number1 (year2015),

[Frenkel and Emara(2023)]
authorpersonMatthew Frenkel and
personHebah Emara. year2023.
ChatGPT & Mechanical Engineering: Examining
performance on the FE Mechanical Engineering and Undergraduate Exams.
journalarXiv preprint arXiv:2309.15866

[Gemini Team(2023)]
authorpersonGemini Team.
Gemini: A Family of Highly Capable Multimodal
journalarXiv preprint arXiv:2312.11805

authorpersonGoogle. year2023.

authorpersonInep. year2022.
titleRelatório Síntese de Área: Ciência da
Computação (Bacharelado / Licencitatura).

authorpersonInep. year2023.
titleExame Nacional de Desempenho dos Estudantes

[Jalil et al(2023)]
authorpersonSajed Jalil, personSuzzana
Rafi, personThomas D LaToza, personKevin Moran,
and personWing Lam. year2023.
ChatGPT and Software Testing Education: Promises
& Perils. In booktitle2023 IEEE International Conference
on Software Testing, Verification and Validation Workshops (ICSTW). IEEE,

[Joshi et al(2024)]
authorpersonIshika Joshi, personRitvik
Budhiraja, personHarshal Dev, personJahnvi Kadia,
personMohammad Osama Ataullah, personSayan Mitra,
personHarshal D Akolekar, and personDhruv Kumar.
ChatGPT in the Classroom: An Analysis of Its
Strengths and Weaknesses for Solving Undergraduate Computer Science
Questions. In booktitle55th ACM Technical Symposium on
Computer Science Education (SIGCSE) V. 1. pages625–631.

[Joshi et al(2023)]
authorpersonIshika Joshi, personRitvik
Budhiraja, personHarshal Dev, personJahnvi Kadia,
personM Osama Ataullah, personSayan Mitra,
personDhruv Kumar, and personHarshal D Akolekar.
ChatGPT in the Classroom: An Analysis of Its
Strengths and Weaknesses for Solving Undergraduate Computer Science
journalarXiv preprint arXiv:2304.14993

[Kasneci et al(2023)]
authorpersonEnkelejda Kasneci, personKathrin
Seßler, personStefan Küchemann, personMaria
Bannert, personDaryna Dementieva, personFrank
Fischer, personUrs Gasser, personGeorg Groh,
personStephan Günnemann, personEyke
Hüllermeier, et al year2023.
ChatGPT for good? On opportunities and challenges
of large language models for education.
journalLearning and Individual Differences
volume103 (year2023), pages102274.

[Lewis et al(2020)]
authorpersonPatrick Lewis, personEthan
Perez, personAleksandra Piktus, personFabio Petroni,
personVladimir Karpukhin, personNaman Goyal,
personHeinrich Küttler, personMike Lewis,
personWen-tau Yih, personTim Rocktäschel,
et al year2020.
Retrieval-Augmented Generation for
Knowledge-Intensive NLP Tasks.
journalAdvances in Neural Information Processing
Systems (NeurIPS 2020)  volume33 (year2020),

[Li et al(2023)]
authorpersonQingyao Li, personLingyue Fu,
personWeiming Zhang, personXianyu Chen,
personJingwei Yu, personWei Xia,
personWeinan Zhang, personRuiming Tang, and
personYong Yu. year2023.
Adapting Large Language Models for Education:
Foundational Capabilities, Potentials, and Challenges.
journalarXiv preprint arXiv:2401.08664

[Liu et al(2023)]
authorpersonHanmeng Liu, personRuoxi Ning,
personZhiyang Teng, personJian Liu,
personQiji Zhou, and personYue Zhang.
Evaluating the Logical Reasoning Ability of
ChatGPT and GPT-4.
journalarXiv preprint arXiv:2304.03439

[Mahon et al(2023)]
authorpersonJoyce Mahon, personBrian
Mac Namee, and personBrett A Becker.
No More Pencils No More Books: Capabilities of
Generative AI on Irish and UK Computer Science School Leaving Examinations.
In booktitle2023 Conference on United Kingdom & Ireland
Computing Education Research (UKICER). pages1–7.

[Mao et al(2023)]
authorpersonRui Mao, personGuanyi Chen,
personXulang Zhang, personFrank Guerin, and
personErik Cambria. year2023.
GPTEval: A Survey on Assessments of ChatGPT and
journalarXiv preprint arXiv:2308.12488

[Mao et al(2024)]
authorpersonRui Mao, personGuanyi Chen,
personXulang Zhang, personFrank Guerin, and
personErik Cambria. year2024.
GPTEval: A Survey on Assessments of ChatGPT and
GPT-4. In booktitle2024 Joint International Conference on
Computational Linguistics, Language Resources and Evaluation (LREC-COLING).

authorpersonAndrew Mayne.
titleImproving GPT-4's Visual Reasoning with Prompting.

[Meyer et al(2023)]
authorpersonJesse G Meyer, personRyan J
Urbanowicz, personPatrick CN Martin, personKaren
O'Connor, personRuowang Li, personPei-Chen Peng,
personTiffani J Bright, personNicholas Tatonetti,
personKyoung Jae Won, personGraciela
Gonzalez-Hernandez, et al year2023.
ChatGPT and large language models in academia:
opportunities and challenges.
journalBioData Mining volume16,
number1 (year2023), pages20.

[Nori et al(2023)]
authorpersonHarsha Nori, personNicholas
King, personScott Mayer McKinney, personDean
Carignan, and personEric Horvitz.
Capabilities of GPT-4 on Medical Challenge
journalarXiv preprint arXiv:2303.13375

[Nunes et al(2023)]
authorpersonDesnes Nunes, personRicardo
Primi, personRamon Pires, personRoberto Lotufo,
and personRodrigo Nogueira. year2023.
Evaluating GPT-3.5 and GPT-4 Models on Brazilian
University Admission Exams.
journalarXiv preprint arXiv:2303.17003

authorpersonOpenAI. year2023a.

authorpersonOpenAI. year2023b.
titleChatGPT can now see, hear, and speak.

authorpersonOpenAI. year2023c.
titleGPT-4V(ision) system card.

authorpersonOpenAI. year2023d.
titleNew models and developer products announced at

[Pires et al(2023)]
authorpersonRamon Pires, personThales Sales
Almeida, personHugo Abonizio, and personRodrigo
Nogueira. year2023.
Evaluating GPT-4's Vision Capabilities on
Brazilian University Admission Exams.
journalarXiv preprint arXiv:2311.14169

[Pursnani et al(2023)]
authorpersonVinay Pursnani, personYusuf
Sermet, personMusa Kurt, and personIbrahim Demir.
Performance of ChatGPT on the US fundamentals of
engineering exam: Comprehensive assessment of proficiency and potential
implications for professional environmental engineering practice.
journalComputers and Education: Artificial
Intelligence  volume5 (year2023),

[Qi et al(2023)]
authorpersonZhangyang Qi, personYe Fang,
personMengchen Zhang, personZeyi Sun,
personTong Wu, personZiwei Liu,
personDahua Lin, personJiaqi Wang, and
personHengshuang Zhao. year2023.
Gemini vs GPT-4V: A Preliminary Comparison and
Combination of Vision-Language Models Through Qualitative Cases.
journalarXiv preprint arXiv:2312.15011

authorpersonAssad Ali Rezigalla.
Item Analysis: Concept and Application.
journalMedical Education for the 21st Century
(year2022), pages105–120.

[Richards et al(2024)]
authorpersonMike Richards, personKevin
Waugh, personMark Slaymaker, personMarian Petre,
personJohn Woodthorpe, and personDaniel Gooch.
Bob or Bot: Exploring ChatGPT's Answers to
University Computer Science Assessment.
journalACM Transactions on Computing Education
volume24, number1 (year2024),

[Santos and Campelo(2023)]
authorpersonMatheus L. O. Santos and
personCláudio E. C. Campelo.
Benchmarking quantized LLaMa-based models on the
Brazilian Secondary School Exam. In booktitle16th
Brazilian Congress on Computational Intelligence (CBIC).

[Toyama et al(2023)]
authorpersonYoshitaka Toyama, personAyaka
Harigai, personMirei Abe, personMitsutoshi Nagano,
personMasahiro Kawabata, personYasuhiro Seki, and
personKei Takase. year2023.
Performance evaluation of ChatGPT, GPT-4, and Bard
on the official board examination of the Japan Radiology Society.
journalJapanese Journal of Radiology
(year2023), pages1–7.

[Tyen et al(2024)]
authorpersonGladys Tyen, personHassan
Mansoor, personVictor Cărbune, personPeter Chen,
and personTony Mak. year2024.
LLMs cannot find reasoning errors, but
can correct them!
journalarXiv preprint arXiv:2311.08516

[Wang et al(2023)]
authorpersonXuezhi Wang, personJason Wei,
personDale Schuurmans, personQuoc Le,
personEd Chi, personSharan Narang,
personAakanksha Chowdhery, and personDenny Zhou.
Self-Consistency Improves Chain of Thought
Reasoning in Language Models.
journalarXiv preprint arXiv:2203.11171

[Wei et al(2022)]
authorpersonJason Wei, personXuezhi Wang,
personDale Schuurmans, personMaarten Bosma,
personFei Xia, personEd Chi, personQuoc V
Le, personDenny Zhou, et al
Chain-of-Thought Prompting Elicits Reasoning in
Large Language Models.
journalAdvances in Neural Information Processing
Systems (NeurIPS 2022)  volume35 (year2022),

[Wu et al(2024)]
authorpersonWenshan Wu, personShaoguang
Mao, personYadong Zhang, personYan Xia,
personLi Dong, personLei Cui, and
personFuru Wei. year2024.
Visualization-of-Thought Elicits Spatial Reasoning
in Large Language Models.
journalarXiv preprint arXiv:2404.03622

[Yao et al(2023)]
authorpersonShunyu Yao, personDian Yu,
personJeffrey Zhao, personIzhak Shafran,
personThomas L. Griffiths, personYuan Cao, and
personKarthik Narasimhan. year2023.
Tree of Thoughts: Deliberate Problem Solving with
Large Language Models.
journalarXiv preprint arXiv:2305.10601

authorpersonJerrold H Zar.
Spearman Rank Correlation.
journalEncyclopedia of Biostatistics
volume7 (year2005).

[Zhai et al(2024)]
authorpersonXiaoming Zhai, personMatthew
Nyaaba, and personWenchao Ma.
Can Generative AI and ChatGPT Outperform Humans on
Cognitive-demanding Problem-Solving Tasks in Science?
journalarXiv preprint arXiv:2401.15081

[Zhang et al(2024b)]
authorpersonRenrui Zhang, personDongzhi
Jiang, personYichi Zhang, personHaokun Lin,
personZiyu Guo, personPengshuo Qiu,
personAojun Zhou, personPan Lu,
personKai-Wei Chang, personPeng Gao, et al
MathVerse: Does Your Multi-modal LLM Truly See the
Diagrams in Visual Math Problems?
journalarXiv preprint arXiv:2403.14624

[Zhang et al(2024a)]
authorpersonYizhe Zhang, personHe Bai,
personRuixiang Zhang, personJiatao Gu,
personShuangfei Zhai, personJosh Susskind, and
personNavdeep Jaitly. year2024a.
How Far Are We from Intelligent Visual Deductive
journalarXiv preprint arXiv:2403.04732

[Zhao et al(2023a)]
authorpersonHaozhe Zhao, personZefan Cai,
personShuzheng Si, personXiaojian Ma,
personKaikai An, personLiang Chen,
personZixuan Liu, personSheng Wang,
personWenjuan Han, and personBaobao Chang.
MMICL: Empowering Vision-language Model with
Multi-Modal In-Context Learning.
journalarXiv preprint arXiv:2309.07915

[Zhao et al(2023b)]
authorpersonWayne Xin Zhao, personKun Zhou,
personJunyi Li, personTianyi Tang,
personXiaolei Wang, personYupeng Hou,
personYingqian Min, personBeichen Zhang,
personJunjie Zhang, personZican Dong,
personYifan Du, personChen Yang,
personYushuo Chen, personZhipeng Chen,
personJinhao Jiang, personRuiyang Ren,
personYifan Li, personXinyu Tang,
personZikang Liu, personPeiyu Liu,
personJian-Yun Nie, and personJi-Rong Wen.
A Survey of Large Language Models.
journalarXiv preprint arXiv:2303.18223

[Zhou et al(2024)]
authorpersonKyrie Zhixuan Zhou,
personZachary Kilhoffer, personMadelyn Rose
Sanfilippo, personTed Underwood, personEce Gumusel,
personMengyi Wei, personAbhinav Choudhry, and
personJinjun Xiong. year2024.
“The teachers are confused as well”: A
Multiple-Stakeholder Ethics Discussion on Large Language Models in Computing
journalarXiv preprint arXiv:2401.12453

§ PROMPT TEMPLATES

The following are the prompt templates used to guide ChatGPT-4 Vision and ChatGPT-4 Turbo's responses during the ENADE 2021 BCS question-answering session. The templates are organized into three categories: initial prompts, reassessment prompts, and reflective prompts. Initial prompts are used to guide the model's first response to each question. Reassessment prompts are used to guide the model's self-reflective response when presented with a differing response standard or answer key. Finally, reflective prompts are used to guide ChatGPT-4 Turbo's analysis of ChatGPT-4 Vision's main challenges and difficulties in answering the questions.

[title=Initial prompts]
For open questions: Answer the question in this image. Explain your reasoning step by step. Inform if the question is unclear or has no possible answer.

[question image]

For multiple-choice questions: Answer the question in this image. Explain your reasoning step by step. At the end, choose which of the five alternatives (A-E) is correct. Inform if there is more than one correct alternative or if all five alternatives are incorrect.

[question image]

[title=Reassessment prompts]
For open questions: Your response is not entirely consistent with the response standard released for this question:

[response standard]

After analyzing the response standard, do you maintain or change your original response? If you decide to change your response, explain the possible reasons that led you to errors in the previous response. Try to identify the concrete causes of the errors instead of listing generic reasons, such as a failure to understand the question statement.

For multiple-choice questions: Your response differs from the answer key released for this question:

[answer key]

After analyzing the answer key, do you maintain or change your response? If you decide to change your response, explain the possible reasons that led you to errors in the previous answer. Try to identify the concrete causes of the errors instead of listing generic reasons, such as a failure to understand the question statement.

[title=Reflective prompts (for ChatGPT-4 Turbo)]
The following are the prompts and responses of a generative AI for the attached question, along with possible assessments by two or more human experts on the generative AI's responses and the general quality of the question. Based on the provided information, summarize the main difficulties faced by the generative AI in attempting to answer the question, including possible difficulties resulting from incorrect recognition of some graphical or textual element present in the statement of the question. Consider only concrete difficulties whose manifestation can be evidenced by the information provided in generative AI's responses. At the end, discuss if these difficulties could be mitigated or eliminated by using prompt engineering techniques or if they would be inherent to the limitations of current large language models (LLMs).

[question image]

[ChatGPT-4 Vision's prompts and responses]

[expert assessments (when available)]

Note that the main difference between the initial prompts for the two types of questions is that multiple-choice prompts explicitly request the model to select the correct answer from the five given alternatives.

In both cases, the prompts ask the model to explain its reasoning and inform if the question is unclear or has multiple or no possible correct answers.

§ CHATGPT-4'S FULL CONVERSATIONS AND EXPERTS' ASSESSMENTS

The following are ChatGPT-4's full conversations and the human experts' assessments—when available—for the three open questions and a representative sample of six valid multiple-choice questions.

§.§ Open Questions

§.§.§ [Question 03]03

This question asks the respondent to analyze a truth table for a set of formulas of propositional logic and answer four questions about the formulas' properties. ChatGPT-4 Vision gave correct answers to the first and fourth questions, a partially correct answer to the second question, and an incorrect answer to the third question. Upon being confronted with the response standard for the question, the model admitted it did a rush analysis of the truth table and confused the labels of some columns. The model also failed to cross-validate information between the formulas and the truth table, leading to the incorrect evaluation of one of the formulas. This question shows the model's understanding of complex mathematical concepts such as propositional logic and its limitations in correctly identifying and reasoning about structured visual information such as tables.

English transcription

A body of knowledge represented in propositional logic uses logical connectives of implication (→) which represents the conditional, conjunction (∧) which represents the logical AND, the disjunction (∨) which represents the logical OR and the negation (¬) which represents the logical operator NOT. Let P be the following set of formulas of propositional logic:

1. a → ¬b

2. b ∧ a

3. ¬b ∨ b

let Q be the following set of formulas of propositional logic:

4. a ∨ b

5. b → ¬a

and let R be the formula

6. ¬b → a

See the truth table for these formulas.

[Table displaying truth values for propositions a and b, and corresponding truth values for formulas 1 through 6. The table has columns labeled "a", "b", "a → ¬b", "b ∧ a", "¬b ∨ b", "a ∨ b", "b → ¬a", "¬b → a", with rows that provide the truth values (True or False) for each formula based on the combination of truth values for a and b.]

It is known that each row in the truth table corresponds to an attribution of truth values for the propositional symbols (a and b) and each column corresponds to the evaluation of the formula for that attribution. Some definitions:

(i) A formula is a tautology if and only if, for every attribution of truth-values, its evaluation is true.

(ii) An attribution of truth-values satisfies a set of formulas if and only if, for every formula in the set, the evaluation is true.

(iii) A set of formulas is satisfiable if and only if there exists an attribution of truth-values that satisfies the set. Otherwise, it is unsatisfiable.

(iv) A formula is a logical consequence of a set of formulas if and only if, for every truth-value attribution that satisfies the set, the attribution also satisfies the formula.

Based on the information presented, answer the following items:

a) Is there any tautology in formulas 1 to 6? Justify your answer.
(value: 2.5 points)

b) Is there any satisfiable set (P or Q)? Justify your answer.
(value: 2.5 points)

c) Is there any unsatisfiable set (P or Q)? Justify your answer.
(value: 2.5 points)

d) Is formula 6 a logical consequence of Q? Justify your answer.
(value: 2.5 points)

Response standard

a) The respondent must say that formula 3 contains a tautology, as it presents a true evaluation for every assignment (column of Vs).

b) The respondent must indicate that the set Q is satisfiable, as all formulas of Q (4 and 5) are true for assignments 3 and 4 (one would be enough).

c) The respondent must indicate that the set P is not satisfiable, as there is no assignment for which the formulas of P (1, 2, and 3) are all true.

d) The respondent must indicate that formula 6 is a logical consequence of Q, as all assignments that satisfy Q (3 and 4) also satisfy 6.

[title=Initial prompt]
Answer the question in this image. Explain your reasoning step by step. Inform if the question is unclear or has no possible answer.

[question image]

[title=ChatGPT-4 Vision's initial response]
Understood, let's address the questions.

a) Is there any tautology in formulas 1 to 6?

First, let's define what a tautology is: A formula in propositional logic is a tautology if, and only if, it is true for all possible truth-value assignments to its variables.

Looking at the provided truth table:
- Formula 1 (a → b): Not a tautology, as it is false when a is true and b is false.

- Formula 2 (b ∧ a): Not a tautology, as it is false for three out of the four possible combinations.

- Formula 3 (¬b ∨ b): Is a tautology, as it is true in all assignments.

- Formula 4 (a ∨ b): Not a tautology, as it is false when both a and b are false.

- Formula 5 (b → a): Not a tautology, as it is false when b is true and a is false.

- Formula 6 (¬b → a): Not a tautology, as it is false when ¬b is true and a is false.

Answer: The only tautology is formula 3.

b) Is there any satisfiable set (P or Q)?

A set of formulas is satisfiable if there is some truth-value assignment that makes all the formulas in the set true simultaneously.

- Set P: We see that the second line of the table (where a is false and b is true) satisfies all three formulas in P.

- Set Q: The third line of the table (where a is true and b is false) satisfies both formulas in Q.

Answer: Both sets, P and Q, are satisfiable.

c) Is there any unsatisfiable set (P or Q)?

Using the information from the previous item, we know that both sets are satisfiable, so none of them are unsatisfiable.

Answer: No set is unsatisfiable.

d) Is formula 6 a logical consequence of Q?

A formula is a logical consequence of a set if, whenever all the formulas in the set are true, the formula is also true.

To check this, we observe any line where all formulas of Q are true and verify if formula 6 is true on the same line.

Formula 6 is true on lines 2, 3, and 4. The formulas in Q are both true on lines 3 and 4. In both these lines, formula 6 is true.

Answer: Yes, formula 6 is a logical consequence of Q.

[title=Reassessment prompt]
Your answers are not completely consistent with the response standard released for this question: [response standard]

After analyzing the response standard, do you maintain or change your original answers? If you decide to change your answers, explain the possible reasons that led you to errors in the previous answers. Try to identify the concrete causes of errors, rather than listing generic reasons like misunderstanding the statement of the question.

[title=ChatGPT-4 Vision's revised response]
I change my original answers to align with the released response standard:

a) The only tautology is formula 3.

b) The set Q is satisfiable, as all formulas of Q (4 and 5) are true for assignments 3 and 4.

c) The set P is not satisfiable, as there is no assignment for which the formulas of P (1, 2, and 3) are all true simultaneously.

d) Formula 6 is a logical consequence of Q, as all assignments that satisfy Q (3 and 4) also satisfy 6.

Upon examining the truth table and previous answers, we can identify some more specific errors:

1. Satisfiable Set P: By stating that set P was satisfiable, it seems I confused the columns. Looking at the truth table, no single line makes all three formulas in P true simultaneously. The error here was clearly a failure in analyzing the truth table, perhaps giving too much attention to one column or line and losing sight of the overall picture.

2. Evaluation of Tautologies: The tautology in formula 3 was correctly identified, but errors in other items may have arisen from a rushed analysis or not properly cross-referencing information between the formulas and the truth table.

3. About Logical Consequence: The original answer to item d was correct. However, a mistake in understanding set P may have led to misunderstandings in the relationships between the formulas and their sets.

4. Failure in the Verification Process: In stating something about satisfiability or logical consequence, an additional check of the truth table would have helped avoid errors. It seems the verification process was not rigorous enough.

In summary, the errors seem to center on a combination of rushed analysis of the truth table and failure to follow a rigorous verification process. The complexity of the problem is not an excuse, as propositional logic and truth tables are something the model should be able to handle accurately. The most concrete failure was in meticulous analysis and cross-validation of information from the truth table with the given formulas and sets.

[title=Expert's scoring of ChatGPT-4 Vision's initial response]
a) Correct answer: the model correctly identified that formule 3 contains a tautology. (2.5/2.5 scores)

b) Partially correct answer: the model correctly identified that set Q is satisfiable, but incorrectly identified that set P is also satisfiable. (1.25/2.5 scores)

c) Incorrect answer: the model incorrectly identified that set P is satisfiable. (0/2.5 scores)

d) Correct answer: the model correctly identified that formula 6 is a logical consequence of set Q. (2.5/2.5 scores)

Total scores: 2.5 + 1.25 + 0 + 2.5 = 6.25/10.0 (62.5/100)

[title=Challenge categories / error types]
Visual Acuity / Misidentification of Visual Elements

Upon self-reflection, the model admitted it did a rushed analysis of the truth table and confused the labels of some columns.

Logical Reasoning / Incorrect Multi-Step Reasoning

The model failed to cross-validate information between the formulas and the truth table, perhaps due to the error described above, leading to the incorrect evaluation of the satisfiability of set P.

§.§.§ [Question 04]04

This question asks the respondent to describe the truth table and draw the circuit diagram of a 1-bit full adder. ChatGPT-4 Vision correctly built the truth table but failed to draw the circuit diagram. This question highlights the model's capacity to understand complex computer architecture concepts such as logic circuits but also the model's current inability to generate domain-specific visual outputs such as a circuit diagram.

English transcription

The sum of two binary numbers is done bit by bit, starting from the right (less significant) to the left (more significant), and the carryover is forwarded, that is, carry out (represented in the figure as Cout), to the next bit as carry in (represented in the figure as Cin). A simple way to implement an N-bit adder is to implement N elementary 1-bit adders. Each 1-bit adder has inputs A, B, and carry in (Cin) and the outputs Sum (S) and carry out (Cout).

DELGADO, J.; RIBEIRO, C. Computer Architecture. Rio de Janeiro: LTC, 2009 (adapted).

In this context, consider the following figure.

[The image shows a block diagram of a "1-bit Full Adder" with three inputs labeled A, B, and Cin and two outputs labeled S and Cout.]

Based on the 1-bit full adder depicted in the figure, describe its truth table and the diagram of its logic circuit. (value: 10.0 points)

Response standard

The respondent should describe the truth table and draw the diagram, as below.

The truth table can be created in different ways, swapping "1"s and "0"s for "V" and "F", or for "T" and "F". Moreover, the rows can appear in any arbitrary order.

The logic circuit can be solved in various ways, with variations in notation, and more than one circuit performing the same operation. The proposed circuit, in fact, combines the two outputs into a single circuit.

Possible notations are presented in the following image, from the IEC 60617-12 standard, and also from ANSI IEEE.

Other possible circuits, without being an exhaustive list, are:

Examples of possible answers for S:

Examples of possible answers for Cout:

[title=Initial prompt]
Answer the question in this image. Explain your reasoning step by step. Inform if the question is unclear or has no possible answer.

[question image]

[title=ChatGPT-4 Vision's initial response]
Understood. Let's address the problem of the 1-bit full adder.

A 1-bit full adder has three inputs: A, B, and Cin (carry-in) and two outputs: S (sum) and 	Cout (carry-out). The aim of this adder is to add the three input bits and produce the corresponding sum bit and carry bit.

Let's start with the truth table:

Now, let's deduce the logical expressions for S and Cout:

For the S output (sum):

S = A XOR B XOR Cin

For the Cout output (carry-out):

There are three scenarios where Cout is 1:

1. A = 1, B = 1

2. A = 1, Cin = 1

3. B = 1, Cin = 1

Cout = (A AND B) OR (A AND Cin) OR (B AND Cin)

Finally, the logical circuit diagram consists of:

1. Three XOR gates to calculate the sum S.

2. Two AND gates and one OR gate to calculate the carry-out Cout.

However, visualizing a logical circuit diagram is not possible here, but you can use the
description above to draw it on paper or use a circuit drawing tool.

In summary, a 1-bit full adder adds three input bits and produces a 2-bit result (sum and carry).
The logic to calculate these bits is determined by the expressions for S and Cout we just deduced.

[title=Expert's scoring of ChatGPT-4 Vision's initial response]
The truth table produced by ChatGPT-4 Vision is correct according to the response standard. However, since the model cannot currently generate domain-specific diagrams, and was not explicitly prompted to "draw" a circuit using some alternative textual notation, it gets only half of the question's total scores.

Total scores: 5.0/10.0 (50/100)

[title=Challenge categories / error types]
Visual Acuity / Lack of	Domain-Specific Visual Output

The model failed to draw the diagram for the logic circuit, as requested by the question.

§.§.§ [Question 05]05

This question asks the respondent to answer questions regarding the behavior and time complexity of a binary heap implementation in C whose code is provided as part of the question. ChatGPT-4 Vision gave an incorrect answer to the first question, where it failed to calculate the final state of a given array after the execution of the binary heap implementation with certain parameters, but correctly informed the binary heap's time complexity as the answer to the second question. This question showcases the model's capacity to understand complex algorithmic concepts such as a sorting algorithm's time complexity but exposes the model's limitations to correctly simulate the execution of a piece of code in a step-by-step manner.

English transcription

A binary heap is an array that can be visualized as a binary tree, where each tree node corresponds to an array element, as can be observed in the figure below.

[An image of a binary tree containing 12 nodes with corresponding array indexes and values above it.]

There exist two types of heaps: maximum heaps and minimum heaps. The maximum heap is a data structure that allows efficient query or removal of the maximum element from the collection. The maximum heap property specifies that a node (calculated by the functions left and right in the companying code) should store a value smaller or equal to its parent.

CORMEN, T. H.; LEISERSON, C. E.; RIVEST, R. L.; STEIN, C. Introduction to Algorithms. 3. ed. MIT Press and McGraw-Hill, p. 131-161, 2009 (adapted).

Considering the following implementation, heapify is a helper function to organize the array (ensuring the heap property for a certain position of the array) and buildHeap is a function that uses heapify to reorganize all the positions of the array (ensuring the heap property of maximum heap for all its elements).

int left(int i)  return (2 * i + 1);
int right(int i)  return (2 * i + 2);

/* a - array, n - number of elements, i - position of the
element that must be placed in proper heap order */
void heapify(int *a, int n, int i)

int e, d, max, aux;

e = left(i);
d = right(i);
if (e < n        a[e] > a[i])
max = e;
max = i;
if (d < n        a[d] > a[max])
max = d;
if (max != i)

aux = a[i];
a[i] = a[max];
a[max] = aux;
heapify(a, n, max);

/* a - array, n - number of elements */
void buildHeap(int *a, int n)

int i;
for (i = (n-1)/2; i >= 0; i–)
heapify(a, n, i);

Based on the information presented, do what is requested in the items below.

a) How will the array int a[] = {2, 5, 8, 13, 21, 1, 3, 34} look like after the execution of the function buildHeap(a, 8). (value: 5.0 points)

b) Present the time complexity in the worst case for the function heapify, using the notation O or Θ. (value: 5.0 points)

Response standard

a) The respondent should show that after executing the function buildHeap the arrangement will be as follows: {34, 21, 8, 13, 2, 1, 3, 5}.

b) The respondent should present that in the worst case for the heapify function the time complexity will be as follows: $O(log n)$, being $n$ the number of elements in the heap.

[title=Initial prompt]
Answer the question in this image. Explain your reasoning step by step. Inform if the question is unclear or has no possible answer.

[question image]

[title=ChatGPT-4 Vision's initial response]
Let's address the presented questions:

a) What will be the arrangement of int a[] = {2, 5, 8, 13, 21, 1, 3, 34} after executing the function buildHeap(a, 8)?

The function buildHeap begins by calling the heapify function at the penultimate level of the tree (or heap). In terms of the array, this translates to starting from the middle of the array and going backwards.

The heapify method is responsible for maintaining the max heap property. If an element is smaller than its children, it will be swapped with the larger of its children.

The initial arrangement is: 2, 5, 8, 13, 21, 1, 3, 34

Now, let's go through the array using buildHeap:

1. We start at i = 3 (remembering that indexing starts at 0), which is the value 13. Its children are 21 and 1. 21 is larger, and therefore, 13 and 21 swap places. Result: 2, 5, 8, 21, 13, 1, 3, 34

2. i = 2, which is the value 8. Its children are 1 and 3. 8 is already larger than both, so nothing changes. Result: 2, 5, 8, 21, 13, 1, 3, 34

3. i = 1, which is the value 5. Its children are 21 and 13. 21 is larger, so 5 and 21 swap places. Moreover, after this swap, 5 will be compared again with its new children (i.e., the heapify method is called again). Result after first swap: 2, 21, 8, 5, 13, 1, 3, 34 After the recursive call, 5 and 13 swap places. Result: 2, 21, 8, 13, 5, 1, 3, 34

4. i = 0, which is the value 2. Its children are 21 and 8. 21 is larger, so 2 and 21 swap places. Result: 21, 2, 8, 13, 5, 1, 3, 34
Thus, after the execution of the function buildHeap, the arrangement will be: 21, 2, 8, 13, 5, 1, 3, 34

b) Present the worst-case time complexity for the heapify function, using the $O$ or $Θ$ notation.

The complexity of the heapify function is determined by the height of the tree, because in the worst case, it may be necessary to swap an element with its child until it reaches the bottom of the tree. The height of a complete binary tree is $log(n)$. Therefore, the worst-case time complexity for the heapify function is $O(log n)$.

Answers: a) 21, 2, 8, 13, 5, 1, 3, 34 b) $O(log n)$

[title=Reassessment prompt]
Your answers are not completely consistent with the response standard released for this question: [response standard]

After analyzing the response standard, do you maintain or change your original answers? If you decide to change your answers, explain the possible reasons that led you to errors in the previous answers. Try to identify the concrete causes of errors, rather than listing generic reasons like misunderstanding the statement of the question.

[title=ChatGPT-4 Vision's revised response]
After reviewing the response standard, I would change my original answers as follows:

a) The arrangement after executing the function buildHeap is {34, 21, 8, 13, 2, 1, 3, 5}.

b) The worst-case time complexity of the heapify function is $O(log n)$.

Reasons for Errors:

a) Error in Executing buildHeap: When executing the function buildHeap, I did not correctly apply the max heap property at some stages. For example, the value 34, which is the highest in the arrangement, should have risen to the root after the complete execution of buildHeap, but this did not happen in my original answer. This indicates a failure in my simulation of the algorithm's execution.

b) For the complexity, my answer was correct, but it could have been explained more clearly and directly.

Concrete Causes of Errors:

1. Algorithm Simulation: In simulating the algorithm, I did not apply all possible swaps that heapify could make. For example, the value 34 should have been promoted higher in the heap, but I missed this step. Algorithm simulation requires careful attention at each step, and it seems there was haste or lack of attention to details in my simulation.

2. Review: At the end of the simulation, I did not review the resulting structure to ensure it met the max heap property, which would have indicated that something was wrong with the original response.

In summary, the errors mainly stem from failing to simulate the algorithm correctly and not reviewing the resulting structure to ensure it adhered to the desired property.

[title=Expert's scoring of ChatGPT-4 Vision's initial response]
a) Incorrect answer: the model produced an incorrect arrangement according to the response standard. (0.0/5.0 scores)

b) Correct answer: the model correctly informed the algorithm complexity in line with the response standard. (5.0/5.0 scores)

Total scores: 0.0 + 5.0 = 5.0/10.0 (50/100)

[title=Challenge categories / error types]
Logical Reasoning / Incorrect Algorithmic Reasoning

The model failed to correctly simulate the execution of the function buildHeap, leading to an incorrect array arrangement.

§.§ Multiple-Choice Questions

§.§.§ [Question 11]11

This question asks the respondent to identify the correct statements regarding the improvement of a deep neural network architecture for a specific image classification task and then select the alternative that refers to those correct statements. ChatGPT-4 Vision accurately identified the correct statements and their corresponding alternative while providing a detailed explanation for its choice. This question demonstrates the model's capacity to understand complex multimodal content in artificial intelligence and reason about it in a multi-step process.

English transcription

A team of computer scientists from a certain animation company has been tasked with developing a system capable of scanning the web to detect sites that might be using the company's animation character images without proper consent. Therefore, the system should receive images as input, and classify them between company images and images not produced by the company.

The figure below outlines a deep neural network architecture and the training process the scientists intend to use.

[The image shows a flow diagram for a deep neural network architecture, consisting of an input layer (Entrada (X)), a series of intermediary layers (Pesos 1, ..., Pesos n, with Camada de Entrada, ..., and Camada Intermediária n between the weight layers), leading to an output layer (Camada de Saída). This is followed by Predicted Outputs (Saídas Previstas (Y')) being compared with Real Outputs (Saídas Reais (Y)) using a Loss Function (Função de Perda), which feeds into an Optimizer (Otimizador) that adjusts the weights (Ajuste dos pesos).]

CHOULLET, F. Deep Learning with Python. New York: Manning Publications, 2017 (adapted).

After one attempt, two difficulties were noticed: 1) the network training time was too long, and 2) the network's accuracy was not at an acceptable level for the company.

Given these findings, consider the following statements.

I. Increasing the number of layers is an alternative that can lead to an improvement in accuracy, as well as decreasing the network training time.

II. Making use of convolutional networks is an alternative that can lead to an improvement in accuracy, however, it may require the use of machines with greater processing power.

III. Increasing the number of processing units (neurons) in the layers may lead to an improvement in accuracy, as well as decreasing the network training time.

IV. Increasing the number of training samples is an alternative that can lead to an improvement in accuracy, as well as decreasing the network training time.

V. Making use of recurrent networks is an alternative that can lead to an improvement in accuracy, however, it may require the use of machines with greater processing power.

It is correct only what is stated in

A) I and IV.

B) I and V.

C) II and III.

D) II and IV.

E) III and V.

Answer key: D

[title=Initial prompt]
Answer the question in this image. Explain your reasoning step by step. At the end, choose which of the five alternatives (A-E) is correct. Inform if there is more than one correct alternative or if all five alternatives are incorrect.

[question image]

[title=ChatGPT-4 Vision's initial response]
Let’s analyze the statements:

I. Increasing the number of layers is an alternative that can lead to an improvement in accuracy, as well as decreasing the network training time.

- This is partially correct. Increasing the number of layers can improve accuracy, as it makes the network deeper and able to capture more complex features. However, it generally does not decrease training time, in fact, it may increase it since there are more parameters to be adjusted.

II. Making use of convolutional networks is an alternative that can lead to an improvement in accuracy, however, it may require the use of machines with greater processing power.

- This is true. Convolutional Neural Networks (CNNs) are widely used for image-related tasks and generally perform better on such tasks compared to fully connected networks. However, due to their nature and the number of operations involved, they may require more computing power, especially for large data volumes.

III. Increasing the number of processing units (neurons) in the layers may lead to an improvement in accuracy, as well as decreasing the network training time.

- Increasing the number of neurons can improve accuracy since it allows the network to capture more features. However, similar to the first statement, increasing the number of neurons will likely not decrease the training time; in fact, it might increase it because there are more parameters to be trained.

IV. Increasing the number of training samples is an alternative that can lead to an improvement in accuracy, as well as decreasing the network training time.

- This is true. More data generally lead to more robust and accurate models. However, with more data, the training time will increase because there are more examples for the network to process.

V. Making use of recurrent networks is an alternative that can lead to an improvement in accuracy, however, it may require the use of machines with greater processing power.

- Recurrent Neural Networks (RNNs) are commonly used for sequences or temporal data. They are not typically the first choice for image classification tasks. Therefore, this statement is potentially misleading in the context of this question.

Given the above analyses, statements II and IV are the most accurate.

Answer: D) II and IV.

§.§.§ [Question 13]13

This question asks the respondent to select the correct statement regarding iterative and evolutionary software development considering a given reference text. ChatGPT-4 Vision chose a different alternative from the answer key based on an inconsistent reasoning process where it sometimes followed a strict interpretation of the question and sometimes applied implicit domain knowledge not directly mentioned in the reference text. This question showcases the model's capacity to understand complex software development concepts but also its struggles with questions requiring implicit knowledge or inferences beyond the provided text. The fact that the three experts did not reach a consensus on the correct answer or the validity of the question also highlights the potential ambiguity of the question and the critical need for better quality control in the exam elaboration. Note: this question was one of the six questions excluded from the final score due to a Weak discrimination level (see Section <ref>).

English transcription

Iterative and evolutionary systems development is an approach that establishes cycles of development with a fixed duration, called iterations. The product of each iteration is a partial, executable, testable, and integrable system. Each iteration includes its own requirement analysis, project, implementation, and test activities. The iterative lifecycle is based on refinements and increments of a system through multiple iterations, with cyclical feedback and adaptation as the main drivers to converge to a suitable system.

CRAIG, L. Using UML and Patterns: An Introduction to Object-Oriented Analysis and Design. 3rd ed. Porto Alegre: Bookman, 2007 (adapted).

Considering the presented text, select the correct option about iterative and evolutionary development.

A) Changing system requirements is something that causes delays in development, so it is advisable to avoid doing it.

B) The development cycle has a fixed duration but may be changed in the case of critical systems.

C) Usability testing should be carried out in the last cycle because it is the moment when the user can test all functionalities.

D) The subsystem generated by the implementation of requirements at the end of an iteration may be used by the customer as a prototype.

E) The usability testing document must contemplate accessibility criteria to serve all system users.

Answer key: E

[title=Initial prompt]
Answer the question in this image. Explain your reasoning step by step. At the end, choose which of the five alternatives (A-E) is correct. Inform if there is more than one correct alternative or if all five alternatives are incorrect.

[question image]

[title=ChatGPT-4 Vision's initial response]
Let's analyze the text and the options provided step by step:

Text: Iterative and evolutionary systems development is an approach that establishes cycles of development with a fixed duration, called iterations. The product of each iteration is a partial, executable, testable, and integrable system. Each iteration includes its own requirement analysis, project, implementation, and test activities. The iterative lifecycle is based on refinements and increments of a system through multiple iterations, with cyclical feedback and adaptation as the main drivers to converge to a suitable system.

Based on the text, let's analyze the alternatives:

A) Changing system requirements is something that causes delays in development, so it is advisable to avoid doing it.

- The text does not mention that changes in requirements cause delays, and iterative and evolutionary systems, in general, are designed to accommodate changes in requirements. Therefore, this option seems incorrect.

B) The development cycle has a fixed duration but may be changed in the case of critical systems.

- The text mentions that the cycle has a "fixed duration," but it does not make any reference to changes in the case of "critical systems." Therefore, this alternative is partially true, but not entirely based on text.

C) Usability testing should be carried out in the last cycle because it is the moment when the user can test all functionalities.

- The text does not specifically mention "usability testing," nor does it state that it should be performed in the last cycle. Therefore, this option seems incorrect.

D) The subsystem generated by the implementation of requirements at the end of an iteration may be used by the customer as a prototype.

- The text mentions that the "product of each iteration is a partial system, executable, testable, and integrable," which can be interpreted as a "prototype" that the client can view and test. This alternative seems to be aligned with the text.

E) The usability testing document must contemplate accessibility criteria to serve all system users.

- Again, the text does not specifically mention "usability testing" or "accessibility" criteria. Therefore, this option seems incorrect.

Alternative D "The subsystem generated by the implementation of requirements at the end of an iteration may be used by the customer as a prototype" is the most aligned with the information presented in the text. Hence, the correct alternative is D.

[title=Reassessment prompt]
Your response differs from the answer key released for this question: [answer key]

After analyzing the answer key, do you maintain or change your response? If you decide to change your response, explain the possible reasons that led you to errors in the previous answer. Try to identify the concrete causes of the errors instead of listing generic reasons, such as a failure to understand the question statement.

[title=ChatGPT-4 Vision's revised response]
I maintain response D as correct based on the analysis of the text provided. Option D aligns with the concept of iterative and evolutionary development, where each iteration results in a product or subsystem that is "executable, testable, and integrable". This subsystem can indeed be used by the client as a prototype or an incremental version of the final system.

Regarding alternative E, which was chosen in the answer key, I can highlight the following points:

- The text does not mention "accessibility" criteria.

- "Usability testing" is not mentioned in the text.

The previous error may have been caused by the attempt to interpret the statement too broadly, trying to infer information that was not explicitly mentioned in the text. Option E may have been chosen based on the idea that usability tests usually consider accessibility, but this is not indicated in the presented text. In objective assessments, it is crucial to base oneself strictly on the information given in the statement and the text, avoiding assumptions or inferences that go beyond what is provided.

In conclusion, the previous error was caused by the attempt to infer information not explicitly mentioned in the text, leading to the choice of an option that was not directly supported by the content presented.

[title=Clarification prompt]
To be clear: if you maintain the choice of alternative D as correct, what do you mean by "previous error"? Who would have committed such an error? The one who elaborated the answer key?

[title=ChatGPT-4 Vision's clarifying response]
I apologize for the confusion. By mentioning "previous error", I was referring to the discrepancy between the choice in the answer key and the analysis made here. If response D is the one I consider correct based on the analysis of the text, then the "error" would be in the creation of the answer key that indicated alternative E as correct. That is, the possible error would be from the part of those who elaborated the answer key or, alternatively, there could be additional information or contexts that were not considered in the analysis made here. Again, I apologize for the confusion and hope to have clarified the point.

[title=Experts' assessments]
Expert #1

ChatGPT is correct.

If the question statement asks to consider the reference text, the correct choice is indeed D. I agree with ChatGPT's analysis of the alternatives.

Expert #2

A) Changing system requirements is something that causes delays in development, so it is advisable to avoid doing it.

- The text does not mention that changes in requirements cause delays, and iterative and evolutionary systems, in general, are designed to accommodate changes in requirements. Therefore, this option seems incorrect.

ChatGPT's answer is dubious, as it is common knowledge in software engineering (SE) that changes in requirements cause delays. There is an incompatibility with the text. To avoid doubt, the question should say: it is something that CAN cause delays ... Therefore, the alternative is not entirely incorrect.

B) The development cycle has a fixed duration but may be changed in the case of critical systems.

- The text mentions that the cycle has a "fixed duration," but it does not make any reference to changes in the case of "critical systems." Therefore, this alternative is partially true, but not entirely based on text.

The text does not mention it, but we know that the cycles must be followed, although their duration varies according to the system or size, so the statement is not entirely incorrect.

C) Usability testing should be carried out in the last cycle because it is the moment when the user can test all functionalities.

- The text does not specifically mention "usability testing," nor does it state that it should be performed in the last cycle. Therefore, this option seems incorrect.

Neither the text mentions nor is it common sense in SE, I agree that the alternative is incorrect.

D) The subsystem generated by the implementation of requirements at the end of an iteration may be used by the customer as a prototype.

- The text mentions that the "product of each iteration is a partial system, executable, testable, and integrable," which can be interpreted as a "prototype" that the client can view and test. This alternative seems to be aligned with the text.

It is common sense in software engineering that we should not use the prototype as the final system. The text mentions a partial system, so the alternative is incorrect.

E) The usability testing document must contemplate accessibility criteria to serve all system users.

- Again, the text does not specifically mention "usability testing" or "accessibility" criteria. Therefore, this option seems incorrect.

The text does not mention it, but it is common sense that usability tests should include non-functional requirements like accessibility. So the alternative is correct.

The question is ambiguous and should be invalidated, as alternatives A, B, C, and E are not contained in the text. It seems to me that the text was used as an inducer of the questions, but the intention is to put general alternatives about the iterative and incremental life cycle. Thinking this way, I arrived at two possible answers A and E. However, as there was no CAN in A, then E would be more appropriate, so the answer key would be correct. However, the candidate has to make a reflection outside the text, so I consider the question ambiguous. The text hinders the interpretation of the alternatives.

Expert #3

Analyzing all the material, I conclude that the official answer key is correct.

Justification: options A-D contain errors based on my knowledge of Software Engineering, knowledge that is not limited to the statement of the question. It seems to me that ChatGPT's choice and justification are based solely on the statement because it repeatedly says that the text does not mention certain concepts. But at the same time, ChatGPT chooses option D, which mentions a subsystem that, in turn, was not defined in the text of the statement. It seems to me that it inferred what a subsystem would be, and in my opinion, its inference is incorrect (i.e., a subsystem is not a partial system).

[title=Conclusion from the experts' assessments]
The response from ChatGPT is inconsistent with the answer key and the three experts did not reach a consensus on what the correct answer would be or whether the question should be invalidated.

[title=Reflectice prompt to ChatGPT-4 Turbo]
The following are the prompts and responses of a generative AI for the attached question, along with possible assessments by two or more human experts on the generative AI's responses and the general quality of the question. Based on the provided information, summarize the main difficulties faced by the generative AI in attempting to answer the question, including possible difficulties resulting from incorrect recognition of some graphical or textual element present in the statement of the question. Consider only concrete difficulties whose manifestation can be evidenced by the information provided in generative AI's responses. At the end, discuss if these difficulties could be mitigated or eliminated by using prompt engineering techniques or if they would be inherent to the limitations of current large language models (LLMs).

[question image]

[ChatGPT-4 Vision's prompts and responses]

[expert assessments]

[title=ChatGPT-4 Turbo's response,colbacktitle=gray!35!white]
The generative AI faced difficulty in attempting to answer the question on iterative and evolutionary systems development, resulting in a discrepancy between its answer and the official key. The main difficulties identified include:

1. Strict Text Interpretation: The AI chose alternative D based on a literal interpretation of the text, which mentions a "partial system" as the outcome of each iteration, interpreting it as a prototype. However, the official answer key indicated alternative E as correct, which the AI did not consider based on the provided text.

2. Focus on Explicit Details: The AI maintained its answer, even after the discrepancy with the answer key was pointed out, arguing that its choice was closely in line with the information given in the text and that criteria for "accessibility" could not be inferred to justify alternative E.

3. Expert Feedback: The human experts provided conflicting feedback. One agreed with the AI, deeming alternative D as correct. Others considered the question ambiguous or suggested that the correct alternative should be E, arguing the question required knowledge beyond the text.

These difficulties suggest that while the AI can analyze text and choose answers based on explicit information, it may struggle with interpretations requiring implicit knowledge or inferences beyond the provided text. Prompt engineering could be used to guide the AI to consider knowledge beyond the text, but this remains a limitation of current LLMs, which rely heavily on the input text and may not fully understand context or the expectations for implicit knowledge in specific domains, such as software engineering. It indicates a continued need for the advancement of AI models to better integrate textual comprehension with contextualized knowledge.

[title=Clarification prompt to ChatGPT-4 Turbo]
And as for the fact that the experts did not come to a consensus on the question, any additional comments?

[title=ChatGPT-4 Turbo's clarifying response,colbacktitle=gray!35!white]
The lack of consensus among human experts on the correct response to the question underscores the complexity and inherent ambiguity in some exam questions. This may arise for several reasons:

1. Ambiguity in the Statement: If the statement of the question is ambiguous or insufficiently detailed, it may lead to multiple valid interpretations.

2. Interpretation Differences: Experts can have different perspectives based on their experiences, contextual knowledge, and understanding of the subject. What may appear clear to one might be debatable to another.

3. Implicit Knowledge Expectations: Some questions may presuppose knowledge that is not explicitly declared in the text of the question. Experts might disagree on how much of this implicit knowledge should be applied when interpreting the question.

4. Variable Evaluation Criteria: Experts might value different criteria when assessing the correct response, such as practical relevance versus theoretical accuracy.

This challenge is not only for AI but also for question design and assessments in general. In the context of AI, such ambiguity and the discrepancy of human opinions expose the limitations of AI systems in dealing with uncertainty and making judgments when there is no clear answer or defined consensus.