Last updated: 1 Jul 1999 (Minor corrections). SCHOOL OF COMPUTER SCIENCE CRITERIA STATEMENTS FOR DEGREE CLASSIFICATIONS Draft notes by Aaron Sloman Some draft statements based partly on internal discussions and various documents from here and elsewhere including help from William Edmondson. Originally written in 1998. ======================================================================= C O N T E N T S 1.PREAMBLE 1.1. The high level objectives of our degree courses. 1.1.1 General university-level skills and knowledge 1.1.2 Specific degree-related skills and knowledge 1.2. Student variation 1.3. The need for flexible and flexibly interpreted assessment 1.4. Spurious mechanisms for achieving objectivity 1.5. The main function of assessment 2. REQUIREMENTS FOR VARIOUS GRADES (Draft) CRITERIA FOR VARIOUS CLASS LEVELS FAIL (Mark range: 34% and below (F)) PASS (Mark range: 35-39% (E)) CLASS III (Mark range: 40-49% (D)) CLASS II ii (Mark range: 50-59% (C)) CLASS II i (Mark range: 60-69% (B)) CLASS I (Mark range: 70%+ (A)) ADDITIONAL NOTES: KEY SKILLS KEY SKILL: PERSONAL COMMUNICATION KEY SKILL: LITERACY KEY SKILL: NUMERACY KEY SKILL: MATHEMATICAL MATURITY KEY SKILL: IT KEY SKILL: TEAMWORK KEY SKILL: LEARNING TO LEARN ======================================================================= 1.PREAMBLE 1.1. The high level objectives of a CS degree courses. There are different sorts of objectives: some common to all other degree courses and some specific to computer science, though they may overlap with closely related disciplines. 1.1.1 General university-level skills and knowledge A university is not a training college. Our objective in teaching is to expose students to a range of concepts, techniques, and theories which will stretch their minds and prepare them for a very wide range of jobs, in a world where knowledge and skills required keep changing. Although we expect students to have highly marketable skills by the time they graduate, we are not primarily concerned to meet specific skills shortages identified in commerce or industry at the time when we are planning our degrees, for generally the students taking such degrees will leave the university several years later, and the requirements will have changed by then. Rather, we aim to equip them both with a collection of generic skills, such as the ability to find things out, to develop their own skills, to communicate well, to identify needs and problems in complex situations, to report and summarise important ideas and theories clearly and accurately, to criticise and analyse theories, practices and procedures (including their own), to produce clear and sound arguments, to use evidence well, to design, implement, test, debug, redesign, document and explain complex systems, and to work well in teams. 1.1.2 Specific degree-related skills and knowledge We also expect graduates to have learnt about a significant subset of the following topics, depending on the options they choose: o what computers are at present, and how they may change in future o what software is, how it works, and the ways in which it can go wrong o how abstract information processing virtual machines are related to the underlying physical machines in which they are implemented o how many different models of computation there are (including conventional computation, various kinds of abstract machines studied by theorists, neural computation, evolutionary computation, and new computational paradigms emerging from developments in molecular biology and quantum physics) o which sorts of information computers can handle, and what sorts of operations they can perform on different sorts of information, including images, corporate databases, statistical data, engineering design databases, equations, user actions, spoken or written natural language utterances or corpora, and information about the operation of computers, etc. o how complex networks of computers are assembled and made to work together, o how complex collections of functionality can be assembled on one computer, o how computer architectures and network architectures can vary and what the tradeoffs are, o what varieties of languages have been developed for programming computers, o how programming languages can differ, and what the implications of those differences are in a variety of dimensions, including the ease with which people can learn the language, ease of software development, speed of compilation, ease of design of compilers, mathematical analysability, speed of execution, size of run-time systems, reliability, flexibility, maintainability, portability, o how formalisms for mathematical modelling and analysis of different types of computation can differ and what their uses are (e.g. logics, transition nets, lambda calculus, systems of equations, etc.) o notations for specifying requirements for computing systems (e.g. what effect a computer-based control system should have on a chemical plant, or an airliner), notations for specifying designs to meet such requirements, methods for checking designs against requirements, methods fort checking that implementations conform to designs, o how the production of software can be facilitated or hindered by the use of different design, development, or management techniques, or software engineering tools, o how computing systems can go wrong and how to prevent, detect or remedy failures or inadequacies. o where the main sources of information about computers, computing, software, etc. are (e.g. internet repositories and search engines, major journals and handbooks, etc.) o how the subject has developed and what some of the major landmarks in its evolution were (including hardware, languages, techniques, mathematical models, application areas), o what the major current trends are and what views have been expressed about likely future developments (this could include controversies, of course). o the social, political and economic relevance of computing, and the ethical issues arising. o some students may acquire deep knowledge of specific application areas o some students may acquire deep knowledge of neighbouring disciplines NOTE: It is not enough to be familiar with specific contemporary examples of machines, languages, mathematical models, development environments, application areas: students must learn underlying general principles which will survive through specific technical advances in subsequent years. In addition we expect students to have developed logical and mathematical ways of thinking relevant to computation, using precise technical formalisms, abstracting away from specific details of programs, computers, and languages, attempting to characterise properties of hardware and software in such a way that they are amenable to mathematical analysis and verification. Students should know what formal proofs are, and should ideally be able to produce them or at least understand and explain proofs produced by others. Since computers operate in a context students should gain some experience of the problems of studying, analysing and describing the context, which may be other computers, physical machinery, software systems, legal systems, organisations of various kinds, or individual human users. The latter can involve deep problems of human perception, learning, memory, problem-solving, motivation, etc. Students should learn to understand the relationships between computing systems, whether isolated machines or networks, and the environment in which they operate. They should be able to use that understanding not only to explain existing systems, but also in order to develop new applications of computers or to analyse failures in existing applications. This means that good graduates will have to have or be able to acquire and use knowledge that is not directly concerned with computers, e.g. knowledge about how an organisation works, or about the perceptual and learning abilities or motivations or human users. In particular, within an AI degree students should learn not only aspects of the above topics, but also about how human minds work, e.g. in perceiving, learning, planning, forming new motives, communicating, collaborating, or competing. This understanding can be put to use both in designing machines which simulate or model aspects of human competence, and in designing machines which function by interacting with humans, helping, them, teaching them, enhancing their abilities, providing new services, monitoring them, etc. This work can include a study of topics in brain science, psychology, linguistics, and philosophy: it is inherently multi-disciplinary. 1.2. Student variation Students differ in their innate ability, the knowledge and skills they have acquired by the time they come to university, their character and self-discipline, their financial support, their family background and support, their short-term preferences and their long-term ambitions. Consequently whatever course structures, syllabus contents, teaching methods, or laboratory facilities we offer them, we can expect very varied forms of learning and development. For that reason, and also because they will be going out to work in a wide variety of types of jobs, it would be inappropriate to attempt to assess them according to rigidly specified requirements in the form of the content of their knowledge and their computer related skills. In fact, we know that many employers are far better at such assessments than we could possibly be, since they have very specific requirements. (This is not true of all employers, e.g. research laboratories.) 1.3. The need for flexible and flexibly interpreted assessment More generally, it is not possible to give precise, objective numerical grades that adequately characterise what students have achieved by the time they leave, for the following reasons: o it is impossible to assign meaningful numerical measures to many of the skills we are interested in developing (e.g. creative design, effective communication, ability to abstract and formalise), o the ways in which students develop are necessarily different, as explained above, o there are no objectively ``right'' ways to weight different dimensions of knowledge and ability even within a particular course module, o the requirements of potential employers are extremely varied, so that measures, or relative weightings of different measures, that are appropriate to one will not be appropriate to others o insofar as grades are intended to be predictors of future performance in a variety of post-graduation careers, they cannot do this accurately, since two students with the same performance in particular first degree examinations may perform very differently later in a more demanding or more specialised context. Nevertheless we work in a system that requires us to give some sort of indication of achievement, and we therefore do our best to do this conscientiously, in the full knowledge that very often it is possible for different experts in the same field to give different grades for the same piece of work, because of their differing professional judgements about what is important, and in the full knowledge that the grades we give are likely to be misleading in many contexts. Although any such system is inherently unfair, types of unfairness are in part balanced out or averaged out by the fact that each student is assessed in different ways (practical work, presentations, written reports and essays, multiple choice questionnaires, and other types of unseen examinations), on different types of course content, and by different people. 1.4. Spurious mechanisms for achieving objectivity The search for complete objectivity, and to some extent also clarity, is unrealistic because the exercise of professional skill/judgement is the main basis for much of the assessment in any university, regardless of subject matter. It actually makes no difference whether or not complex exam procedures are used, with multi-part questions giving a few percent to each of many different points/topics/whatever. Even if it can be shown that such techniques produce marks more consistently, the subjective bias will already have been manifested in the particular marking scheme, and a different marking scheme, preferred by other professionals, would produce different results. I.e. the professional judgement in such cases is simply transferred to the construction of the questions, the selection of preferred answers among possible answers, and the relative weights given to the different questions or sub-questions: the objectivity remains illusory. Difficulty in making standards clear to students is a necessary corollary, to the extent that they will only be able to understand the standards when they are themselves professional. This requires that students (and others) trust the professionals to do their job conscientiously, without pretending to achieve impossible standards of precision and objectivity. 1.5. The main function of assessment In this context it may be argued that the PRIMARY function of assessment is not to produce an accurate characterisation of our students. Rather it is to help to motivate them and guide their choices as to how to spend their time while at university, and in some cases to contribute significantly to their learning. Particular forms of assessment can play a very important educational function. For instance project work is not only a means of assessment but a means of developing the abilities thereby assessed. Writing essays helps students learn to search for literature, to understand, analyse, criticise and evaluate the work of others, and to expound their own ideas, as well as helping them develop their linguistic skills. (However, without detailed individual feedback on both content and style, the educational opportunities presented by essays and other assignments are partly wasted.) Unseen examinations are much maligned, but they can make students look back at work they have done previously, and can help them to obtain a new deeper understanding of what they have been learning, using a synoptic vision required in preparation for unknown questions. By revising for a variety of examinations in a short time they may be helped to integrate and relate different topics whose mutual relevance was not clear while they were mastering the details separately. Of course, students may need help in gaining real educational benefit from these various means of assessment, and that often requires individual support and feedback: increasingly difficult in these times of pressure towards higher so-called ``efficiency''. ======================================================================= 2. REQUIREMENTS FOR VARIOUS GRADES (Draft) In general students need to produce work which: 1. Reflects awareness of the course content and familiarity with any special conventions, notations, techniques, concepts and theories which may be involved. 2. Addresses the exact question or project specification or essay title given, without omitting important items or adding irrelevant padding. 3. Is approximately the required length, or within the specified bounds, without exceeding the limits (where length is specified). Essays and reports must: 4. Demonstrate that the student has done reading relevant to the course, and not merely attended lectures and read lecture handouts, 5. Conform to the conventions of academic writing and presentation, and adhere to any format or structure required for the assignment. 6. Contain a bibliography which includes all the references that you use in your assignment, properly formatted with all required information and with appropriate citations in the main text. Having met these requirements, how good a grade the student gets will depend on how well the work meets the more detailed requirements given below. ======================================================================= CRITERIA FOR VARIOUS CLASS LEVELS FAIL (Mark range: 34% and below (F)) Work may be failed for many reasons including non-submission, plagiarism, collusion, answering a different question from the one asked, writing with insufficient clarity, or simply not meeting the requirements for a pass work, listed below. PASS (Mark range: 35-39% (E)) Work of this type displays the following characteristics: 1. The student has clearly learnt the basics of the subject, can explain them to some degree, has acquired a minimal set of programming skills, and has gained some benefit from attending the course. (More than could be gained simply by reading the popular press and watching TV programs.) 2. The level of competence or knowledge achieved would enable the student to work in an environment where some understanding of the subject is useful, though not necessarily as a trusted professional, nor as a teacher of the subject. 3. The programming or systems analysis skills acquired may be appropriate to working under the close supervision of a manager, though the student may not be able to design and develop software unaided. 4. In some cases the students will be able to document or explain programs produced by others, even if they cannot produce such programs themselves. 5. The students may have gained enough knowledge of computing to work in an environment where computers are used, or where communication is required with programmers and computer managers, or where computing products are sold. 6. There may be evidence of ability to get a higher grade according to the criteria listed below, but this may be marred by poor presentation, serious gaps in knowledge, serious errors, bad spelling or grammar, lack of organisation, assertions with insufficient arguments, late submission, etc. CLASS III (Mark range: 40-49% (D)) The overall impression created by the candidate's submission is that it is adequate, but only adequate. The student will at least have achieved enough to obtain a pass mark, as described above, and in addition displays most of the following characteristics: 1. The student has clearly benefited from attending the course, shows understanding of a significant subset of the main concepts, notations, techniques, and theories presented in the course. 2. The student shows evidence of having read and understood key texts from the recommended reading list and having attended most of the lectures, and understood a significant amount. 3. Explanations are reasonably clear, and essays are well organised with adequate spelling and grammar. The student is able to explain to a non-expert what the course is about. 4. The student is able to answer the questions set, and avoid irrelevant digressions or inappropriate answers. In general, examination instructions are followed. 5. The student is able to expound ideas of others well, at least in the simpler cases, even if he or she is not able to analyse, criticise or develop those ideas. 6. The student is able to design, implement, test, debug and document programs of relatively simple sorts, and might be a useful member of a software development team, though mainly working on designs produced by others. 7. The student is able to do some independent research in order to find out what has been achieved by others. 8. The student can construct convincing arguments by assembling evidence and drawing conclusions reliably. 9. There may be evidence of ability to get a higher grade according to the criteria listed below, but this may be marred by poor presentation, poor arguments, errors or gaps in knowledge, bad spelling or grammar, lack of organisation, late submission, etc. CLASS II ii (Mark range: 50-59% (C)) The overall impression created by the work is that it is good; the candidate has attained a standard that meets not only the criteria for a CLASS III mark described above but also a significant subset of the following: 1. The student has a sound knowledge of the main areas of the course (as defined by the combination of compulsory and optional courses taken by the student) and can display this knowledge by giving clear, accurate, and comprehensive explanations. 2. There is plenty of evidence of work going beyond a study of the material presented in lectures and handouts. 3. The student can give reasonable answers to questions not previously encountered, and is able to analyse and criticise the work of others as well as expound it. 4. The student can go beyond answering the easiest questions and has some grasp of the more difficult material in the course, including the ability to explain that material to others. 5. The use of notation is accurate and the forms of argumentation are generally sound. There is no confusion between ideas or conclusions supported by evidence and argument and those which are merely based on intuition, ungeneralisable personal experience, or prejudice. 6. Answers to questions are always relevant and presented in an interesting and informative way. 7. The student is self-critical and able to identify weaknesses as well as strengths in his or her own work, whether it is an essay, an argument a piece of software, a proof, etc. 8. The level of programming and system development skill achieved is such that the student can solve non-trivial tasks, explain how the programs work, and produce good documentation. 9. The student has not necessarily shown sufficient research ability to go on to do a PhD but is nevertheless likely to benefit from further study and could be recommended for an MSc course in computer science. 10. The student could do well in a job which involves managing or working closely with software developers, producing specifications for software, maintaining complex software produced by others, writing documentation, writing news or magazine articles on computing developments, or perhaps teaching beginners programming in a school or college or industrial setting. CLASS II i (Mark range: 60-69% (B)) In order to achieve this sort of mark the student should at least achieve the standards described above, and in addition give an impression of producing high quality work meeting most of the following criteria: 1. All or almost all the material presented in the course has been understood very well and the student can explain it sufficiently accurately and clearly to show promise as a teacher of the subject in higher education. 2. The student reads beyond the set material and is able to synthesise and analyse critically the work of different authors. 3. Notations and formal techniques are well understood, and the student can use them in solving quite hard problems, or producing difficult proofs. 4. There are signs of an ability to identify new unsolved problems or unanswered questions and to undertake research towards solving them, whether it is empirical research involving collection and analysis of data, theoretical research constructing proofs or counter-examples to supposed proofs, or designing and implementing sophisticated software. 5. Software produced by the student is non-trivial, well designed, clearly structured, elegant, thoroughly tested, robust, and well documented. The student is able to distinguish different levels of documentation, for end users, for developers and system maintainers, and is able to explain design options and justify design decisions. The student shows a clear understanding of the mathematical properties of algorithms (e.g. can analyse their complexity, or termination conditions) and can demonstrate them both by analysis and by experiment. 6. The student is able to give clear, confident and interesting, presentations on his or her work or the work of others, and to deal with questions adequately and to respond adequately to criticism or questions. 7. There are very few factual or technical errors, or gaps in essays, project reports or answers to examination questions. 8. The student's grasp of English, including spelling, grammar, vocabulary, and literary style is of a high order, e.g. comparable to that found in good journals. (In the case of students who are not native speakers of English, some allowance may be made.) 9. The student gives clear evidence of independent thinking, always critically analysing information presented, rather than accepting material in lectures or in literature on authority. 10. There is evidence of ability to synthesise and relate material presented in different courses, drawing out connections and implications not made explicit in those courses. In some cases the student can relate the course material to information from another discipline in which the student has done independent reading without necessarily attending a formal course. 11. An employer could expect such a student to be a good designer of software, or systems analyst, or project manager, or researcher, with the prospect of rising to a senior position eventually. 12. The student is able to organise his or her learning and project work well, submitting items on time and where appropriate cooperating well with others. 13. The student could clearly benefit from continued study either in a master's degree or a research degree, depending on whether there is sufficient evidence of originality and depth. CLASS I (Mark range: 70%+ (A)) In order to achieve a mark in this range the student should at least meet the criteria specified above, and in addition give the impression of having a deep and comprehensive understanding of the subject matter and meeting many of the following criteria: 1. Essays and answers to examination questions not only provide the required information clearly presented, but also gives evidence of subtlety and sophistication in making distinctions, drawing inferences, identifying flaws in published work, and developing new themes. 2. Oral presentations are of very high clarity, showing excellent communication skills, e.g. alternating effectively between high level ideas and low level details, and the ability to answer difficult questions well, and detect subtle misunderstandings in questioners. 3. There is evidence of great breadth of reading, including extended literature searching, and possibly having found up-to date and relevant materials beyond those on the course reading list, including possibly literature from other disciplines. 4. There is evidence of great depth of understanding as shown by the accuracy of reporting, the ability to produce original illustrative examples, the ability to make incisive contrasts and comparisons between the work of different authors, the ability to come up with new criticisms or better versions of old proofs or better answers to problems, and the ability to identify interesting new unsolved problems and make some steps towards solving them. 5. The student's written work shows complete mastery of scholarly presentation: references to texts are carefully integrated into the argument, quotations are judiciously selected (with proper references), and bibliographies are comprehensive and well presented. 6. The student's work is generally stimulating and thought provoking even to the course tutor. 7. Programming projects tackle difficult problems and solutions are clear, highly modular, elegant, intricate only where necessary, well commented, and the program works as specified. The student shows an ability to understand limitations of the current state of the art and to think about ways of going beyond them. Precisely how high the mark is can depend on how difficult the task is, how good the solution is, how good the code is, and above all, how good the report on the project is. Where the student tackles a theoretical (non-programming) project the report shows a high degree of independence of thought, the ability to analyse difficult problems, to find good generalisations, to produce and criticise non-trivial proofs, to select and use good notations, and an understanding of how the work potentially relates to other areas of computer science. 8. The student's ability to identify problems, to think independently, and to develop new solutions, arguments, notations, or theories indicate potential to do a PhD. 9. In some cases the student may obtain a first class mark (70% +) on the basis of extremely solid, thorough, accurate, comprehensive reporting, on the basis of consistent hard work, though without very great originally. Outstanding answers (80% + ) should be arrestingly original, and (for 90% +) almost complete in some obvious sense. Perfection earns 100%. ======================================================================= ADDITIONAL NOTES: KEY SKILLS This is a first draft list of general key skills not already included in the degree-specific skills and know-how referred to above. (That is why programming is not mentioned here.). The following need to be expanded: KEY SKILL: PERSONAL COMMUNICATION This should be embedded in all years of study (presentations of projects, of reports, of seminar papers, etc.). KEY SKILL: LITERACY This includes both the production and reading/processing of written material, including `technical' material in the form of charts, graphs, etc. It should be embedded (in work on essays, reports, etc.), in all years. KEY SKILL: NUMERACY This is presumed of all students - GCSE maths and a science subject at A level are standard requirements for entry to courses. KEY SKILL: MATHEMATICAL MATURITY This is hard to define but includes the ability to create and use and reason about non-numeric formal structures (e.g. grammars, sets of axioms, proofs, parse-trees, etc.) KEY SKILL: IT The general ability to use computers, word-processors, internet browsers, search engines, etc. is now a common requirement for many degrees. Such skills have nothing specific to do with Computer Science (unlike techniques for developing the mechanisms). These skills are taken for granted throughout the course, but should be developed explicitly (e.g. students must not be allowed to think there's only ONE word processor!) KEY SKILL: TEAMWORK Includes group projects. KEY SKILL: LEARNING TO LEARN Students doing Computer Science and AI do not generally have a pre-university school-based experience of the subject matter of which they can say their university course is the same but more advanced. This means they have to learn about learning, to some degree. Two specific issues are the requirement to learn formal logical/structural approaches to problem solving, in the general sense, and the requirement to learn how to extract generic points from specific examples, so that they can learn a new programming language and evaluate its suitability for specific projects/applications. In general the process of learning to learn should be embedded throughout the degree programme. One danger of too much spoon-feeding (e.g. detailed lecture handouts) may be that students do not learn to do enough themselves, including taking notes in lectures and then writing them up.