Logo: Deutsches Institut für Internationale Pädagogische Forschung


ReCo: Automatically Processing Text Responses from Large-Scale Assessments

The project Automatic Response Coding (ReCo) centers around text responses in tests. The computer program ReCo automatically assesses whether a text response is correct, for example to the PISA test, such as “The author aims at saving the trees.” Moreover, ReCo extracts further features, for example whether a student adds knowledge to their response beyond the explicit information in the text.

The computer program ReCo was originally developed at the Technical University of Munich and in the Centre for International Student Assessment (ZIB). In cooperation with TUM and ZIB, the TBA Centre at DIPF is now developing the software package further. The project comprises the general software development as well as scientific studies employing ReCo. For this, TBA is responsible for the conceptual and technical development on the one hand and acts as the project manager for projects such as ReCo-Multi on the other hand. Also, TBA supports external research groups using the program.

Enhancing Educational Measurement

When constructing tests, research groups are often faced with the question which response format should be implemented. If they offer response options such as in the multiple choice format, responses can be easily processed. The downside is that this closed format entails guessing and other undesirable effects. But if the respondents can write their response into an open text field, the processing is often significantly complicated. At the same time, open response formats are better suited to assessing deeper understanding than closed ones are for most domains, because the respondents are required to produce a response themselves rather than just recognizing the correct one.

Therefore, the OECD already for PISA 2000 decided to include some tasks with open response format. The resulting effort for processing the text responses is particularly challenging for large-scale studies like PISA. Each participating country needs to train a team of coders diligently in order to evaluate the large amount of text responses from more than 500,000 students in total in the recent rounds. The coding team has guidelines for this task so that the coding takes place as objectively and comparable across countries as possible.

Because human beings have subjective perspectives and cultural backgrounds, the risk of inconsistent coding needs to be considered. Owing to their lack of stamina, humans are also prone to errors when being exposed to such a large volume of data. Computer programs such as ReCo can improve consistency and, in turn, objectivity within as well as across countries. Also, the large volume of the data is not an obstacle for the computer guaranteeing further consistency at a large scale. First studies with German data showed that ReCo is capable of such automatic coding to a promising extent.


ReCo-Multi: ReCo Learns Multiple Languages


Free Program

ReCo can be used upon request. Currently, the usage is constrained to German text data only. Over the course of the project, TBA will develop a graphical user interface for the program and both will be freely available for download.





Cooperations: Technical University of Munich, Centre for International Student Assessment (ZIB) e.V.

Duration: since 04/2017

Status: current

Contact: Fabian Zehner, Frank Goldhammer

last modified Jun 22, 2017