AR, I had carried out many interdisciplinary projects in the meantime, and I had also experienced the usual problems in projects like this, but in this specific case there were no real barriers. Kristina is a math educator who is very familiar with psychology, and I'm not too distant from mathematics education. I was always somewhat interested in mathematics

#### 9 Educational Research on Learning and Teaching … 231

education even before then. And as our perspectives were not that divergent, there was a lot of productive cooperation; it's also helps when there's mutual affinity, as things then go relatively easily.

#### *What consequences did this close cooperation have for the results or for the way you tackle the problems?*

AR: We conducted some shorter controlled studies managed primarily by us and by the psychologists, and that were inspired by the math-education perspective. On the other hand, the more complex classroom studies were done by math educators. Thus everyone's contribution was in his or her field of expertise. This went quite well, and we also benefited from the advantage of obtaining findings from the lab, from more controlled lab-type studies, as well as from the classroom - that's ideal if you want robust findings.

In the second step, I also always need to know what works in the classroom setting, so this cooperation also had the advantage of satisfying this desire of an educational psychologist, namely what I find in the lab can be applied in the classroom.

#### *Could you also claim there was an influence on math education in general or educational psychology coming from your project?*

AR: I would say yes, at least in the German math-education community. I got the feeling that worked-out examples didn't play a genuine role initially. Most of the math educators have absorbed a bit of old-fashioned pedagogy. And the idea developed in psychology of using worked examples as specific means of fostering understanding was sometimes hard to get across to math educators. At least I found it hard to explain - to convince those colleagues; however, this project and our results made the math educators more curious about the potential of worked examples for fostering understanding and so it was a kind of door-opener for this idea to those educator communities.

#### *Would you say that there has even been an impact on the classroom, in the German mathematics classroom?*

AR: Perhaps, some ideas from our research are included in some mathematics textbooks. So there may be a minor influence, but I would guess that it is primarily evident in the design of textbooks. I would not say that the typical German math teacher knows much about the theory of example-based learning.

#### *How would you say that this specific project influenced your own further studies, your own research later?*

AR: I think it had a huge influence, because we grew to realize that worked examples can be applied for algorithmic content as well as other fields. And so we used structurally similar worked examples for non-mathematical areas, and they worked well in those areas, which opened up a totally new field of application possibilities for worked examples. That was a strong influence.

The second benefit was connected to mathematics. I was then also interested in whether I could set aside other restrictions of typical worked-out examples that math educators often pointed out, for example multiple solutions. We then devised worked examples with multiple solutions. And they work if you use them in a good way.We aso devised worked-out examples containing typical errors and guided students to process, to listen in a beneficial way.

Finally over the last several years I have written theoretical papers about what I call focused information processing. Many of these basic ideas originated from this project on proof finding.

The two examples from research demonstrate how empirical research profits from combining theoretical and methodological perspectives of general education research (educational psychology) and of research in mathematics education. It is important to note that, in both examples the "binding force" in the collaboration was not a simple sort of work sharing, such as. bringing content expertise and methodological expertise into a project. It was rather the complementary theoretical perspective of a relevant situation in teaching and learning: The commonalities of (subject-independent) selfregulation and (subject-specific) problem solving made it fruitful to combine the two aspects and seek parallels or synergies in their combined instruction. Furthermore, it was the partners' goal in both examples to extend the knowledge in their respective area of interest, e.g., taking the step from worked examples to heuristic worked examples. This step was generated in a productive struggle between the researchers' field of general learning and mathematical proving with respect to the meaning of each of their theories for their partner's field.

Furthermore, one also notes in both projects a certain degree of circumspection in developing and implementing an intervention: The researchers carefully plan the interventions not only considering their power to (dis)prove a hypothesis empirically but also to accommodate an everyday classroom environment, e.g., via a multi-step approach—the research presented here builds upon many studies that the researchers conducted already (on their own) to develop and validate their theories.

The aforementioned projects are inspiring examples of research in (mathematics) education in Germany, and many more researchers have had similar experiences. However, while multi-step research focusing on central phenomena of mathematics learning and teaching are quite prevalent in mathematics education research in German-speaking countries, interdisciplinary research is still the exception, since researchers in subject-matter education and general education often tend to have different backgrounds and belong to different university faculties.

## **9.3 Mixed-Methods Research**

The second section introduces two research projects that draw equally from qualitative and quantitative methods in a mixed-method design. As an introduction, a condensed theoretical framework is presented that we refer to again in the discussion of the two research projects.

In recent decades, researchers in mathematics education have expanded their repertoire with research designs that suffice for and can tackle especially multifaceted research problems by integrating different methodological practices, eventually with a mixture of qualitative and quantitative research questions suitable for a complex empirical and theoretical research topic (Johnson and Onwuegbuzie 2004; Hart et al. 2009). However, a review of the latest literature reveals many different theoretical justifications for and definitions of mixed-methods research (MMR), some of which are contradictory (cf. Kelle 2008; Teddlie and Tashakkori 2009).

Although Denzin (1970) originally referred to a combination of multiple forms of qualitative research methods only (Denzin 2012), especially his concept of triangulation has often been referred to in German-speaking countries as a theoretical research background for integrating and employing several (i) data sources, (ii) investigators, (iii) theories, and (iv) methodological practices—both qualitative and quantitative—within a single research design. Remote from its original understanding, the term triangulation is frequently referred to with two different meanings (Schulz 2010; Kelle and Buchholtz 2015, p. 331): (a) "triangulation as a mutual validation of research results" and (b) "triangulation as an integration of complementary perspectives on the subject being investigated to create a more complete image of the research domain."

Being suitable for both of these meanings, the importance of MMR can be justified by taking a closer look at the specific strengths and weaknesses of qualitative and quantitative methods, and how a balance can be struck by different types of MMR designs (Kelle 2008; cf. Schulz 2011): Qualitative research is restricted to the careful choice of a few cases. Moreover, the analysis of unstandardized data is highly dependent on the individual researcher. This "immediately raises questions about the generalizability of findings and about the intersubjectivity of interpretations" (Kelle and Buchholtz 2015, p. 336). On the other hand, qualitative methods are often useful for exploration, detection, and discovery. They help us "construct new theoretical concepts, categories, and sometimes even whole theories about the domain under study" (ibid.). Quantitative research relies on standardization and requires reliable and objective (observer-neutral) data. This enables the investigation of many cases, which contributes to a theoretically grounded basis for generalizations. Quantitative methods are ideally suited for a theory-driven approach, "whereby precise hypotheses are formulated at the onset, then operational definitions are formulated for the concepts these hypotheses comprise, measurement instruments are constructed, and data are collected and analyzed subsequently" (ibid., p. 334). They are therefore appropriate tools to examine clear-cut causal statements. However, the researchers' theoretical knowledge or that of the corresponding discipline may be limited. Furthermore, context-bound patterns, structures and rules may form an integral part of particular life worlds under investigation that may be unknown to the researchers, especially if the socio-cultural background of researchers and respondents differs. Such a lack of knowledge may cause severe problems when formulating hypotheses, defining variables, and creating research instruments. Specifically, when examining causal statements, important variables may be "omitted with low levels of explained variance as a consequence", intervening variables may be overlooked, or functional relations between certain variables may be incorrectly specified "so that causal processes underlying the investigated phenomena are not adequately understood" (ibid., p. 335; cf. Kelle 2008; Schulz and Wirtz 2012).

Being driven by awareness of the specific strengths and shortcomings of monomethodological research designs, MMR designs may be used as strategies "for overcoming each method's weaknesses and limitations by deliberately combining different types of methods" (Brewer and Hunter 1989, p. 11). Meanwhile, various designs and manifold categorizations for MMR designs exist and are discussed in Germanspeaking countries (e.g., Kelle 2008; Kuckartz 2014; Kelle and Buchholtz 2015; Burzan 2016). To discuss many of them would clearly go beyond the scope of this chapter. Instead, we present a concise view into the world of MMR designs that suffices as a framework for discussing the MMR design examples discussed later in this chapter. We concentrate on four prototypical MMR designs:

Complex designs require extraordinary effort that often can be coped with by wellcoordinated teams only. If individual researchers hope to create a MMR design, they will require expertise in both qualitative and quantitative research methods. Furthermore, they should be aware of the specific strengths and weaknesses of qualitative and quantitative research methods so as to combine and integrate them productively and well targeted.

In the following, we present and discuss two examples of MMR designs from Germany: Kathleen Philipp made use of a sequential mixed-methods design (Philipp 2012; Philipp and Leuders 2012). She analyzed students' strategies while solving several mathematical problems and developed a competence model about experimental thinking in mathematics. This laid the groundwork for an intervention study that confirmed that experimental skills in mathematics can be fostered effectively.

Prediger and Wessel (2013) and Wessel (2014) implemented an integrated, parallel mixed-methods design. They fostered students' understanding of fractions, and scaffolded the learning processes by facilitating students' abilities to talk about fractions and their meaning. Their intervention's efficacy was investigated by a randomized controlled trial and qualitative analyses of the videotaped teaching-learning processes.

## *9.3.1 Example 3: Understanding and Teaching Experimental Thinking in Mathematics*

Knowledge acquisition in the natural sciences is based on experiments. Experimental thinking includes the generation and confirmation of hypotheses. There are similar processes of knowledge generation in mathematics: relationships between different kinds of objects are explored, hypotheses are generated and examined by producing and analyzing examples—as described by e.g. Euler or Pólya. This is the starting point of the project by Philipp et al. (2012) described below.

Despite the analogy, knowledge acquisition in mathematics is quite seldom associated with experimentation (e.g., Bartolini Bussi 2009). Proof and the axiomatic method dominate when establishing mathematical knowledge is discussed. However, that neglects the essential role that experimental thinking in mathematics plays in (i) generating and confirming hypotheses by examining examples before coming to any final proposition, and (ii) in arriving at a mathematical proof.

Concerning the natural sciences, scientific discovery can be described as a process of Dual Search (SDDS, Klahr and Dunbar 1988). The search in two problem spaces (one hypothetical, the other experimental) shapes hypothesis generation, experimental design, and the evaluation of hypotheses.

In their first study, Philipp et al. analyzed the experimental activities of pupils in mathematical problem situations (Leuders et al. 2011). A theory about experimental thinking in mathematics was developed relying on those findings while keeping the theoretical background of experimental thinking in mind. In the following study, experimental thinking skills in mathematics were identified and operationalized. This measurement was used in a pre-post test design to assess the learning effect of an intervention that fostered students' skills in experimental thinking.

The first study's cohort consisted of nine students (prospective teachers) and twelve pupils (grades 3 and 4). They were videotaped while solving mathematical problems. The problems had been chosen to be open-ended and to trigger many hypotheses and examples to solve the problem. One of those problems was the "sums of consecutive numbers" (see Fig. 9.4: "stair numbers").

The analysis made use of Grounded Theory. Procedures were identified and coded. The system below of four core procedural categories became apparent via axial coding: (1) generating examples, (2) structuring examples, (3) generating hypotheses, and (4) testing hypotheses. These four core categories are interrelated in the experimental-thinking process while trying to solve a problem (Fig. 9.5).

In the next quantitative study, the intervention was developed according to approved design principles for fostering problem-solving strategies (Bruder 2003) and included four phases: the introduction of problems, explaining problem-solving procedures, reflecting on procedures and applying them to other situations. Participants were sixth graders (middle school): 126 students were assigned to the intervention group, and 101 students to the control group. The intervention lasted about three weeks. A pre- and post-test with a control group were administered immediately before and after the intervention, a follow-up test about six weeks later. The pre-,

The number researchers (Till, Ole and Maria) investigate stair numbers.

What can you find about stair numbers?

post- and follow-up-tests included items for measuring "structuring examples" and "testing hypotheses". The analysis revealed a significant increase in large effects in both areas, which remained stable over 6 weeks. The large effect sizes support the authors' claim that such procedures can be fostered effectively in regular mathematics classrooms.

#### *Interview with Kathleen Philipp (KP), about her experiences within the MMR design described above*:

#### *How did you come up with your research question?*

KP: "When I was a teacher instructing my pupils in class, I often observed curiosity in children's mathematical thinking. I was given the chance to take part in a large research project as a PhD student on the topic of "experimentation" in mathematics or natural sciences. Being embedded in that research context, our research goal was to analyze the significance

**Fig. 9.4** Sums of consecutive

numbers—example problem (cf. Philipp 2012, Abb. 16.1.)

#### 9 Educational Research on Learning and Teaching … 237

of experimentation in mathematics, and I was pleased to be able to take a closer look at the experimental thinking of students in the mathematics classroom."

#### *How did your theoretical background influence the research design?*

KP: "The theoretical background concerning processes that are characterizable as experimental approaches was very helpful for me to understand the importance of such processes. When Euler, and later Polya, described experimental processes, I realized it was worth trying to see whether children can do mathematics in a similar manner. One of the challenges in the beginning was to find tasks for students that allowed them to find their own ways to solve a problem, even when they were unfamiliar with such tasks. It was also clear that the interesting processes are internal ones and that it would be hard to observe and analyze them, in particular when young children are involved. To overcome these challenges, I used different methods to collect data.

The theoretical background of the two problem spaces from Klahr and Dunbar helped me understand the complex processes many students demonstrated in these settings. It was very obvious that examples and hypotheses play an important role in mathematical experimentation. But there was something else we needed to understand. I refined the existing theory about two problem spaces and identified a third space that characterizes the kind of strategies used to switch between the two spaces, i.e., between examples and hypotheses."

#### *How did you choose and develop your research strategies?*

KP: "First I identified the experimental strategies students use when exploring mathematical problems. When I noticed that students do mathematics in a manner resembling that of mathematicians, I wondered if it would be possible to teach such experimental strategies in class. To investigate this, I had to think about a means of measuring such strategies. Then I developed tasks that would reveal whether experimental strategies were being applied to find a solution. The third step was to develop a teaching unit in the standard curriculum in which I could integrate the training of experimental strategies. Finally, I had the teaching unit taught in several classes. All classes were taught by their regular mathematics teachers. At the end, I compared how frequently the respective students used experimental strategies successfully with classes that had not participated in the training session but who had been taught the same contents by relying on the school textbooks."

*Which kinds of triangulation(s) did you implement in your mixed*-*method design?*

KP: "I triangulated several theories and data sources. To generate and analyze my data, I applied both qualitative and quantitative methods."

#### *Why did you implement those kinds of triangulation? Can you describe the interplay of or roles of 'theory development' and 'theory confirmation' in your study?*

KP: "The development of my research project required different methods. To identify experimental strategies, I had to take a qualitative approach, because the aim was to investigate and understand students' approaches in depth, and to create a foundation for developing a theory of experimentation while learning mathematics. I therefore applied the Grounded Theory to analyze the students' approaches, which has the advantage of enabling you to develop a theory from your data. Thus, as is typical when using Grounded Theory, it was feasible to identify core strategies that students apply in experimentation.

Quantitative methods were necessary to measure the use of such experimental strategies, which made it possible to investigate a larger group of students (about 230) and make comparisons with a control group.

On the one hand, different methods were necessary to find answers to the specific research questions. On the other hand, the measuring and fostering of experimental strategies validated the theory. I relied on different data sources to observe and assess experimental strategies."

*Was implementation in the classroom important from the start?*

KP: "Yes, implementation in the classroom was important for me personally. Working as a teacher for many years, it was my research concern to link my research findings to actual practice. In my opinion, it is crucial that research results in our field be relevant in mathematics education."

*How did the project influence your later research projects personally?*

KP: "For me, this research project was fundamental to the evolution of my research interests. It is still important to me that research results be closely associated with practice. Another aspect is that I have a deep interest in understanding people's thinking and their thought processes."

## *9.3.2 Understanding and Teaching Fractions*

Our description of the fourth project example follows Prediger and Wessel (2013) and Wessel (2014) who work on language challenges and conceptual mathematical challenges for students with low language proficiency. Language proficiency is the key determinant of mathematics achievement—not immigrant status or being multilingual (Heinze et al. 2009; Prediger et al. 2018). Students with low language proficiency encounter reading obstacles as well as difficulty in developing conceptual understanding (Prediger et al. 2018).

To support students' constructions of meanings for new mental objects and relationships (Steinbring 2005), the authors draw upon the principle of relating verbal, symbolic, graphic, and concrete representations (Duval 2006; Lesh 1979; Bruner 1967). Conceptual learning processes involve the acquisition of new linguistic, graphic, and symbolic means for expressing mathematical concepts and their meanings (Schleppegrell 2010). This illustrates how deeply interwoven mathematics learning and language learning are (Pimm 1987).

The principles of relating representations has been extended to relating different linguistic registers also, such as establishing first- and second-language registers, mathematical technical registers, a school language register, and the students' everyday registers (see Fig. 9.6: Prediger and Wessel 2013). "*A register can be defined as the configuration of semantic resources that a member of a culture typically associates with the situation type… in a given social context*" (Halliday 1978, p. 111).

The school register is characterized by context-reduced and complex linguistic means. It differs from the everyday register by appearing conceptually written even if medially oral. Unlike the technical register, the school register is seldom taught explicitly in school. In particular, this is problematic for underprivileged language learners who acquire only everyday language and vocabulary at home but nevertheless require the school register for higher thinking skills (Clarkson 2009), while privileged (first- or second-) language learners are exposed to the school register already at home (Schleppegrell 2004; Cummins 2000).

An intervention was designed that included translations between all registers and related them to each other. The choice of suitable activities for each moment in the

**Fig. 9.6** Registers and representations (Prediger and Wessel 2013, p. 438)

learning process was guided by the design strategies of "pushed output" (Swain 1985) and scaffolding (Hammond and Gibbons 2005). The language- and mathematicsintegrated intervention aimed to initiate the construction of mathematical meaning in combination with opportunities to establish the required language means (Prediger and Wessel 2013: Fig. 9.7):

The intervention study took the mixed-methods approach to investigate effects from two research questions in a triangulating manner (Prediger and Wessel 2013, p. 443): "(1) To what extent do students who participate in the language- and mathematics-integrated intervention improve their achievement in the fraction test? (2) What is the situational potential of the intervention activities to initiate students' constructing of meanings and activating of linguistic means in the school and technical register?"

The study sample consisted of 72 seventh-grade second-language learners with below average math performance and limited German language proficiency. The intervention consisted of six lessons lasting 90 min in 2-to-1 sessions (two students with one teacher). A pre- and post-test randomized controlled trial design with a control group was realized (Fig. 9.8).

The control group was taught by their regular teacher with the usual fraction textbook repetition program. The 18 × 540 min were video-taped, and relevant episodes were selected according to research question 2. The data corpus for the qualitative analysis comprised selected videos, all written materials from the interventions, and the teachers' lesson plans.

The quantitative analysis of the learning gains revealed a low-to-medium effect (intra-group effect size measured by d - 0.42) in the control group, but a very strong effect (d - 1.22) in the intervention group. Those differences became significant in a later study with a larger cohort (Prediger and Wessel 2018).

**Fig. 9.7** Macro scaffolding (Prediger and Wessel 2013, p. 444); "Anteil" in (e) is used deliberately by the authors in their translation of the task. It signifies the mental image connected to a fraction as a part of a whole

Quantitative analysis of learning effects was complemented by a qualitative analysis to exploratively reconstruct the situational potential of the design strategies and instructional activities in the learning processes (Prediger and Wessel 2013: Fig. 9.8). The qualitative analysis included in-depth analysis of the videotaped teaching–learning processes: (1) the aim of sequential analysis was to reconstruct individual and interactional processes of meaning construction for fractions, (2) several qualitative coding procedures for students' utterances were conducted to capture different linguistic aspects, and (3) students' development of mathematical meaning and lin-

**Fig. 9.8** Mixed-methods design (Prediger and Wessel 2013, p. 445)

guistic means were reconstructed in their interaction with concrete elements of the instructional design.

This qualitative analysis identified the specific potential of macro- and microscaffolding in combination with the design strategy of pushed output.

The MMR design was reflected in the study findings (Prediger and Wessel 2013, p. 453): "Whereas these quantitative results alone might have been explained on the basis of the intensity of individual teaching, the case studies gave some first insights into how the design elements offered fruitful learning opportunities, since they showed the deep interconnection of linguistic and mathematical learning processes: …With the help of macro and micro scaffolding, low-achieving students successfully proceeded from their everyday language to a language for thinking and talking about structural relationships."

#### *Interview with Susanne Prediger (SP), about her experiences within the above***-***described MMR design:*

#### *How did the research question emerge? Was it a rational decision?*

SP: "A research question is also a matter of personal choices, but it is of course based on a rational decision. Regarding the challenges of language learners in mathematics: by 2011, we had a situation where we had a lot of statistical evidence of connections between language proficiency and mathematics learning. But we did not know much about how to foster language learning. We thus chose to continue to dig deeper and beyond the current descriptive knowledge, to find answers about how to foster language learners and what could work for them."

#### *How did the theoretical background influence the research design?*

SP: "That it definitely does. In our case it's how you conceive of the role of language in the learning process. When we consider language only as relevant in its communicative function, it might suffice to gather knowledge about how to approach word problems and alleviate language difficulties in word problems. But we start from the assumption that language has a deep cognitive function within mathematical learning processes. If we want to delve deeper into that, we need to study the learning processes, and see how the interplay between language and mathematics really works. That is why we had to initiate mathematics- and language-integrated processes. In everyday classrooms, we observed hardly any conceptual learning opportunities for students with low language proficiency, which created the need to establish an instructional design specifically for those students as a research context for our research questions."

#### *What kinds of triangulations did you implement in your mixed*-*methods design?*

SP: "First of all it is a triangulation of data, because we wanted on the one hand to gather data about the intervention's function to see if it works. To test what works, you need a pre- and a post-test, you need to conduct a randomized controlled trial. But if you want to learn *how* it works, you really need data on the actual processes, meaning video data and qualitative analysis.

On the theoretical level, we preferred working with networking theories. It is a similar idea, but it is not just about putting the two next to another. You really have to integrate the different theories. Regarding this topic of language and mathematics education, of course you need theories about mathematics learning and you need theories about language and its role, and they have to be combined."

#### *Can you describe the interplay between 'theory development' and 'theory confirmation' in your study?*

SP: "In the classical sense we only discuss confirming theories, which is all about hypotheses and validating them. The randomized controlled trial is the research approach of choice for validating hypotheses. In our case, we could validate the hypothesis that students with low language proficiency learn better when they are given language- and mathematics-integrated learning opportunities.

However, before you can validate a hypothesis, you must develop categories to characterize phenomena and to generate hypotheses to be validated. This is where qualitative, explorative research approaches are most suitable.

In our research, we really combined these two approaches: we validated a hypothesis that had been quite general and invested a lot of time and energy to generate a theory on a more fine-grained level. For this, we had to develop the categories in which we could formulate phenomena and refined hypotheses. The importance of generating categories is often underrated.

In the present project the validation of the hypothesis that a language- and content-integrated intervention might be more effective was only a starting point. To elaborate upon our theory, we had to investigate the processes in depth to understand what exactly happened in the learning processes. And this qualitative research enabled us to generate categories that we could use as a starting point and for initial hypotheses for the next project. Within the next project, which is now being funded by the German Research Foundation, we again have some hypotheses that we can validate or reject, and we will generate new questions and new aspects to be studied in greater depth. (Prediger and Wessel 2018)

Thus, there's always interplay between hypothesis generation and its validation. Indeed, the categories are really the most important element in terms of theory generation. There is a worthwhile article by Cobb and diSessa, who talk about ontological innovations. Their article puts emphasis on the generation of categories as the major output of research."

*What do you personally prefer: to start with a qualitative study, or take a quantitative approach?*

SP: "(Laughs) Neither nor - it's more like going backward and forward. Sometimes, like in this project, there was quantitative evidence of a problem with language proficiency and mathematics achievement. Thus, we started to understand what the problem was qualitatively, and conducted design research cycles to devise an intervention that we could put to the test in a randomized controlled trial. Yet I would never want to conclude with a randomized controlled trial, because we need to understand how things work and what really happens in these processes. So, it really is a backward-forward movement. I'm really pleased that we have funding for two successive projects, including the one described here. In both we dug deeper and deeper into the details about what students really need, into what kinds of fostering in language terms they require, and into the type of fostering that is genuinely effective.

It's like the question about the hen and the egg - you can't tell where to start, you have to take a combined, cyclical approach."

#### *Summarizing comments on the two MMR projects and the interviews*

Many researchers in mathematics education consider themselves to be engaged in an applied discipline, where research should contribute to improving education. Learning mathematics with understanding is a complex process, and (Hart et al. 2009, p. 39) "mixed methods research may be an appropriate response to calls for greater generalizability of results while maintaining enough detail about the processes of teaching and learning to be valid and useful." This idea on the purpose and use of MMR in mathematics education was expressed clearly by the two researchers we interviewed, Kathleen Philipp and Susanne Prediger. It goes together with the concern expressed by Hart et al. (ibid.), that research in mathematics education does not only need to know "*if* particular educational experiments improve learning with understanding but also *how* those results are achieved and *why* we can expect them to be replicated elsewhere."

The two examples above and the interviews with the researchers illustrate how MMR in particular can contribute a great deal of both scientific and practical knowledge to the evidence on "what works" (cf. Chatterji 2005): in particular, the MMR project discussed by Kathleen Philipp (cf. Philipp 2012; Philipp and Leuders 2012) exemplified how the use of systemic, contextually-grounded studies in early phases may be followed by more sharpened, analytic experimental studies in later research phases. The MMR project discussed by Prediger and Wessel (2013) and Wessel (2014) underscored that the combined use of more than one research method may help us reveal patterns, develop new constructs and deepen the understanding of relationships between constructs, especially causal relationships. However, as Prediger expressed it in her interview, it is not necessary "to end with a randomized controlled trial, because we want to understand how the things work and what really happens in the processes."

## **9.4 Conclusion**

The four examples of research presented here can by no means claim to be representative for educational research in mathematics in German-speaking countries. However, they do illustrate several of the characteristics and strategies of research often encountered and explicitly discussed within the research community—in many other countries beyond German-speaking Europe.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Chapter 10 Large-Scale Studies in Mathematics Education Research**

## **Kristina Reiss, Andreas Obersteiner, Aiso Heinze, Ursula Itzlinger-Bruneforth and Fou-Lai Lin**

**Abstract** Large-scale studies assess mathematical competence in large samples. They often compare mathematical competence between groups of individuals within or between countries. Although large-scale research is part of empirical educational research more generally, it is also linked to more genuine mathematics education research traditions, because sophisticated methods allow for empirical verifications of theoretical models of mathematical competence, and because results from largescale assessments have influenced mathematics education practices. This chapter provides an overview of large-scale research in mathematics education in German speaking countries over the last decades. After a brief review of historical developments of large-scale assessments in Germany, we focus on the development of competence models in Germany and Austria. At the end of this chapter, we reflect on recent developments and discuss issues of large-scale assessments more generally, including an international perspective.

**Keywords** Large-scale assessment · Mathematical competence · Competence models · PISA study

A. Obersteiner Freiburg University of Education, Freiburg im Breisgau, Germany

F.-L. Lin National Taiwan Normal University, Taipei, Taiwan

K. Reiss (B)

TUM School of Education, Technical University of Munich, Munich, Germany e-mail<EMAIL_ADDRESS>
## **10.1 Introduction**

In recent years, researchers have assessed mathematical competence in large samples using sophisticated statistical methods. Large-scale assessment of mathematical competences requires close collaboration between researchers from mathematics education, statistics, and psychology. Accordingly, the lines of research presented in this chapter are linked to both mathematics education research traditions and, more broadly, large-scale empirical educational research. The chapter is structured into five sections. The following paragraphs provide an introduction to the contents of the five sections.

Popular large-scale studies such as PISA (Programme for International Student Assessment) have received much public attention and have led to a stronger focus on the outcome of school education in general and of mathematics education in particular. In Germany, international large-scale studies did not receive much attention before 1995, when Germany took part in the Third International Mathematics and Science Studies (TIMSS) for the first time. The results showed that German lower and upper secondary school students' mathematical performance did not meet the expectations of teachers, educators, and the general public. German students performed below the international average and showed acceptable results only for routine problems (Baumert et al. 1997). The results of PISA 2000 (Baumert et al. 2001) were again disappointing. Consequences from these studies were intensive debates among educators and stakeholders, and the launch of educational programs to improve mathematics instruction at school. Another consequence was the agreement to use large-scale assessments on a regular basis to monitor the outcome of school education. We elaborate on these developments in Sect. 10.2.

The developments in large-scale empirical educational research are related to mathematics education research because assessing mathematical competence in large samples was accompanied by the development and the empirical verification of theoretical models of mathematical competence. Assessing students' mathematical competences requires models of what mathematical competence should be. Initial models were predominantly based on theoretical and normative considerations but rarely on empirical evidence. Reiss and colleagues (e.g., Reiss et al. 2007a, b; Reiss et al. 2012) developed a model for primary mathematics education that took into account theoretical and normative perspectives and was continuously refined on the basis of empirical evidence. The model suggests five levels of mathematical competence ranging from technical background knowledge and routine procedures to complex mathematical modelling. Section 10.3 of this chapter provides an overview of this model of mathematical competence.

To monitor the outcome of education on a regular basis, new institutions were founded in Germany, such as the Institute for Educational Quality Improvement (IQB, Berlin) and the Center for International Student Assessment (ZIB, Munich). The idea of system monitoring is, however, not specific to Germany. Other countries founded similar institutions and developed similar models of mathematical competence to assess students' competences on a regular basis. In Austria, the Federal Institute for Educational Research, Innovation and Development of the Austrian School System (BIFIE) was founded on a similar basis as institutions in Germany. One core function of the institute was to establish and disseminate knowledge about educational standards in mathematics. These standards were based on a theoretical framework which included a model of mathematical competence with three dimensions (process, content, and complexity). This model is based on the existing mathematics curriculum and represents a new structure for teaching, learning and assessment of mathematics. In Sect. 10.4, we elaborate on this model of mathematical competence and its development in Austria.

In conclusion, large-scale studies allow monitoring the outcome of mathematics education on the system level. The broad data these studies provide have been used to empirically validate theoretical models of mathematical competence and have contributed to a more realistic view on what students are capable of learning at school. However, there are several general issues of large-scale studies such as their purpose and their risks that need to be discussed. At the end of this chapter, in Sect. 10.5, we reflect on recent developments and discuss more general issues of large-scale assessments that reach beyond the traditions in German-Speaking countries.

## **10.2 Large-Scale Assessment: Impact on Mathematics Educational Research and Practice in Germany**

The aim of this section is to provide an overview of the impact of large-scale assessment (LSA) studies on mathematics educational research and practice in Germany. The section is structured into four subsections. The first subsection provides some historical background information about the public discussion of the LSA results from TIMSS 1995 and PISA 2000 in Germany. In the next three subsections, we describe the impact of LSA on mathematics educational research, on mathematics education as a scientific discipline, and on schools and mathematics teachers, respectively.

## *10.2.1 LSA of Mathematical Competence in Germany: How It Started (TIMSS 1995 and PISA 2000)*

The history of (international) LSA of mathematical competence started in Germany in 1995. Until then, Germany did not take part in international comparison studies on outcomes of mathematics education. Moreover, until 2000 there were no comparisons between the 16 federal states in Germany. The latter is particularly interesting because in the 1970s and 1980s there were heated (political) debates within and between the federal states about the most effective school system, the educational goals, and the preferred teaching styles. Although there was disagreement among the federal states, politicians were convinced that Germany in general had a highlevel educational system. Hence, many were disappointed by the results of TIMSS 1995 for lower and upper secondary students in mathematics. Germany scored below the international average and the performance level was only acceptable for routine problems (Baumert et al. 1997). The consequence was a discussion within the educational sciences and educational administrations (Blum and Neubrand 1998) but there was hardly any discussion in the public. The results of TIMSS 1995 were supplemented by the findings of TIMSS Video 1995. This study confirmed findings from earlier case studies (e.g., Bauersfeld 1978) describing a typical teaching style in German mathematics classrooms (Stigler et al. 1999; Neubrand 2002). A feature of this teaching style was a low variation of teaching methods, which often meant direct instruction accompanied by low-level question-answer-sequences (so-called "funnel pattern", Bauersfeld 1978). Moreover, a low quality of tasks and task implementation was reported. As a consequence, the responsible politicians decided in 1997 to participate in international comparisons and to compare educational outcomes of the 16 federal states on a regular basis.

In 2000, Germany took part in the first PISA study. Like the results of TIMSS, the results of PISA were again relatively weak, from both a criterion-oriented and an international comparative perspective (Deutsches PISA-Konsortium 2001). For mathematics, the study identified a large group of low-performing students (about 25%) and only a small group of high-performing students (about 2%). Moreover, there was a strong relation between students' achievement and their socio-economic background, and there were large differences between the 16 German federal states. A specific weakness of German students was the ability to use their mathematical knowledge in real-life situations. In contrast to the TIMSS 1995 results, the PISA results were debated in the broader public. For example, the fact that German students and students from the U.S. were quite similar with respect to the level of mathematics competence and the relation between achievement and socio-economic background was discussed intensively.

## *10.2.2 Impact on (Mathematics) Educational Research in Germany*

The political decision to participate in international LSA provided various opportunities for educational research. Accordingly, the LSA itself as well as the results of the LSA had an impact on (mathematics) educational research in Germany. On the one hand, it influenced descriptive research, and on the other hand, it induced studies aiming at explanations for the descriptive LSA findings.

The motivation for further descriptive research studies relied on the mission of educational research to provide a basis for political decisions to improve the educational system. With TIMSS 1995 and PISA 2000 it became clear that educational policy in Germany was based on insufficient information about the educational reality. Hence, subsequent to the first international LSA on the secondary level, educational research used the opportunities of national extensions of the follow-up cycles as well as of additional LSA in other age groups. For example, in PISA 2003, the German sample was supplemented by an additional sample of students with migration background (so-called oversampling). It turned out that language skills and socio-economic status (SES) were the main factors for the comparatively low mathematics performance of students with migration background. Moreover, the PISA 2003 sample was tested again one year later to collect information about students' learning progress in mathematics. The findings indicated that there was large variation in mathematics competence development between school classes. Only 58% of students showed a significant increase of mathematical competence from grade 9 to grade 10 on the mathematics PISA scale (for more details on how mathematical competence was conceptualized, see Sect. 10.3.1).

Several additional LSA studies were conducted with respect to mathematical competence in other populations than secondary students, such as:

In addition to descriptive research several explanatory research studies and study programs were conducted. In essence, the aim of these projects was to explain the previously obtained descriptive results and to generate knowledge which might help to improve mathematics education in schools. Examples for research programs are the priority programs "Educational Quality of Schools" and "Competence Models", supported by the German Science Foundation (DFG), or the framework program "Empirical Educational Research", supported by the Federal Ministry of Education and Research. Examples of studies that were conducted within these programs are:

## *10.2.3 Impact on Mathematics Education as a Discipline*

The international LSA studies had different kinds of impact on the mathematics education community. As described in the previous section, LSA studies induced further research studies. However, there was and still is an additional impact on a meta-level, especially on the research content and research methods as well as on the discussion about the specific role of mathematics education research in the broader field of educational research.

Regarding the impact on the research content and methods, LSA studies firstly facilitated a normative discussion about the constructs which were addressed in LSA. Among others, there were scientific debates on the conceptualization of mathematical competence, the quality of mathematics instruction, and teacher professional competence. Related to these debates the goals of mathematics education in schools as well as in teacher education were scrutinized.

Secondly, LSA facilitated the "empirical turn" in mathematics education research. As an example, quantitative-empirical research articles in the main German journal on mathematics education (Journal für Mathematik-Didaktik; JMD) published after 2005 appear to be more sensitive to methodological challenges of empirical research than articles published in the 1980s and 1990s. Especially, decisions on adequate research designs, the quality of test items (item development based on competence models) or the quality of processing raw data (reliability and objectivity) have become more important in the scientific discussion. In sum, the mathematics education community now has higher scientific standards concerning quantitativeempirical research.

A third important impact of LSA was and is that it raised questions about the features that constitute mathematics education as a scientific discipline. This question was triggered by the fact that the results on students' achievement in mathematics in TIMSS 1995 and PISA 2000 were mainly presented by researchers from educational science and psychology (although a few researchers from mathematics education were involved in these studies). Accordingly, the question came up what the specific contribution of mathematics educational research is in comparison to that of mathematics-related research in educational science. Interestingly, a similar question concerning the status of mathematics education came up again in the context of research on mathematics teachers' professional knowledge. For the conceptualization of the construct pedagogical content knowledge (PCK) and the development of related test items, researchers had to answer the question what mathematics educational knowledge should be. The challenge of this question became even clearer and was triggered by Lee Shulman's notion that PCK is simply an amalgam of content knowledge and pedagogical knowledge (Shulman 1987). This issue is still a matter of discussion within the educational community (Kirschner et al. 2017).

## *10.2.4 Impact on Schools and Mathematics Teachers*

In addition to the impact of LSA on the scientific field of mathematics education, the LSA studies have consequences for the educational practice in schools and for mathematics teacher education at universities. The most striking consequence of LSA was the nationwide implementation of educational standards for mathematics combined with a monitoring system. In Germany, since 1945 the federal states are in charge of school education. The organization of the educational system in a decentralized manner was a consequence of the Nazi regime in Germany and was meant to avoid that a central government could control the educational system. Although a standing conference of all 16 ministries of education exists, until 2003 the federal states did not agree on precise goals and desired outcomes of school education (except for the German Abitur, which is the final examination from upper secondary schools and a general university-entrance qualification). In the 1990s, first ideas for educational standards in grade 10 were discussed. The results of TIMSS 1995 and PISA 2000 accelerated this process. In particular, the results of PISA 2000 in general and the huge differences in mathematics achievement between the different German federal states in particular put a lot of pressure on politicians. The German constitution (Grundgesetz) requires that the federal government and the governments of the federal states ensure an equal standard of living in all federal states (which means, in particular, equal educational opportunities). Hence, in 2003, educational standards for mathematics for the lower secondary level (grade 9/10) were implemented. In 2004, such standards were implemented for the primary level (grade 4), and in 2012 for the Abitur (upper secondary level, grade 12/13). In 2004, the federal "Institute for Educational Quality Improvement (IQB)" at the Humboldt University of Berlin was founded. The IQB is responsible for the evaluation of the educational standards. In 2006, a nationwide test-based monitoring system was implemented with highquality test instruments. Teachers are obliged to administer these tests in grade 3 and grade 8, that is, one year before students should reach the educational standards for the primary or the secondary level, respectively. One important aspect of the monitoring system is that teachers analyze their students' test results so that they get information about their students' difficulties. Based on this information teachers can prepare specific learning support so that especially low achieving students get a chance to reach the educational standards after grade 4 or 10.

In addition to the above-mentioned changes of the educational system, new ideas were brought into teacher education and teacher professional development. For example, in 1998 the nationwide teachers' professional development (PD) program SINUS1 started (and lasted until 2013). It included primary and secondary school teachers from more than 2500 schools. SINUS provided a specific infrastructure for PD (Prenzel et al. 2009). Teachers from different schools worked in local professional learning communities (PLC). The basis for each PLC was a pool of "state of the art material" prepared by researchers, which described topics from mathematics education. The PLC chose material from this pool and worked on the topics. Moreover, representatives of the PLCs met regularly in regional meetings and regional representatives met regularly in national meetings. The SINUS program for primary school teachers was evaluated based on TIMSS instruments. It turned out that students taught by teachers in so-called SINUS schools showed better achievement in mathematics than students taught in comparable schools without SINUS (Dalehefte et al. 2014).

There were also changes in the university-based teacher education system. Firstly, obligatory standards for mathematics teacher education in Germany were established in 2008. Secondly, the poor results of non-certified mathematics teachers in TEDS and the low mathematics achievement of their students in the evaluation of the educational standards had an important influence on the federal states to change the regulations for teacher education. In most federal states, all primary school teachers must now pass a substantial number of mathematics courses at the university. Moreover, all future teachers have to obtain a Master's degree or a comparable qualification before they are allowed to teach in school, regardless of school level (i.e., primary through upper secondary level).

In addition to the consequences on the institutional level described so far, LSA affected genuine mathematics educational research. We illustrate this impact using research on mathematical competence in the following two sections.

<sup>1</sup>SINUS - "Steigerung der Effizienz des mathematisch-naturwissenschaftlichen Unterrichts" (improving the efficiency of mathematics and science teaching).

## **10.3 Consequences from Large-Scale Assessment: An Example of Research on Mathematical Competence in Germany**

In this section, we first explain why models of mathematical competence are necessary to implement and evaluate standards for school mathematics. We then describe how models of mathematical competence are developed in a process that takes into account both theories of mathematical development and empirical evidence. We illustrate this process using data from a large-scale national assessment in Germany. Finally, we show how additional qualitative analyses can contribute to our understanding of mathematical development on an individual level.

## *10.3.1 Understanding Mathematical Competence*

As described above (Sect. 10.2) standards for school mathematics have been implemented in several countries in the last decades (e.g. Kultusministerkonferenz 2003, 2004, 2012, in Germany; National Council of Teachers of Mathematics 2000, in the U.S.). The influential Common Core State Standards Initiative (2012) asserts that standards "define what students should understand and be able to do" regarding mathematical content like number and quantity, algebra, functions, geometry, statistics and probability, as well as regarding typical mathematical practices like problem solving, reasoning and argumentation, modeling, use of tools, communication, use of structures and regularity. Standard-oriented classroom instruction aims at a profound understanding of mathematics and seeks to support students in applying their knowledge. Accordingly, standards are meant to support students' competence acquisition. This concept plays an important role with regard to standards, and it is also used in the German context. Weinert's (2001, pp. 27–28) definition is broadly accepted; it defines competences as "cognitive abilities and skills possessed by or able to be learned by individuals that enable them to solve particular problems, as well as the motivational, volitional and social readiness and capacity to utilize the solutions successfully and responsibly in variable situations." Standards thus refer to a conceptualization of competence that is similar to the concept of literacy on which the framework of PISA is based.

Students' competences may be assigned to mathematical content and practices but this assignment should take into account that students perform at different levels of proficiency. PISA 2012 provides a general description of these different levels based on international test data (OECD 2013). Table 10.1 summarizes six levels of proficiency of 15-year old students according to the OECD report (OECD 2013, p. 61).

This description of proficiency levels is important in order to understand students' performance particularly in the context of international comparisons. However, this description has limitations from a mathematics education point of view and with

regard to mathematics classroom practices. The reason is that the levels of proficiency are not "fine-grained" and do not specify details of mathematical processes and products. In particular, as this model is not meant to be a developmental model, it does not explain how learners may proceed from one level of proficiency to the next, or specify students' knowledge gaps when they fail to do so. Including such information in the model would be useful for analyzing student errors on specific items and for understanding learning processes (Vygotskij 1978).

Further refining the model would presuppose a better knowledge of learning processes, which would require detailed studies of students' performance. Moreover, profound theoretical considerations are necessary. It is important to identify the prerequisites for solving a specific problem with regard to the requirements concerning the mathematics behind the problem and with regard to general or everyday knowledge: For example, calculating 37 + 28 asks for basic numerical knowledge, and probably the application of various methods of calculation which presupposes knowing and understanding them. Determining the number of legs, which five dogs, eight birds, or seven goldfish have, on the other hand, requires basic numerical knowledge but also "every-day knowledge". Understanding mathematical competence must therefore take into account alternating empirical and theoretical considerations in order to properly describe competences and their development.

## *10.3.2 Modeling Mathematical Competence*

Attempts have been made to describe mathematical competence and its development particularly from a theoretical point of view resulting in (mainly) normative models of mathematical competence (Reiss et al. 2007b). Evaluating these models has two aspects. On the one hand, we need empirical studies with an adequate number of tasks and students in order to verify or falsify the theoretical considerations. On the other hand, we need empirical studies, which demonstrate the distinction between levels of competence with regard to typical misunderstandings in order to initiate support in teaching these topics.

In the following, we illustrate how these aspects were addressed in a recent research agenda. First, we present data from a national assessment ("IQB-Ländervergleich") of a representative sample of 27,000 fourth-graders in Germany (Reiss et al. 2012), which aimed at monitoring the German educational standards in mathematics. Moreover, we describe how data may be used to inform teachers about strengths and weaknesses of their students.

## **<IP_ADDRESS> Modeling Mathematical Competence with Data from a National Assessment**

The test assessed mathematical competence in a variety of subdomains. Data were scaled such that the mean was 500 and the standard deviation 100. The breadth of the levels was set to 70; however, with respect to low-performers and high-performers boundaries were open. In the following, we illustrate the model in the domain of numbers.

• Level I (≤389): Basic technical background (routine procedures on the basis of simple conceptual knowledge)

Students at this level are familiar with simple mathematical terms and procedures. These terms and procedures can be reproduced correctly within a mathematical context, respectively within a context, which is familiar or well-trained. Specifically, students are proficient in exercises that require addition or multiplication of numbers up to ten. They can utilize these concepts during mental calculation exercises, partly written or written-only calculation exercises, if the exercises do not pose any specific difficulties. Additionally, they have to be able to apply these types of calculations in simple word problems correctly. Also, students can compare numbers based on their values and they can interpret numbers on place value panels without problems, especially in the range 1–1000.

• Level II (390–459): Simple use of basic knowledge (routine procedures within a clearly defined context)

Students are able to utilize their basic knowledge for simple, clearly structured and well-known mathematical problems. For example, problems requiring addition, subtraction, and multiplication can be solved using written or partly written algorithms. The students also conduct rough estimations and correctly recognize the dimensions of their results. In addition, students utilize the structure of the decimal system, and recognize general mathematical principles. These principles are taken into account when continuing simple number sequences, conducting structured counting procedures and engaging in systematic trials. They detect and apply familiar proportional attributes. Students can convert simple numbers into specified units, even when the units differ in value or when numbers have decimals. In addition, students can interpret clearly structured graphs, figures and tables even if they entail a large number of information.

• Level III (460–529 points): Recognition and utilization of relationships within a familiar context (both mathematical and factual)

Students are able to utilize their knowledge flexibly in various problems within a familiar context. Specifically, students handle numbers and operations within the curricular scope securely, and conduct numerical estimations well—even for large numbers. Students can recognize and describe structural aspects—at least when the contents have been practiced well. Students are able to continue number sequences that follow relatively complex rules; in addition, they identify incorrect numbers within straightforward number sequences.

• Level IV (530–599): Secure and flexible utilization of conceptual knowledge and procedures within the curricular scope

Students can utilize their mathematical knowledge securely, even when the context is not familiar. They correctly describe their own methods for calculation, understand and reflect on the approach of other students and are proficient in all mathematical calculations that are part of the curriculum. Following instructions, students manipulate and systematically change numbers in place value panels, even if the numbers are large (up to one million). They recognize the rules behind even complex number patterns and continue the patterns correctly. Calculations with quantities are performed securely and flexibly, especially calculations that involve approximations and estimations.

• Level V (≥600): Modeling complex problems involving independent development of adequate strategies

Students can work on mathematical problems in each subject area adequately, securely and flexibly even if the context is unknown. Students are able to work on a highly advanced level both in using adequate strategies and in giving meaningful evaluations and generalizations. They use their in-depth knowledge from the curriculum flexibly even in unfamiliar situations, communicate their methods comprehensibly and explain why they chose their specific method. Mathematical arguments are assessed adequately, and complex situations are modeled and worked upon even if calculations require difficult processes such as the use of tables, compound quantities or numbers in decimal notation.

## **<IP_ADDRESS> Modeling Competence Between Quantitative and Qualitative Approaches**

Models of competence are based on well-defined demands; they identify mathematical tasks with varying difficulty and match students' competences and task characteristics. There is a straightforward scientific paradigm, the correct or wrong answer. However, such a paradigm has limitations because the correctness of answers does not allow immediate conclusions about the cognitive processes underlying these answers. In the following, we describe a study (for further details, see Obersteiner et al. 2015) wherein we tried to analyze students' answers in depth in order to better understand the differences between the various levels of competence with respect to errors and misconceptions. Data come from a sample of 3rd-graders who took part in an annual testing of their mathematical competences ("Vergleichsarbeiten"; VERA). The items addressed mathematical argumentation in the domain of number and operations as well as in the domain of patterns and structures.

Item 1 (see Fig. 10.1) addresses place values. The theoretical level of competence of this item according to the model was II, as was the empirical level. More than half of the students gave correct answers, 23% did not answer to the question. The most common mistake was referring to the ones only or to the tens only. These children obviously had a preliminary understanding of the place value system but were unable to extend it to this complex situation.

#### **Fig. 10.1** Item 1: place value

Item 2 (see Fig. 10.2) addresses properties of numbers. In this case, the theoretical level of competence (III) was lower than the empirical level (IV), which means that the students had more problems with solving it correctly than was anticipated. About one third of the students gave correct answers, another third did not answer to the question. The most common mistake was not referring to the magnitude. Children were not able to use their knowledge of numbers in a problem-solving situation.

Item 3 (see Fig. 10.3) addresses number sequences. For this item, the theoretical and empirical levels of competence were identical and both high (IV for part a, V for part b). Only 27% (part a) and 14% (part b) of the students, respectively, answered the two questions correctly. The most common mistake was that students did not mention that the magnitudes of each two numbers add up to 100. Children's written work suggested that some were able to identify the correct solution but were unable to communicate their knowledge.

**Fig. 10.2** Item 2: numbers pairs

## *10.3.3 Summary*

We have illustrated how international large-scale assessments and the introduction of educational standards for school mathematics had an impact on mathematics education research and practice in Germany. Competence models have been developed that were based on both theoretical considerations and empirical evidence from largescale assessments. To increase our understanding of how we can support students develop from a lower level of competence to a higher one, additional qualitative approaches that include an analysis of where exactly students go wrong at a certain level of competence can be fruitful. Such approaches can eventually lead to more fine-grained models of mathematical competence and its development.

## **10.4 Consequences from Large-Scale Assessment: Competence Models and Evaluation of Classroom Practice in Austria**

The current system of LSA in Austria comprises three strands: international assessment as external benchmark, standardized assessment of national education standards (BIST-Ü) as a main lever for school quality development, and informal competence measurement (IKM) as a self-evaluation tool for teachers. This section first provides a brief history of the system and then focuses on the two strands of national assessments. We introduce the competence models and the national education standards, describe briefly the implementation of the assessment, and then explain the reporting system.

## *10.4.1 Development of Large-Scale Assessment in Austria*

As in other German speaking countries, LSA are a relatively young phenomenon in Austria. By participating in TIMSS in 1995, Austria, like Germany (see Sect. 10.2), started to participate in international LSAs. The results were mixed: Austria was in the group of top performers on the elementary school level but not on the lower secondary school level. Austria had a very small group of top performers, and there was a comparably large gender gap (Beaton et al. 1997; Mullis et al. 1997). However, the results were not of much interest to policy makers or the broader public. This changed in 2001, when the results from PISA 2000 were presented and publicly discussed. Of most concern was (and is) the group of students identified as "students at risk". In mathematics, 22% of students in Austria currently fall into this category (Suchán and Breit 2016). Moreover, there is a comparably small group of high performing students.

In response to these results, the Ministry of Education of Austria assigned a committee to devise a "master plan". One of the goals of the committee was to change the culture in the school system from input-oriented to output-oriented. In the report (Eder et al. 2005), the committee listed a number of measures intended to augment school quality, amongst others the introduction of national educational standards and the assessment thereof. At about the same time, competence models for the subject domains were developed and empirically validated in pilot studies. In 2009, a law was passed (Bundesgesetzblatt II Nr. 1/2009) that defined national educational standards in mathematics, German (both grades 4 and 8), and English (grade 8 only), which were based on national curricula. The law also mandated regular assessment of the competences and "meaningful feedback" to students, teachers, and school principals as well as a system of monitoring reports in order to foster evidencedriven activities. A baseline study was conducted in 2009 (lower secondary school level, n - 204 schools) and 2010 (elementary school level, n - 267 schools) in order to measure the level of competences before national education standards were implemented. Since 2012, competences in mathematics, German, and English are assessed, alternating every year in a census in grades 4 and 8. Feedback is prepared by the BIFIE on several levels for each national LSA (see Table 10.2).

## *10.4.2 Mathematics Competence Models in Austria*

For all subjects and grades covered, competence models were devised. In 2007, the Institute for Didactics of Mathematics at the University of Klagenfurt (Austria) pub-

**Table 10.2** Feedback recipients

lished a brochure in which a competence model for mathematics in lower secondary schools was introduced (Institut für Didaktik der Mathematik 2007). In this section, we only describe the model used for grade 8 mathematics. The competence model uses three dimensions to describe mathematical competence: process, content, and level of complexity (see Table 10.3).

The process domains as well as the content domains contain four distinct areas each; the level of complexity is divided into three hierarchical levels. Since these dimensions of content, processes, and complexity are orthogonal to each other (see Fig. 10.4), the model consists of 4 × 4 × 3 - 48 nodes altogether. For every node, a can-do statement describes students' skills and abilities (Verordnung der Bundesministerin für Unterricht, Kunst und Kultur über Bildungsstandards im Schulwesen 2009), for example "the students are able to describe algebraic, tabular or graphic representations of structures and (functional) relations and interpret both in a given context. In doing so, it is necessary to make connections to other mathematical contents (terms, theorems, representations) or processes" (process 3/content 2/complexity level 2, see grey cube in Fig. 10.4).

Mathematics materials used in school are supposed to use this structure and cover all process and content domains. Test items for national tests are developed on the basis of these nodes; only items that can be classified unambiguously on a node are used in the test, in the sense that the group of test developers and at least one external expert reach consensus on the classification. Since the level of complexity is not equal to empirical difficulty, at reporting stage, empirical difficulty along with the classification according to educational standards (competence level) are also published. As an example, the item in Fig. 10.5 has been classified as follows.

**Table 10.3** Dimensions of mathematical competence

**Fig. 10.4** Competence model, mathematics, grade 8. Adapted from Institut für Didaktik der Mathematik (2007)

**Fig. 10.5** Example item

• Answer: 16 km. Empirical difficulty: 53% of students in Austria solved this item (BIFIE 2012a).

An expert group consisting of mathematics teachers and mathematics education researchers described in four levels the degree to which the education standards are reached (see Table 10.4).

In contrast to the descriptions used in international assessments, these levels were defined and described using a Standard Setting method called Item-Descriptor-Matching method (Freunberger 2013). The method requires comparison of the performance shown on the test items with the theoretical framework (in this case, the national curriculum and national standards). The exercise was undertaken before the main assessment was carried out, using field trial data to sort items by empirical difficulty. As the first national assessment in mathematics was strongly debated, this was necessary to keep the Standard Setting out of political discussion. The performance level descriptors were centered around level 2, "standards reached", as this represents the desired outcome. The competence model and the performance level

#### **Table 10.4** Levels of competence, mathematics, grade 8

#### **Level 3—standards exceeded**

Students possess fundamental knowledge and skills in all parts of the mathematics curriculum and advanced knowledge structures, which exceed the requirements of level 2, specifically more pronounced abilities of abstraction and higher proficiency in combining parts of knowledge, methods or rules. They are able to apply these independently in novel situations in a flexible way

#### **Level 2—standards reached**

Students possess fundamental knowledge and skills in all parts of the mathematics curriculum and can use these in a flexible way. They are able to find and apply problem solving strategies, to describe and reason about their approach. They are able to handle verbal, graphical and formal representations of mathematical facts in a flexible way and can apply these appropriately. They are able to extract relevant information from differently represented facts (e.g. texts, data material, graphics) and can interpret them in the respective context. They are able to relate their mathematical knowledge and can check, evaluate and/or reason about mathematical statements

#### **Level 1—standards partially reached**

Students possess fundamental knowledge and skills in all parts of the mathematics curriculum and can master reproductive tasks and carry out routine procedures

#### **Below level 1—standards not reached**

*There is no further description of this level, as students on this level typically show insufficient competences/skills*

descriptors represent the core of the feedback to all stake-holders and can be found in all reports (see Table 10.2).

## *10.4.3 Reporting of Assessment of National Education Standards*

#### **<IP_ADDRESS> Reporting on the System Level**

System monitoring is one of the main goals of the assessment of national education standards. A detailed report is published every year on a national and a regional level (cf. Schreiner and Breit 2012a, b; Schreiner et al. 2018). The outcomes should inform education policy makers about issues and areas for improvement. Since the assessment has only started the second cycle, it is yet too early to identify sustainable development. However, we can state that students have performed better in mathematics in 2017 (national average 542 points) than in 2012 (535 points) and in the baseline assessment in 2009 (500 points). This increase in performance was highest in the content domain of statistics (550 points vs. 544 points vs. 500 points). In part, this might have to do with instructional practices; teachers can choose not to cover all areas of the curriculum. Presumably, statistics had been covered by teachers less often than other areas prior to the assessments: A study among first-year students of mathematics education showed that statistics is also the most unpopular content area, and that students complained about "comprehension difficulties", "bad lessons" and "no relation to everyday life" (Süss-Stepancik and George 2016). This probably also affects teaching (Süss-Stepancik and George 2017). In the years preceding more recent assessments, the (dis-)likes for content areas and process domains could have been evened out by the fact that all areas are covered by the national tests because teachers might say "now that you test it, we teach it". Teachers may also be more aware of the assessments because instructional material now contains more references to national education standards and competence models than in 2009.

#### **<IP_ADDRESS> Reporting on the School and Class Level**

The feedback to school principals and teachers is designed to foster school development and improvement of instructional practice. Based on a model of factors influencing development of school quality and teaching strategies (Helmke 2004), Wiesner et al. (2015) devised a framework modeling the use of feedback for schools and teachers for development. A central point is the standardized assessment which gives objective feedback, in how far schools and classes are able to convey the competences described in the national education standards. The first step comprises the correct analysis and interpretation of the results in order to identify weaknesses and potentials for improvement. Reflecting the outcomes, teachers and school heads then can design actions for improvement. In an evaluation step, the impact of the implemented actions is scrutinized and the quality development cycle starts again. In order to support analysis and interpretation of results, individualized feedback for each class taught is created for to teachers using standard templates (see Table 10.2). This feedback describes not only the outcomes but also other variables like students' socio-economic background, their wellbeing at school, and their motivational situation. Moreover, it includes a "fair comparison" (Pham and Robitzsch 2014), that is, a comparison with schools/classes showing similar characteristics. In addition to feedback on levels of mathematics competence and feedback in terms of points reached overall as well as feedback on the results of subgroups in the class, teachers get feedback on the results of their class(es) in the process and content domains. In contrast to conventions in many other countries, school or class results are not reported publicly as such reporting is not regarded as helpful in terms of quality development. In order to give schools space to develop and work with the results, no school ranking is published. Aggregated reports on regional and national level are published online without reference to specific schools. School results are reported to schools and school authorities only.

As an example, Fig. 10.6 shows the feedback on the process domains (note: data are fictitious). It contains the Austrian mean for all domains (see flag) as well as the school mean (see punctuated line) and the class mean (dot with confidence interval). Individual students are represented as dots. The higher up the bar, the higher the achievement; the wider spread the dots are, the larger the variance within the class. The table below the chart lists the number of students in school and class and the mean including the confidence interval for each domain.

**Fig. 10.6** Results on class (group) level by process domain (BIFIE 2012b)

#### **<IP_ADDRESS> Other Report Levels**

School authorities receive school results only, without details on the classroom level. In many cases, this level of granularity is acceptable as in Austria the student population within schools tends to be more homogenous than between schools. Therefore, classes within the same school tend to perform more consistently than classes in different schools. By law, students are entitled to their individual feedback. During the test session, they receive a unique code for their individualized feedback available via a secure online platform. Although students are the biggest recipient group, the impact of the feedback on them is probably quite negligible as the results must not be used for grading. The student data is considered confidential and no-one but the student should have access to it.

## *10.4.4 Informal Competence Measurement*

In contrast to the assessment of national education standards, the informal competence measurement is focusing on the classroom level only (Wiesner et al 2017). Teachers can decide whether or not to use this free, web-based tool in their classes. The online platform can score selected response items automatically. Teachers only need to code open response items themselves. The platform instantly generates feedback on the class level and on the level of individual students. It provides feedback for mathematics in general and for process and content domains separately. For each item, national benchmarks stemming from pilot tests are provided. Since the use of the tool is voluntary, the exposure is somewhat hard to measure; roughly 15% of students take it in grade 6 and about 25% of students in grade 7 (in mathematics). Teachers may choose to share the results but are not required to.

**Table 10.5** Comparison of mandatory national assessment and informal competence measurement

(continued)

**Table 10.5** (continued)

The competence models used in this tool are the same as the models that are used for the national assessment. The comparison in Table 10.5 is restricted to lower secondary school.

## *10.4.5 Summary*

The different strands of large-scale assessment in Austria allow for different viewpoints on students' mathematics competence, depending on the main uses and users. The international LSAs show how students in a specific country perform compared to other countries' students. This is useful feedback for policy makers and interesting for the public. The national assessments allow for comparable and meaningful feedback to teachers and school principals as well as students and parents, school authorities and the broader public about the levels of competence reached as well as areas where there is room for improvement. Figure 10.7 summarizes the intended interactions between curriculum, competency models, national education standards, assessments, and feedback.

As illustrated in this figure, there are many channels for feedback to relevant actors and stakeholders in the system. Further evaluation of the impact of national education standards and their standardized and mandatory assessment will hopefully further clarify the role of education standards.

**Fig. 10.7** Interactions and embedding of large-scale assessments in Austria

## **10.5 Summary and Discussion**

Large-scale assessment plays an increasingly important role on various levels of educational research and practice in German-speaking countries. The developments in Germany and Austria described in this chapter show the potential of LSA in several respects. Large-scale studies have an impact on curriculum development, policy making, and the development of theories of mathematical competence. They also can inspire further research to address issues that large-scale studies have identified as being relevant for competence development. For example, the discussion about what mathematical competence should be has led to the agreement that competence is more than proficiency and encompasses cognitive but also non-cognitive aspects, such as motivation and interest (Weinert 2001). Also, it would be useful if competence models provided information not only on students' competence but also on what students did not yet understand. Such information would be particularly valuable for teachers who need information on students' strengths and weaknesses in order to support their learning. Thus, future research could focus on these aspects.

However, large-scale assessments also have limitations. In the remainder of this section, we first reflect on general issues of large-scale assessments, and then discuss specific limitations and potential risks for research and practice.

## *10.5.1 General Issues of Large-Scale Assessments from an International Perspective*

We discuss general issues of large-scale assessments from three perspectives: their purposes, their processes, and their outcomes.

#### **<IP_ADDRESS> Purposes: The Studies Reflect the Needs of a Country**

The decision to conduct large-scale studies is often accompanied by the intention to solve specific educational problems in a country. In the case of Germany and Austria, for example, the starting point of large-scale studies was mainly influenced by the international assessments of students' mathematics performances (i.e., TIMSS and PISA). The results of those studies were disappointing and received much public attention, which in turn pushed stakeholders to improve the situation. Empirical studies provided information about the situation in classrooms. In other words, they enabled investigating certain phenomena more closely to eventually improve the situation for teachers and students, and for developing the curriculum.

In Germany, to support primary and secondary school teachers' professional development, the country started a nationwide teacher professional development program (SINUS). Moreover, there were studies investigating teachers' competencies in teaching mathematics from a national perspective (e.g., COACTIV), as well as from an international perspective (e.g., TEDS-M). In a next step, studies investigated not only school students' and teachers' mathematical competences but also those of adults (e.g., PIAAC). In order to trace the development of competence, the NEPS project was started in 2008. It includes participants of a wide age range to assess their mathematics competence both cross-sectionally and longitudinally.

#### **<IP_ADDRESS> Processes: Revealing the Functions of Studies**

To discuss issues related to the aspect of processes, we make an analogy to the TIMSS framework (see Mullis and Martin 2013), which describes how OECD adapts the processes of knowing, applying, and reasoning in designing test items and assessing what students learned in mathematics. We consider these three processes suitable for interpreting how the countries view large-scale studies through the process of (1) knowing what phenomenon they want to investigate, and (2) applying what they have known from precedent studies to explore the new domains they want to investigate further (i.e., a new round of knowing process), or to solve the present educational problems they met. In order to process the application, (3) re-analyzing the collected data with new methods creates opportunities for new studies exploring additional phenomena or outcomes.

The primary data collected from large-scale assessments provide abundant and useful information. The information from these primary data usually contributes directly to educational fields, such as policy making and comparison within a country or between countries. The other potential method to deal with the abundant data is secondary analysis (Glass 1976). Glass introduces the two purposes of this method, one is the re-analysis of data for the purpose of answering the original research question with better statistical techniques, and the other is to answer new questions with old data. An example is the secondary analysis of TIMSS data (Robitaille and Beaton 2002).

In the example of research on mathematical competence in Germany, the purpose was for example the assessment of fourth-graders' mathematical competence, aiming at developing a competence model. This model is related to competence levels of students' performances in PISA 2012 (OECD 2013). The primary data were collected from the survey of nationwide students with validated items. After analyzing those primary data, the competence model was constructed for specific mathematical content areas. In a next step, students' written responses were investigated to better understand students' errors and misconceptions (see Sect. 10.3).

In Austria, the process was similar to Germany, but the example of Austria shows how competence models can be useful to provide feedback on different levels of the educational system. Data with different ecologies were used to identify new educational phenomena in different schools or classes (King 1997). In this way, large-scale studies can inform policy-makers to make realistic decisions based on empirical data.

#### **<IP_ADDRESS> Outcomes of Large-Scale Studies**

As already mentioned, the purpose of large-scale studies is often to solve specific educational problems. The examples from Germany and Austria reported in this chapter showed how the development of competence models benefited from students' performance data from large-scale assessments. Such competence models are dynamic regarding their contents and their participants in related assessments. Moreover, the Austrian study further applied the survey of competence into the system monitoring and suggestions to instructional improvement as the intended outcomes. The example of Germany illustrated how data can be used for another outcome, namely secondary analyses of students' answers to better understand their reasoning and eventually inform instruction concretely and constructively.

## *10.5.2 Potential Risks for Research and Practice*

There are potential risks of LSA for research and practice. Although LSA induced progress and changes in mathematics educational research as well as in mathematics classroom, there are some aspects which can be considered as critical.

A first aspect is that LSAs provide a huge amount of empirical data on students' and teachers' mathematics achievement. This can create an imbalance between empirical and theoretical perspectives because it can be tempting to mix up empirical data with empirical evidence. Hence, it is important to keep in mind that generating and interpreting empirical data requires theories, and that theories are also necessary to transform empirical data into empirical evidence. This is particularly important in cases when LSA data develop a "life on their own". Examples are the increasing amounts of purely descriptive empirical results with little scientific impact (i.e. low explanatory power) or the illusion of "exact" results due to empirical data-bases and over-interpretation of statistical results.

A second aspect is the norm setting. For example,

In conclusion, the research community should critically analyze which role LSA already has and which role LSA should have for research and practice in mathematics education.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.


*2.5. Temperature-Color Cross-Modality*

Regarding cross-modality between temperature and color, works like [5,20] showed that there is a color-temperature association with the color warm–cold spectrum. In [20], the authors designed and performed a method for conveying color information both through thermal intensity and thermal change rate. The method was tested with ten users and showed successful results. In [5], a method for discerning up to 50 colors by feeling three different temperatures was designed. It included extensive interviews and user tests with visually impaired users. Both works proved that temperature cues as a way of expressing color can be a successful interaction method and that visually impaired users can differentiate temperatures quite comfortably as long as there is, at least, a difference of 3 ◦C between them. This fact will be applied in our multi-sensory method for expressing colors.