In Study 1, we compared CAT based on a robust item pool with CAT based on true item parameters and item parameters with uncertainty in them. Average test information functions are shown in Fig. 15.4.

The items in the robust item pool have been corrected for possible overestimation of the parameters. The resulting average information for CATs based on the robust item pool is lower than the information provided by CATs based on an item pool with uncertainty. In the middle of the ability distribution, the difference is only 2%, but towards the tails it is close to 10%. CATs were also simulated based on item parameters that were not disturbed by uncertainty. For these items it holds that they really provide most of their information when the theta estimated equals the difficulty parameter. Towards the tails of the distribution, there was quite a difference in average test information function. In the middle of the distribution, CATs are almost as or even more informative than CAT based on the disturbed item parameters. Root mean squared errors (RMSEs) for the various ability values are shown in Fig. 15.5.

Standard CAT, where uncertainty is not taken into account, resulted in an RMSE that is 6–17% higher than a CAT using the same item pool, but now with the item parameters assuming their true values. Thus, the efficiency of Standard CAT was

theta

**Table 15.1** Resulting average RMSEs for various methods for dealing with uncertainty for various test lengths (n)

overestimated by the same 6–17%. CAT based on robust item pools performed much better. RMSE was 5–9% higher than in simulations with perfectly known parameters, thus it overestimated the efficiency by 5–9%.

The second study focused on the method of robust CAT. In Fig. 15.6, the RMSE of the ability estimates is shown for various ability levels and various settings of . The results for = 25% (solid line) and = 50% (large dashes) cannot be distinguished. For = 75% (small dashes) the RMSE is slightly higher for abilities close to θ = 0. For = 100% (dash/dotted line), the RMSE is slightly higher for all ability levels. Overall, the differences in RMSE are very small.

The third study compared various methods for dealing with uncertainty in CAT. Impact of uncertainty was studied for various test lengths. Average RMSE was calculated over all ability values.

In Table 15.1, the results of various methods for dealing with uncertainty are shown for various test lengths.

Overall, it can be noticed that longer tests provide more information, and the differences between various methods in RMSE become smaller. Standard CAT resulted in the smallest RMSE. Robust CAT performed only slightly worse. Robust item pools performed almost comparable to robust CAT. Both methods based on exposure control performed slightly worse. As expected, the combination of robust item pools and exposure control performed even worse than the exposure control method. Finally, Soyster's method, which is very conservative by nature, performed the worst. Some small deviances of this general pattern were noted, but this might be due to the relatively small sample size in this study.

### **15.6 Conclusion**

In this chapter, the outline of a procedure for robust CAT was presented as an answer to the problem over capitalization on uncertainty in the item parameters in CAT. In this method, a robust item pool based on expected Fisher information and the robust item selection method of Bertsimas and Sim (2003) are combined. First, it was demonstrated how robust item pools can be used in CAT. In a large simulation study, it was illustrated that robust item pools can be implemented successfully, and that the resulting CATs are much closer to the real values than standard CAT that does not take uncertainty in the item parameters into account. Figure 15.6 illustrates how various implementations of robust CAT provide different results. = 100% of the items is equivalent to selecting all the items from the robust item pool, where the other values of only select a percentage of the items from this pool. The impact of on the RMSE turned out to be small, but for ≤ 50% of the items, the best results were obtained. An explanation for the small impact of Robust CAT might be found in the construction of the robust item pool and the nature of CAT. In the robust item pool, expected information is calculated based on the assumption of a normally distributed estimation error. Large adaptations of the provided information are only made for small number of items. As was illustrated in Fig. 15.3, differences between Fisher information and robust item information are only small for most of the items. On top of that, only a few items will be selected per candidate where the robust item information is really much smaller than Fisher information due to adaptive item selection. Larger differences might be found in case of larger estimation errors in the item pool.

The method of Robust CAT was also compared with other methods for dealing with uncertainty in the item parameters in CAT. Robust CAT generally provided the smallest RMSEs. Only applying robust item pools, performed almost as well. Besides, the exposure control method did not perform that much worse. More conservative methods like the combination of a robust item pool with exposure control and Soyster's method had larger RMSEs. It should be remarked however, that the differences are relatively small.

All of these results were based on averages over large numbers of replications. It might be interesting to see what happens at the individual level. The Robust CAT method was developed to prevent overestimation of the precision at the individual level as well. The exposure control method, on the other hand, does not take overestimation at the individual level into account. For example, when the maximum exposure rate of the items is set equal to *rmax* = 0.2, this implies that items with overestimated discrimination parameters will still be used for 20% of the candidates. Especially for small CATs with test length smaller than 20 items, the impact might be considerable. Further research will be needed to reveal for which percentage of the candidates is affected.

Finally, it needs to be mentioned that Belov and Armstrong (2005) proposed using an MCMC method for test assembly that imposes upper and lower bounds on the amount of information in the test. Since there is no maximization step in their approach, item selection is not affected by the capitalization on chance problem. On the other hand, this approach does not take uncertainty in the item parameters into account at all. This could lead to infeasibility problems (Huitzing et al. 2005), as illustrated in Veldkamp (2013). Besides, MCMC test assembly was developed for the assembly of linear test forms, and therefore application to CAT is not straightforward.

### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Chapter 16 On-the-Fly Calibration in Computerized Adaptive Testing**

**Angela Verschoor, Stéphanie Berger, Urs Moser and Frans Kleintjes**

**Abstract** Research on Computerized Adaptive Testing (CAT) has been rooted in a long tradition. Yet, current operational requirements for CATs make the production a relatively expensive and time consuming process. Item pools need a large number of items, each calibrated with a large degree of accuracy. Using on-the-fly calibration might be the answer to reduce the operational demands for the production of a CAT. As calibration is to take place in real time, a fast and simple calibration method is needed. Three methods will be considered: Elo chess ratings, Joint Maximum Likelihood (JML), and Marginal Maximum Likelihood (MML). MML is the most time consuming method, but the only known method to give unbiased parameter estimates when calibrating CAT data. JML gives biased estimates although it is faster than MML, while the updating of Elo ratings is known to be even less time consuming. Although JML would meet operational requirements for running a CAT regarding computational performance, its bias and its inability to estimate parameters for perfect or zero scores makes it unsuitable as a strategy for on-the-fly calibration. In this chapter, we propose a combination of Elo rating and JML as a strategy that meets the operational requirements for running a CAT. The Elo rating is used in the very beginning of the administration to ensure that the JML estimation procedure is converging for all answer patterns. Modelling the bias in JML with the help of a relatively small, but representative set of calibrated items is proposed to eliminate the bias.

F. Kleintjes e-mail<EMAIL_ADDRESS>
S. Berger · U. Moser Institute for Educational Evaluation, Associated Institute of the University of Zurich, Wilfriedstrasse 15, 8032 Zürich, Switzerland e-mail<EMAIL_ADDRESS>
U. Moser e-mail<EMAIL_ADDRESS>
© The Author(s) 2019 B. P. Veldkamp and C. Sluijter (eds.), *Theoretical and Practical Advances in Computer-based Educational Measurement*, Methodology of Educational Measurement and Assessment, https://doi.org/10.1007/978-3-030-18480-3\_16 307

A. Verschoor (B) · F. Kleintjes Cito, P.O. Box 1034, 6801 MG Arnhem, Netherlands e-mail<EMAIL_ADDRESS>
### **16.1 Introduction**

In recent years a comprehensive body of research on computerized adaptive testing (CAT) has been performed. Kingsbury and Zara (1989) refined item selection with a method for content control. Sympson and Hetter (1985) dealt with overexposure, while Revuelta and Ponsoda (1998) treated the subject of underexposure. Eggen and Verschoor (2006) proposed a method for difficulty control.

Still, the process to develop an operational CAT is an expensive and time consuming task. Many items need to be developed and pretested with relatively high accuracy. Data collection from a large sample of test takers is a complex task, while it is crucial to be able to calibrate these test items efficiently and economically (Stout et al. 2003). Item calibration is especially challenging when developing a CAT targeted to multiple school grades (Tomasik et al. 2018), or when the target population is small and participation low. Several rounds of pretesting may be necessary in those cases to arrive at accurate estimates of item characteristics. Unfortunately, a threat of pretesting is that motivational effects can cause disturbances in the estimation process (e.g., Mittelhaëuser et al. 2015).

There are enough reasons to look for alternative procedures that can be employed to diminish the burden of pretesting for use in digital testing, especially in CAT. In this chapter, strategies for on-line calibration that can replace pretesting entirely, or at least to a certain extent, are evaluated. Ideas of extending a calibrated item pool are not new. They range from extending an existing item pool with a limited number of new items – replenishment – to periodic calibration of all items and persons in real time: *on-the-fly calibration*. Until recently the latter possibilities were limited by computational potential and infrastructure. But in view of the increased demand for CAT, on-the-fly calibration is drawing attention.

### *16.1.1 Replenishment Strategies and On-the-Fly Calibration*

Replenishment strategies were developed in order to extend existing item pools or replace outdated items. Thus, a precondition for replenishment strategies is that a majority of items have been previously calibrated. Only a relatively small portion of items can be pretested. Test takers will usually take both operational and pretest items, carefully merged, so that they cannot distinguish between the two types. The responses on the pretest items often are not used to estimate the ability of the test takers. This procedure is referred to as seeding. Usually, but not necessarily, responses for the seeding items are collected and only when a certain minimum number of observations has been reached, the item pool will be calibrated off-line. In on-line methods, the seeding items are calibrated regularly in order to optimize the assignment of the seeding items to the test takers, using provisional parameter estimates of those seeding items.

Stocking (1988) was the first to investigate on-line calibration methods, computing a maximum likelihood estimate of a test taker's ability using the item responses and parameter estimates of the operational items. Wainer and Mislevy (1990) described an on-line MML estimation procedure with one EM iteration (OEM) for calibrating seeding items. All these methods can be applied to replenish the item pool, that is, to add a relatively small set of new items to the pool in order to expand the pool or to replace outdated items.

Makransky (2009) took a different approach and started from the assumption that no operational items are available yet. He investigated strategies to transition from a phase in which all items are selected randomly to phases in which items are selected according to CAT item selection algorithms. Fink et al. (2018) took this approach a step further by postponing the calibration until the end of an assessment period and delaying reporting ability estimates, so that the method is suitable for high stakes testing for specialist target populations like university examinations.

Usually, however, an operational CAT project is not started in isolation. Frequently, several items are being reused from other assessments or reporting must take place in relation to an existing scale. In those circumstances, data from these calibrated items and scales may be used as a starting point to collect data for the CAT under consideration. Using this approach it is interesting to investigate how many and which type of calibrated items are required to serve as reference items. Furthermore, we assume a situation in which instant parameter updating and reporting is a crucial element, and thus the time needed for calibration should be kept at a pre-specified minimum.

### *16.1.2 On-the-Fly Calibration Methods*

In this chapter we concentrate on the 1PL or Rasch model (Rasch 1960) for which the probability of giving a correct answer by a test taker with ability parameter θ on an item with difficulty parameter β is given by

$$P(X=1|\theta) = \frac{e^{(\theta-\beta)}}{(1+e^{(\theta-\beta)})}.\tag{16.1}$$

For item calibration, we consider three methods:

#### **<IP_ADDRESS> Elo Rating**

According to the Elo rating system, player*A*with rating *rA* has an expected probability of

$$E\_{AB} = \frac{10^{(r\_A - r\_B)/400}}{1 + 10^{(r\_A - r\_B)/400}} \tag{16.2}$$

of winning a chess game against player *B* with rating *rB*. After the game has been played, the ratings of the players are updated according to

$$
\dot{r\_A'} = r\_A + K(S - E\_{AB}) \tag{16.3}
$$

and

$$
\dot{r\_B} = r\_B - K(S - E\_{AB}),
\tag{16.4}
$$

where *S* is the observed outcome of the game and *K* a scaling factor. From Equations (16.2)–(16.4) it can be seen that Elo updates can be regarded as a calibration under the Rasch model, albeit not one based on maximization of the likelihood function. Several variations exist, especially since *K* has been only loosely defined as a function decreasing in the number of observations. Brinkhuis and Maris (2009) have shown, however, that conditions exist under which the parameters assume a stationary distribution.

It is clear that the Elo rating scheme is computationally very simple and fast, and therefore ideally suited for instantaneous on-line calibration methods. However, little is known about the statistical properties of the method, such as the rate of convergence, or even if the method is capable at all of recovering the parameters at an acceptable accuracy. The Elo rating method is widely used in situations in which the parameters may change rapidly during the collection of responses, such as in sports and games. The method has been applied in the Oefenweb (2009), an educational setting in which students exercise frequently within a gaming environment.

#### **<IP_ADDRESS> JML**

JML maximizes the likelihood *L* of a data matrix, where *N* test takers have each responded to *nj* items with item scores *xij* and sum score *sj*, where *Pj* is the probability of a correct response and *Qj* = 1 − *Pj* refers to the probability of an incorrect response. The likelihood function can be formulated as

$$L = \prod\_{j=1}^{N} \frac{n\_j!}{s\_j!(n\_j - s\_j)!} P\_j^{s\_j} Q\_j^{n\_j - s\_j}. \tag{16.5}$$

By maximizing the likelihood with respect to a single parameter, while fixing all other parameters, and rotating this scheme over all parameters in turn, the idea is that the global maximum for the likelihood function will be reached. In order to maximize the likelihood, a Newton-Raphson procedure is followed that takes the form

$$
\beta\_i^{t+1} = \beta\_i^t - \frac{\sum\_{j=1}^N P\_{ij} - \chi\_{ij}}{\sum\_{j=1}^N P\_{ij} Q\_{ij}} \tag{16.6}
$$

for updating item parameter β*i*, and

$$
\theta\_j^{t+1} = \theta\_j^t + \frac{\sum\_{i=1}^{\eta\_j} P\_{\bar{y}} - x\_{\bar{y}}}{\sum\_{i=1}^{\eta\_j} P\_{\bar{y}} Q\_{\bar{y}}} \tag{16.7}
$$

for updating person parameter θ*j*.

Like the Elo rating, JML is a simple and fast-to-compute calibration method. However, there are a few downsides. In the JML method, the item parameters are structural parameters. The number of them remains constant when more observations are acquired. The person parameters, on the other hand, are incidental parameters, whose numbers increase with sample size. Neyman and Scott (1948) showed in their paradox that, when structural and incidental parameters are estimated simultaneously, the estimates of the structural parameters need not be consistent when sample size increases. This implies that when the number of observations per item grows to infinity, the item parameter estimates are not guaranteed to converge to their true values. In practical situations, JML might still be useful as effects of non-convergence may become apparent only with extreme numbers of observations per item. The biggest issue with the use of JML, however, is a violation of the assumption regarding the ignorability of missing data. Eggen (2000) has shown that under a regime of item selection based on ability estimates, a systematic error will be built up. He has shown that only for MML, violation of the ignorability assumption has no effect.

#### **<IP_ADDRESS> MML**

MML is based on the assumption that the test takers form a random sample from a population whose ability is distributed according to density function *g*(θ|τ ) with parameter vector τ . The essence of MML is integration over the ability distribution, while a sample of test takers is being used for estimation of the distribution parameters. The likelihood function

$$L = \prod\_{j=1}^{N} \int P(\mathbf{x}\_{j}|\boldsymbol{\theta}, \boldsymbol{\xi}, \tau) g(\boldsymbol{\theta}\_{j}|\tau) d\boldsymbol{\theta}\_{j} \tag{16.8}$$

is maximized using the expectation-maximization algorithm (EM). ξ is the vector of item parameters here. EM is an iterative algorithm for maximizing a likelihood function for models with unobserved random variables. Each iteration consists of an E-step in which the expectation of the unobserved data for the entire population is calculated, and an M-step in which the parameters are estimated that maximize the likelihood for this expectation. The main advantage of the MML calibration method is that violation of the ignorability assumption does not incur biased estimators. A disadvantage, however, is that MML is usually more time consuming than the Elo rating or JML, and thus it might be too slow to be employed in situations where an instantaneous update is needed.

In order to replenish the item pool, Ban et al. (2001) proposed the MEM algorithm, where in the first iteration of the EM algorithm, the parameters of the ability distribution for the population are estimated through the operational items only. However, if the number of operational items is very small, this procedure results in highly inaccurate estimates. For the problem at hand, where not many operational items are available, the MEM method is not applicable, and the regular MML might be considered.

### *16.1.3 The Use of Reference Items in Modelling Bias*

Although bias cannot be avoided when calibrating CAT data with JML, previously collected responses for a part of the items can be used to model this bias. We assume that the bias is linear, i.e. that by applying a linear transformation βˆ - *<sup>i</sup>* = *a*βˆ *<sup>i</sup>* + *b*, estimators βˆ - *<sup>i</sup>* eliminate the bias. When JML is used to calibrate all items in the pool, the reference items, whose previously estimated parameters we trust, can be used to estimate transformation coefficients *a* and *b*. Let μ*<sup>r</sup>* and σ*<sup>r</sup>* be the mean and standard deviation of the previously estimated parameters of the reference items, while μ*<sup>c</sup>* and σ*<sup>c</sup>* are the mean and standard deviations of the parameter estimates for the same items, but now taken from the current JML-calibration, then the transformation coefficients are *<sup>a</sup>* <sup>=</sup> <sup>σ</sup>*<sup>c</sup>* <sup>σ</sup>*<sup>r</sup>* and *<sup>b</sup>* <sup>=</sup> <sup>μ</sup>*c*−μ*<sup>r</sup>* <sup>σ</sup>*<sup>r</sup>* . All reference items retain their trusted parameter values, while all new items in the calibration are updated to their transformed estimates.

Even though MML calibrations do not incur bias when applied to CAT data, it is technically possible to follow a similar procedure, but this should not result in substantially improved parameter estimates.

### *16.1.4 The Need for Underexposure Control*

A solution must be found for a complication that will generally only occur in the very first phases when few observations are available. For some response patterns, the associated estimates of the JML and MML methods assume extreme values. In the case of perfect and zero scores, the estimations will even be plus or minus infinity. On the other hand, items are usually selected according to the maximization of Fisher information. This means that once an item has assumed an extreme parameter value, it will only be selected for test takers whose ability estimation is likewise extreme. And thus, those items tend to be selected very rarely. If the parameter estimation is based on many observations, this is fully justified. But if this situation arises when only a few observations have been collected, these items will effectively be excluded from the item pool without due cause.

In previous studies, this situation was prevented by defining various phases in an operational CAT project, starting with either linear test forms (Fink et al. 2018) or random item selection from the pool (Makransky 2009). When the number of observations allowed for a more adaptive approach, a manual decision regarding transition to the next phase was taken. Since we assume a situation in which decisions are to be taken on the fly and have to be implemented instantaneously, manual decisions are to be avoided. Therefore, a rigid system to prevent underexposure must be present so that items with extreme estimations but with very few observations will remain to be selected frequently. Similar to the findings of Veldkamp et al. (2010), we propose an underexposure control scheme based on eligibility function

$$f(n\_i) = \begin{cases} X - n\_i(X - 1)/M, & n\_i < M \\ 1, & \text{else} \end{cases} \tag{16.9}$$

where *ni* is the number of observations for item *i*, *M* is the maximum number of observations for which the underexposure control should be active, and *X* is the advantage that an item without any observations gets over an item with more than M observations. This means effectively that there is no transition from different phases for the entire item pool, but each transition will take place on the level of individual items.

Overexposure may further improve the estimation procedure, but since items with only few observations form a larger impediment to the cooperation between estimation procedure and item selection than a loss of efficiency caused by overexposure, overexposure control is out of scope for this study.

### *16.1.5 A Combination of Calibration Methods*

Although Joint Maximum Likelihood (JML) calibration would meet operational requirements for running a CAT in terms of computational performance, its bias and its inability to estimate parameters for perfect or zero scores makes it unsuitable as a strategy for on-the-fly calibration. To overcome this issue, we propose a combination of Elo rating and JML as a strategy that meets the operational requirements for running a CAT. The Elo rating is used in the very beginning of the administration to ensure that the JML estimation procedure is converging for all answer patterns. Modelling the bias in JML with the help of a relatively small, but representative set of calibrated items, is proposed to eliminate the bias in JML. Although Marginal Maximum Likelihood (MML) estimation is not known to give biased estimates, it may turn out that this method also benefits from modelling the bias and starting with an Elo rating scheme.

Next to the use of well-calibrated reference items there is the need for underexposure control to ensure that we will collect an acceptable minimum of observations for all items to make calibration feasible.

### **16.2 Research Questions**

Research on on-the-fly item calibration in an operational setting for use in a CAT has different aspects. The operational setting requires the process to be accurate enough and quick enough. The main research question is: Does the proposed approach of a combination of Elo and JML yield sufficient accuracy for both item and persons parameters, under operational conditions? If the research question can be positively answered, there is no need for a large calibrated item pool for a CAT implementation. New items will be embedded in a set of reference items, while instantaneous on-thefly calibration enables us to base test takers' ability estimates on both new and reference items. At the same time, the use of reference items help to improve item parameter estimations.

Elaborating on the main research question, we formulate the following research sub questions:

Two simulation studies were conducted. The first study concentrated on the investigation of the use of reference items in the elimination of bias. The second study concentrated on a comparison of the methods with respect to their ability to recover the item and person parameters, in relation to the computation time needed.

### **16.3 Simulation Studies**

All simulations were based on the same item pool consisting of 300 items, 100 of which had "true" parameters drawn randomly from a standard normal distribution, supplemented by 200 items drawn from a uniform distribution from the interval (−3.5, …, 3.5). Furthermore, all simulated students were drawn from a standard normal distribution and took a test with fixed length of 20 items. Simulated students were assigned to the CAT one after another. The first item for each simulated student was drawn randomly from the entire item pool. For selecting the remaining 19 items, the weighted maximum likelihood estimator (WMLE) by Warm (1989) was used for the provisional ability, while items were selected that had the highest eligibility function value as given in (16.9), multiplied by the Fisher information at the provisional ability. At the start of each run, all item parameter estimates were set to zero, apart from the reference items that retained their trusted value. After each simulated student had finished the test, the item pool was calibrated. Two criteria were used to evaluate the measurement precision of the calibration methods: average bias and root mean square error (RMSE) of the item and person parameter estimators.

The Elo rating was used in the very beginning of the simulation runs. Using the Elo rating, in combination with the underexposure control, effectively prevented the situation that extreme estimations occurred and that items were thus no longer selected. All items that had a minimum of 8 observations and had a response vector consisting of both correct and incorrect answers were calibrated with JML or with MML. At this moment, extreme parameter estimates purely due to chance begin to become somewhat unlikely. For the items not having this threshold yet, the Elo rating was retained.

### *16.3.1 Use of Reference Items in Elimination of Bias*

To validate the reference items approach with JML, we compared the results from JML with and without reference items. After sorting the item pool in ascending difficulty, the reference items were chosen to be evenly spread in an appropriate difficulty range. It may be assumed that such an evenly spread reference set will yield more accurate estimations for the transformation coefficients than, for example, a normal distribution. Items with more extreme difficulties, however, may be selected less frequently and less optimally than items with moderate difficulty. This difference in selection may well influence the extent to which the reference items model the bias. Therefore, two different sets of reference items were simulated in order to investigate which set is able to eliminate the bias best. The first set comprised 20 items with parameters from the interval (−2.5, …, 2.5), while in the second set, 20 items from the full range were selected. The item parameters in both reference sets were evenly spread. In summary, we investigated three different conditions in the first simulation study: Two different ranges of reference items (limited vs. full range), while the third condition was the application of JML without reference items. Each condition was repeated 1000 times and each of these runs consisted of 3000 simulated students. This coincided with the moment that the average number of observations is 200 per item, which is a reasonable choice for calibrations under the 1PL model. Furthermore, bias for each item, averaged over the simulation runs, was investigated when 600 simulated students had finished their tests. This moment coincides with the moment that the average number of observations is 40. This point can be seen as being in a transition from the situation in which no information at all is available to the situation in which calibrations under the 1PL model are starting to become stable.

#### **<IP_ADDRESS> Results**

Figure 16.1 shows two scatter plots of βˆ against β at the end of a single run, i.e. when each item had on average 200 observations. The left scatter plot depicts the estimates for the JML, while the right one shows the estimates in case of JML with the limited set of reference items. It shows a typical situation that can be observed in all simulation runs. It can be seen that a linear transformation based on a set of reference items is effective in reducing the bias, even if it can be surmised that the reduction is not perfect for items with extreme parameter values. Nevertheless, the JML without reference items shows bias to such a large extent that it has not been considered any further in the remainder of the simulations.

In Fig. 16.2, the bias in βˆ is given. The bias for the limited set of reference items is presented in the left graphs, in the right graphs the bias for the full set. The top row shows the bias for each item, averaged over the simulation runs, when 600 simulated students have finished their test, the bottom row at the end of the runs. When an average of 40 observations per item have been collected, the exclusion of extreme items in the reference set lead to a pronounced overcompensation, while using items from the full difficulty range resulted in overcompensation only at the extremes. The situation at an average of 200 observations per item is different: using reference items from the full difficulty range lead to a clear outward bias for the medium items and an inward bias for the extreme items. In the early phases of the simulations, a linear transformation based on the full set gave the best results, but overcompensation caused a situation in which the error in parameters gradually increased. As it may be surmised that items with medium difficulty will be selected more frequently than items with extreme difficulty, it can be expected that the ability estimates will be more accurate when extreme items will be excluded from being reference items. A second

**Fig. 16.1** Scatter plot of βˆ against β for JML without (left) and JML with reference items (right – limited set) at an average of 200 observations per item

**Fig. 16.2** Bias of βˆ for JML with two different sets of reference items at an average of 40 and 200 observations per item

disadvantage for using extreme items in the reference set is the overcompensation as it can be seen when a considerable number of observations has been collected. Therefore, only JML with a set of reference items selected from a limited difficulty range was considered in the second study. This condition is from here on referred to plainly as JML.

### *16.3.2 Comparison of the Methods*

The settings of the second set of simulations were similar to those of the first set. Like the first set of simulations, we evaluated the bias for person and item parameters after 3000 simulated student, thus making a direct comparison possible. As an extension, we reported the RMSE for the person parameter after each simulated student, as well as the average of the RMSEs for all item parameters. In order to investigate convergence of the methods beyond an average of 200 observations per item, each run consisted of 15,000 simulated students instead of 3000, thus extending the simulation to an average of 1000 observations per item. The conditions that were investigated were:

#### **<IP_ADDRESS> Results**

Figure 16.3 shows the bias of βˆ for the four different simulation conditions, taken after 3000 simulated students. The top left graph shows the results for the Elo method, the top right graph for JML, and bottom left for MML, while in the bottom right graph the results for MML with reference items are presented.

As can be inferred from Eqs. (16.3) and (16.4), the Elo rating update scheme is an effective method in preventing extreme estimations in cases of perfect and zero scores, but in the long run this appears to become a disadvantage. The Elo rating shows a large inward bias in Fig. 16.3, while MML shows an outward bias. The two conditions using reference items showed the best results. JML yielded an increasing inward bias for extreme item parameters, and the use of reference items in the case of MML compensated the bias almost completely.

Recovery of person parameters is equally important as the recovery of item parameters, if not more important. The purpose of a test usually is an evaluation of test takers, item calibration is merely a means to achieve this. Figure 16.4 shows the bias in θˆ. The bias in θˆ reflected the situation for βˆ: The Elo rating showed substantial inward bias, MML an outward bias, while the use of reference items reduced bias also in the person parameter estimates. In the case of JML, the extreme ability ranges showed some inward bias while a small overcompensation took place for MML.

Figure 16.5 illustrates the development of RMSE(βˆ) during the simulation runs, now averaged over all items. As can be seen, the average RMSE(βˆ) rapidly diminished in the first phases of the process. The Elo rating, combined with the underexposure control, effectively prevented estimations with extreme values when the first simulated students took their tests. The disadvantage of the Elo rating becomes

**Fig. 16.3** Bias of βˆ for Elo, JML, MML and MML with reference items at an average of 200 observations per item

**Fig. 16.4** Bias of θˆ for Elo, JML, MML and MML with reference items at an average of 200 observations per item

**Fig. 16.5** Average RMSE(βˆ) at increasing numbers of observations

apparent later on in the runs: The Elo rating is slowest to converge of the methods under consideration. At an average of 1000 observations per item, RMSE(βˆ) was on average 0.351. This situation was reached at an average of approximately 70 observations per item for JML and 60 for MML with reference items. JML converged faster in the initial phase, but with increasing numbers of observations, MML started to outperform JML. MML with reference items had a clear advantage over JML from 50 observations per item. The average RMSE(βˆ) was 0.383 at that moment.

**Fig. 16.6** Average RMSE(θˆ) at increasing numbers of observations

Without using reference items, MML outperformed JML from approximately 500 observations per item, when the average RMSE(βˆ) was 0.191.

Figure 16.6 presents a different situation. It shows the development of average RMSE(θˆ). In all conditions, apart from the case of MML without reference items, average RMSE(θˆ) decreased rapidly. At approximately 50 observations per item, average RMSE(θˆ) decreased below 0.50 for the Elo ratings. This value was reached at approximately 30 observations per item in the JML and MML with reference items. In the MML condition without reference items, the lowest value for average RMSE(θˆ) was 0.51 when an average of 1000 observations per item were collected. As a baseline, a simulation was conducted where all item parameters assumed their true value in the item selection algorithm and thus no calibration took place. In this situation, average RMSE(θˆ) reached a value of 0.48. This value was reached only in the Elo rating condition, albeit because θˆ showed a large inward bias.

#### **<IP_ADDRESS> Computational Performance**

One important issue for on-the-fly calibration in operational settings still remains open: computational performance. Ideally, a calibration method gives highly accurate estimations almost instantly, but in case that such an ideal is not reached, a compromise between accuracy and speed must be made. Unfortunately, in operational settings speed depends not only on the method to be used, but also on other factors like system architecture and interfacing between various software modules. Therefore a complete evaluation of the compromise between accuracy and speed falls outside the scope of this chapter. Nevertheless, measurements of computer times during the simulations do give an indication of the relative performance of the calibration methods under consideration.

The Elo rating method is clearly the fastest method, since it makes use of only the responses of one test taker in each cycle and does not consider the entire data matrix. Running on a laptop equipped with an i5-5Y57 CPU at 1.6 GHz, one Elo update cycle typically used less than 0.1 ms, regardless of the number of previous responses on the respective items. On the other hand, JML and MML were substantially more computation intensive and showed different time complexity. After 300 simulated students, that is, when on average 20 observations were collected, the average time for one JML calibration cycle took approximately 800 ms. The MML calibration took on average approximately 2.3 s to complete. At 600 simulees, the computation times had increased to 1.6 and 3.8 s for JML and MML, respectively. The computation time increased roughly linearly to the end of the simulation runs, when each JML cycle needed an average of 42 s and a single MML cycle used approximately 140 s to complete. During all phases of the runs, the MML showed negligible differences in computation time between the condition with and without reference items.

### **16.4 Discussion**

From a measurement perspective, MML clearly outperforms JML and Elo ratings, while the use of reference items has an advantage in stabilizing the scale in the early phases of data collection for an item pool while in operation. In the case of JML, the use of reference items should always be considered. But also in the case of MML, reference items are an effective means to eliminate bias. Computational performance, however, heavily depends on external factors such as system architecture and cannot be evaluated in a simulation study as we have conducted. What is clear, though, is that the Elo ratings are extremely fast to compute so that they can be employed in synchronous processes immediately after receiving a response. This is a reason why many large-scale gaming and sports environments use some variant of this method. Furthermore, it is a very useful method in the beginning of data collection processes, when the number of observations is still very low. On the other hand, JML and MML are considerably slower, so that it can be expected that employment of those methods will be deemed feasible only in asynchronous processes. As a consequence, the update of parameters will not be instantaneous but will take a few seconds or more to be effective. In a setting with many thousands of test takers taking the test concurrently, this would be an unwanted situation and the Elo rating system would probably be the method of choice.

When more accurate estimations than those provided by the Elo rating are needed, MML can be used. If reference items are available, JML is an option as well, whereby it could be expected that MML with reference items produce more accurate estimations at the cost of a somewhat heavier computational burden as compared to JML. In both cases it can be expected that item parameter estimations will have stabilized when an average of approximately 200 observations per item have been collected. At this stage, each consecutive calibration will not change parameter values substantially. Thus, it might be wise not to calibrate the items after each test taker, but to use a less intensive routine. Such a regime could well be an off-line calibration, whereby, for example, item fit statistics can be inspected and actions undertaken when items appear to be defective.

Two important questions were left unanswered in this study: "How many reference items are needed?" and "How many new items can be added?". It is obvious that the answer to both questions depend on factors such as model fit and accuracy of the reference item estimates. As all responses were simulated under ideal circumstances, that is, all items showed a perfect model fit and all reference items were modelled as having no error, we assume that using two reference items would have been enough to accurately estimate the transformation coefficients, resulting in findings similar to the ones reported here.

The first simulation study, validating the reference items approach, showed that a linear transformation does not completely remove the bias incurred by violation of the ignorability assumptions. A more advanced model of the bias involving a non-linear transformation would probably improve the parameter recovery for the JML calibration, potentially up to the level currently seen with MML using reference items.

Based on findings from the simulation studies, we suggest to apply the following strategy in an on-the-fly calibration in CAT: The entire process of data collection and on-the-fly calibration in an operational CAT setting should start with an Elo rating update scheme in order to avoid extreme parameter estimates, combined with a rather rigorous underexposure control scheme. After this initial phase, a switch to JML using reference items should be made. When increased accuracy is worth the additional computational burden of using MML, a new switch could be made and the on-the-fly calibration can either be slowed down or switched off entirely when parameters appear to be stabilized.

### **References**

Elo, A. (1978). *The rating of chess Players, past and present*. London: B.T. Batsford Ltd.

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Chapter 17 Reinforcement Learning Applied to Adaptive Classification Testing**

**Darkhan Nurakhmetov**

**Abstract** This study investigates how computerized adaptive classification testing task can be considered as a sequential decision process and made accessible to Reinforcement Learning. The proposed method performs a sequential item selection that learns which items are most informative, choosing the next item depending on the already administered items and the internal belief of the classifier. A simulation study shows its efficiency for tests which require to make a confident classification decision with as few items as possible. Solutions for a variety of practical problems using the proposed method are considered in this study.

### **17.1 Introduction**

The key component of a computerized classification testing (CCT)/adaptive testing (CAT) program is the "adaptive" item selection algorithm, which should find the best suited items for each test taker based on his or her estimated ability.

Several item selection methods have been proposed based on item response theory, including the maximization of item information at the current θ estimate (Reckase 1983), the cut-score (Spray and Reckase 1994, 1996), maximization of item information across a region for θ (Eggen 1999), a log-odds ratio (Lin and Spray 2000), and maximization of item information across θ (Weissman 2004).

In addition to correct classification, several other concerns have to be taken into account when selecting the next item in an operational CAT program. Various nonstatistical constraints need to be considered, such as content balancing and exposure control. Several heuristics are proposed for this purpose, including the weighted deviation modelling (WDM) method (Stocking and Swanson 1993), the normalized weighted absolute deviation heuristic (NWADH; Luecht 1998), the maximum priority index (MPI) method (Cheng and Chang 2009); randomization strategies, e.g. randomesque strategy (Kingsbury and Zara 1989); conditional selection strategies,

D. Nurakhmetov (B)

University of Twente, Enschede, The Netherlands e-mail<EMAIL_ADDRESS>
<sup>©</sup> The Author(s) 2019

B. P. Veldkamp and C. Sluijter (eds.), *Theoretical and Practical Advances*

*in Computer-based Educational Measurement*, Methodology of Educational Measurement and Assessment, https://doi.org/10.1007/978-3-030-18480-3\_17

e.g. targeted exposure control strategy (Thompson 2002), shadow test approach (van der Linden and Veldkamp 2005); combined strategies (Eggen 2001), e.g. combined application of Sympson-Hetter strategy (Sympson and Hetter 1985) and Progressive strategy (Revuelta and Ponsoda 1998); maximum information with content control and exposure control (Eggen and Straetmans 2000).

Differences in item selection methods may lead to different choices of items for the same test takers and, consequently, to different classification decisions. The choice of the item selection method is therefore one of the most important parts of computerized adaptive testing.

Test content balancing and item exposure control are separate steps that are generally integrated in the item selection heuristics by limiting the items available for selection or by adding small probability experiments to limit over- or underexposure of items. This study applies a new adaptive item selection method. It explores the possibility to solve problems, such as exposure control and content balancing, that may occur in computerized adaptive classification testing (CACT), and simultaneously attempts to optimize item selection.

### **17.2 Method**

Fisher information is commonly used for item selection, but this information measure is based on an estimate of the ability parameter, and the ability estimate is not very stable, particularly in the beginning of a CAT administration. Therefore, when the estimate is not close to the true value, using the Fisher information criterion might result in inefficient item selection. The observation that item selection procedures may favor items with optimal properties at wrong ability values is generally known as the attenuation paradox (Lord and Novick 1968, Sect. 16.5).

The foundation of new methodology for incorporating expert test development practices in the construction of adaptive classification tests is the application of *reinforcement learning*. It has been applied successfully to various selection problems, including robotics, elevator scheduling, telecommunications, a few games, such as backgammon, checkers and go (see Sutton and Barto 1998). Reinforcement learning refers to goal-oriented algorithms, which learn how to attain a complex objective or maximize along over a particular dimension over many steps.

There are two major factors that make reinforcement learning powerful: the use of samples to optimize performance and the use of function approximation to deal with large environments. Reinforcement learning can solve the difficult problem of correlating immediate actions with the delayed returns they produce. Reinforcement learning can be used in the situation, where there is no information about the environment and the only way to collect it is to interact with it. This could be considered to be a genuine learning problem, which is fit to adaptive classification testing, for example to get information about a test taker's state (classification state) it is needed to give him/her an item (action) and get a response.

In short, an improved approach of item selection is presented, which not only optimally spans the item pool (input space), but also optimizes with respect to data consumption, that is, minimizes the test length needed to classify respondents. Going beyond traditional item selection methods in the computerized adaptive testing, in this paper, we lay out and demonstrate an approach of selecting items in sequence, making the decision which item to select next dependent on previously selected features and the current internal state of the supervised method that it interacts with. In particular, our sequential item selection (SIS) algorithm will embed Reinforcement Learning (RL) into classification tasks, with the objective to reduce data consumption and associated costs of item selection during classification. The main question of this chapter is: "Where do I have to look next, in order to keep data consumption and costs low while maintaining high classification results?" or "Which item should I deliver to the test taker next, in order to keep test constraints satisfied while maintaining high classification results?"

The Framework is mapped out in the following section. After introducing the general idea, the formal definition of sequential classifiers and rephrasing the problem as a Partially Observed Markov Decision Process (POMDP) is represented (see Astrom 1965). In addition, an algorithm to take exposure and content control into consideration is introduced.

### **17.3 Framework**

### *17.3.1 General Idea*

Machine learning is often applied to classification problems: mapping an input x to one of a finite set of class labels C. Regarding *classification testing*, the formulation of the initial problem remains the same: based on a series of item responses of the test takers (vector *x*), class *C* is estimated.

In classification testing, item selection is needed as key option for good classification results: filtering out less informative items, while paying attention to exposure and content control. Reformulating this to classic machine learning: item selection is a combinatorial optimization problem that tries to identify those items, which will minimize the generalization error, with different constraints. In particular, item selection can be seen as a process that tries to reduce the amount of redundant data.

Turning classification into a sequential decision process results in item selection and classification becoming an adaptive and intertwined process: deciding which item to select next depends on the previously-selected items and the behavior of the classifier on them. This will be achieved by using a fully trained classifier as an environment for a Reinforcement Learning agent, that learns which item to select next, receiving the reward on successful classification of the partially uncovered input response pattern. The general schema of reinforcement learning is described below (Fig. 17.1).

Figure 17.1 represents how reinforcement learning works in general: an agent interacts with an environment by taking actions, which is translated into a reward and a representation of the state, which are fed back to the agent. Typical framing of RL can be mapped to CACT as follows:

*Agent.* An agent takes actions. In CACT, the algorithm which select items or terminate a test is an agent.

*Action.* Action is the set of all possible moves the agent can make. An action is almost self-explanatory, but it should be noted that agents choose among a list of possible actions. In CACT, the list includes all the items and the action which refers to termination of the test.

*Environment*. The environment is the world through which the agent moves. The environment takes an agent's current state and action as input, and returns as output the agent's reward and its next state. In CACT, the environment could be the rules of our classification process, such as, the metric used for making decisions, the maximum number of items in the test, the exposure rate, etc.

*State.* A state is a concrete and immediate situation in which an agent finds itself; i.e. a specific moment, an instantaneous configuration that puts an agent into relation to other significant things. In CACT, it can be the number of items that have already been administered, the probabilities of selecting next items, etc.

*Reward.* A reward is the feedback by which we measure the success or failure of the agent's actions. Rewards can be immediate or delayed. In CACT, rewards can be granted after an agent terminates the test. The reward is delayed, when it is granted after the test has ended. An immediate reward in CACT can be provided as a penalty for the agent, when an agent chooses an item that has already been administered, or an item that would bring the exposure rate over the limit.

### *17.3.2 Sequential Classification*

First of all, it is needed to formulate CACT as a Partially Observed Markov Decision Process (POMDP), making the problem sequential and thus accessible to Reinforcement Learning algorithms. For detailed information about POMDP, the reader is referred to Astrom (1965). For now we focus on the implementation of POMDP. Therefore, some specific notation and some definitions have to be introduced.

Specific notation used in this chapter:

( )—ordered sequences,

{ }—unordered sequences,

*a* • *k*—appending an element *k* to *a*.

Besides, several concepts have to be defined:

*Power sequence.* Related to power sets, a power sequence, denoted as powerseq(*M*), is defined as a set of all permutations of all elements in the power set of *M*, including empty sequence ( ). As an example, for *M* = {0,1}, the resulting powerseq(*M*) = {( ), (0), (1), (0,1),(1,0)}.

*Episode*. Each test administration is called an episode. During an episode, the item history *ht* ∈ powerseq(*F*) is the sequence of all previously selected items in an episode up to and including the current item at time *t*.

*Cost*. Costs associated with accessing an item *f* are represented as negative scalars *r*− *<sup>f</sup>* <sup>∈</sup> <sup>R</sup>,*r*<sup>−</sup> *<sup>f</sup>* < 0.

*Reward.* A non-negative global reward is defined as*r*<sup>+</sup> <sup>∈</sup> <sup>R</sup>,*r*<sup>+</sup> <sup>≥</sup> 0 for correctly classifying an input.

*Classifier*. Classifiers in general are denoted with the symbol *K*. A sequential classifier is defined by *K\**, and it is to be a functional mapping from the power sequence of item responses to a set of classes, i.e., *K*<sup>∗</sup> : powerseq- { *f* (*x*)}*<sup>f</sup>* <sup>∈</sup>*<sup>F</sup>* → *C*.

A requirement of CACT as POMDP is to process the sequence one input at a time in an online mode, rather than classifying the whole sequence at once. Besides, as output, a class label is added to each input. Therefore, *K\** requires some sort of memory. Recurrent Neural Networks (RNN) are known to have implicit memory that can store information about inputs seen in the past (see Hopfield 1982; Hochreiter and Schmidhuber 1997). In this chapter RNN and RNN with Gated Recurrent units (GRU) are used (Cho et al. 2014).

Dealing with a POMDP implies that we need to extract an observation from the classifier that summarizes the past into a stationary belief. Most classifiers base their class decision on some internal belief state. A Feed Forward Network (FFN), for example, often uses a softmax output representation, returning a probability *pi* in [0,1] for each of the classes with *<sup>C</sup> <sup>i</sup>*=<sup>1</sup> *pi* = 1. If this is not the case (e.g., for purely discriminative functions like a Support Vector Machine), a straightforward belief representation of the current class is a *k*-dimensional vector with 1-of-*k* coding.

To finally map the original problem of classification under the objective to minimize the number of items to be administered to POMDP, the elements of the 6 tuple (*S*, *A*, *O*, P,Ω, *R*) which describes POMDP can be described as follows. The state *s* ∈ *S* at timestep *t* comprises the current input response pattern *x,* the classifier *K\**, and the previous item history *ht*-*<sup>1</sup>*, so that *st* = (*x*, *K*∗, *ht*−<sup>1</sup>). This triple suffices to fully describe the decision process at any point of time. Actions *at* ∈ *A* are chosen from the set of items *F\ht*-*<sup>1</sup>*, i.e., previously administered items are not available. The observation *ot* ∈ *O* is represented by the classifier's internal belief of the class after seeing the values of all items in *ht*−*<sup>1</sup>*, written as *ot* = *b*(*x*, *K*∗, *ht*−<sup>1</sup>) = *b*(*st*). The probabilities *pi* for each class serve as an observation to the agent: *ot* = *b*(*x*, *K*∗, *ht*−<sup>1</sup>) = - *p*1, *p*2,..., *p*|*C*<sup>|</sup> .

Assuming a fixed *x* and a deterministic, pretrained, classifier K\*, the state and observation transition probabilities P and collapse and can be described by a deterministic transition function T, resulting next state *st*+<sup>1</sup> = *Tx* (*st*, *at*) = (*x*, *K*∗, *ht*−<sup>1</sup> · *at*) and next observation *ot*+<sup>1</sup> = *b*(*st*+1). Finally, the reward function *<sup>R</sup><sup>a</sup> ss* returns the reward *rt* at timestep *t* for transitioning from state *st* to *st*+*<sup>1</sup>* with action *at*. Given *c* as the correct class label, it is defined as:

$$r^t = \begin{cases} r^+ + r\_{a\_l}^- \\ r\_{a\_t}^- \end{cases} \\ \\ \\ \begin{aligned} (h\_\tau(\alpha))\_{0 < \tau \le t}) = c \\ \qquad \qquad el \,\nu \end{aligned}$$

### *17.3.3 Item Selection*

In CACT, one needs to ensure that an item (action) is only chosen at most once per test taker (per episode), i.e., the set of available actions at each given decision step is dependent on the history *ht* of all previously selected actions (items) in an episode (for one test taker). Note that this does not violate the Markov assumption of the underlying MDP, because no information about available actions flows back into the state and therefore the decision does not depend on the item history.

Value-based reinforcement learning can solve this problem. The action value V is the expected return for selecting action in a state and following policy. By manually changing all action-values V(o, at) to −∞ after choosing action (item) at, this leads to all actions not previously chosen in the current episode having larger value and be preferred over *at*. A compatible exploration strategy for this action selection without replacement is Boltzmann exploration (Cesa-Bianchi et al. 2017). Here a probability of choosing an action is proportional to its value under the given observation:

$$p(a\_t|o\_t) = \frac{e^{V(a\_t|o\_t)/\mathfrak{r}}}{\sum\_a e^{V(a\_t|o\_t)/\mathfrak{r}}},$$

where τ is a temperature parameter that is slowly reduced during learning process for greedier selection towards the end. Thus, when selecting action *at+1*, all actions in *ht* have a probability of *e*−∞ = 0 of being chosen again. At the end of an episode, the original values are restored.

Having defined the original task of classification as a POMDP and solved the problem of action selection without replacement (item selection), it is possible to use existing algorithms for solving this class of problems. Since the transition function is unknown to the agent, it needs to learn from experience, and a second complication is the continuous observation space.

### *17.3.4 Algorithm*

In this chapter, the actor-critic algorithm is used (see Mnih et al. 2016). The Actorcritic algorithm maintains a policy π((*tt*, *at*)|*st*; θ ) and an estimate of the value function *V*(*st*; θυ ), where θ—parameters, which are learned by iteratively minimizing a sequence of loss functions, and policy represents how items (actions) are picked from the item pool; *t*—termination gate—action which represent the termination step of the test. Policy and a value can be briefly described as follows:

*Policy.* A policy is the strategy that the agent employs to determine the next action based on the current state. It maps states to actions, the actions that promise highest reward.

*Value.* The expected long-term return, as opposed to the short term reward. Value is defined as the expected long-term return of the current state under the policy.

The actor-critic algorithm operates in a forward view and uses a mix of *n*-step returns to update both the policy and the value-function. The general schema of the actor-critic algorithm is shown in Fig. 17.2. The policy and the value function are updated after every *tmax* actions or when a terminal state is reached. The update performed by the algorithm can be seen as ∇<sup>θ</sup>logπ((*tt*, *at*)|*st*; θ )*A* - *st*, *at*; θ,θ where *A* - *st*, *at*; θ,θ is an estimate of the advantage function given by*k*−<sup>1</sup> *<sup>i</sup>*=<sup>0</sup> <sup>γ</sup> *<sup>i</sup> rt*+*i*+ γ *kV*(*st*+*<sup>k</sup>* ; θv)−*V*(*st*; θv), where *k* can vary from state to state and is upper-bounded by *tmax* .

The state sequence *s* is hidden and dynamic, controlled by an RNN sequence model. The Actor-critic algorithm performs a "classification" action *aT* at the *T*-th step, which implies that the termination gate variables equal *t1:T*= *(t1*= *0, …, tT*−*1*= *0, tT* = 1*)*. Basically, at the *T*-th step, the actor-critic model (ACM) gets enough information to terminate tests and make a classification decision. The ACM learns a stochastic policyπ((*tt*, *at*)|*st*; θ )with parameters θ to get a distribution of termination actions, to continue administering items, or to stop, and of a 'classification' action,

**Fig. 17.2** Typical framing of actor-critic algorithm

if the model decides to stop at the current step. The termination step *T* can vary from test taker to test taker. However, it is possible to set the maximum of *T*, to ensure that a test ends anyway at step *T*. Then, if T is reached, classification decision is made by existing information from steps 1 to *T*.

The parameters θ are trained by maximizing the total expect reward. The expected reward for a test taker is defined as:

$$J(\theta) = \mathbb{E}\_{\pi\_{(t\_{1:T}, a\_T; \theta)}} \left[ \sum\_{t=1}^T r\_t \right]$$

The reward can only be received at the final termination step when a classification action *aT* is performed. We define *rT* = 1 if *tT* = 1 and the classification is correct, and *rT* = −1 otherwise. The rewards on intermediate steps are zeros, except for cases when items are repeatedly selected, in that case rewards are also −1. This is done to penalize the agent for picking same items for the same test taker; and cases when selecting an item results in exceeding the limit in exposure rate, in that cases reward −0.5. *J* can be maximized by directly applying gradient-based optimization methods. The gradient of J is given by:

$$\nabla\_{\theta} J(\theta) = \mathbb{E}\_{\pi(t\_{\mathbb{L}:T}, a\_{T}; \theta)} [\nabla\_{\theta} \log \pi \left( (t\_{\mathbb{L}:T}, a\_{T}; \theta) r\_{T} \right)]$$

Motivated by the REINFORCE algorithm (Williams 1992), the following computation of ∇<sup>θ</sup> *J* (θ) is estimated:

$$\begin{aligned} & \mathbb{E}\_{\pi(t\_{\mathbb{L}\cdot T}, a\_{T}; \theta)}[\nabla\_{\theta} \log \pi((t\_{\mathbb{L}\cdot T}, a\_{T}; \theta)r\_{T}]} \\ & = \sum\_{(t\_{\mathbb{L}\cdot T}, a\_{T}) \in \Lambda^{\dagger}} \pi(t\_{\mathbb{L}\cdot T}, a\_{T}; \theta) [\nabla\_{\theta} \log \pi((t\_{\mathbb{L}\cdot T}, a\_{T}; \theta)r\_{T} - b\_{T}], \end{aligned}$$

where A† represents all the possible episodes, and *T*, *t1:T* , *aT* and *rT* are the termination step, termination action, classification action, and reward, respectively, for the (*t*<sup>1</sup>:*<sup>T</sup>* , *aT* ) episode, while. *bT,* which is called the reward baseline in the RL literature is introduced to lower the variance (Sutton 1984). It is common to select *bT* <sup>=</sup> <sup>E</sup><sup>π</sup> [*rT* ](Sutton et al. 1999), and this term can be updated via an online moving average approach: *bT* = λ*bT* + (1 − λ)*rT* . However, this might lead to slow convergence in training the ACM. Intuitively, the average baselines {*bT* ; *T* = 1…*T max*} are global variables, independent of test takers. It is hard for these baselines to capture the dynamic termination behavior of an ACM. Since an ACM may stop at different time steps (different test length) for different test takers, the adoption of a global variable without considering the dynamic variance for each test taker is inappropriate. To resolve this weakness in traditional methods and to account for the dynamic characteristics of an ACM, an instance-dependent baseline method to calculate ∇<sup>θ</sup> *J* (θ) is proposed. The gradient can be rewritten as:

17 Reinforcement Learning Applied … 333

$$\nabla\_{\boldsymbol{\theta}}J(\boldsymbol{\theta}) = \sum\_{(\boldsymbol{t}\_{\mathrm{i}:T},a\_{T}) \in \mathbb{A}^{\dagger}} \pi(\boldsymbol{t}\_{\mathrm{i}:T},a\_{T};\boldsymbol{\theta})[\nabla\_{\boldsymbol{\theta}}\log\pi((\boldsymbol{t}\_{\mathrm{i}:T},a\_{T};\boldsymbol{\theta})\boldsymbol{r}\_{T}-\boldsymbol{b}),$$

where the baseline *b* = (*t*1:*<sup>T</sup>* ,*aT* )∈A† π(*t*1:*<sup>T</sup>* , *aT* ; θ)*rT* is the average reward on the <sup>|</sup>A†<sup>|</sup> episodes for the *<sup>n</sup>*-th training test taker. It allows different baselines for different training test takers. This can be beneficial since the complexity of training test takers varies significantly.

### **17.4 Experiments**

In this section, evaluation of performance of the ACM is presented. A simulation study utilized a Monte Carlo simulation methodology, with 1000 test takers multiplied by 100 administrations simulated, to evaluate differences in efficiency and accuracy. The population of test takers was randomly drawn and selected from a *N(0,1)* distribution. With Monte-Carlo simulation, item responses are generated by comparing a randomly generated number 0.0 < *r* < 1.0 to the probability of an each possible response for each test taker to each item. The probability is calculated using the Generalized Partial Credit Model (GPCM) (see Muraki 1992) and the true test taker's θ, which is known.

If items are randomly selected, this means that, after each administration, each item that has not been used has an equal chance of being selected. However, with such a method of selecting items, the contents of the test are not matched to the ability of the candidate, consequently, accuracy of decisions may result as inappropriate, and testing is not efficient. Simulation studies have shown that random selection eliminates out the possible gains of adaptive testing. If items are randomly selected, there is an average gain in item pool utilization and exposure rate, but also a loss of accuracy.

Although, in CACT, the optimum item selection method makes sure that each test taker gets a different test, it still occurs that some items from the item pool are administered very frequently while others are never used or hardly ever used. The gains in efficiency go along with two following characteristics of the utilization of the item pool, which result in the following problems:

Table 17.1 contains the classification accuracies, maximum exposure rate and item pool utilization for the simulation studies. Table 17.1 also shows that problems, that have been explained above, do not occur with random item selection, where all items are used. However, with random selection there is a loss in accuracy.

The RL-CACT, selecting items on the basis of ACM with *T* =10 (maximum length of the test) and with *T* =15, uses 65 and 71% items out of item pool, respectively. This

**Table 17.1** Results of simulation studies

is done by implementation of the ACM reward policy as explained in the previous section of this chapter.

Regarding the accuracies, ACM with both T = 10 and T = 15 performed with 89.8 and 92.1% of correct decisions, respectively. Maximum exposure rates are 33.1 and 34.3%. One can vary these rates by changing the reward for the agent, e.g. if one wants to have maximum exposure rate less than 30%, it can be done by tightening penalty for overexposing items.

### **17.5 Discussion**

Classification CAT is formulated as a POMDP and thereby it is made accessible to RL methods. In this chapter, the main focus is on minimization the number of items (test length) needed to make a confident classification decision. This is done by training the RL agent to pick items that lead to quick and precise classification. Exposure control is maintained by using the different rewards for action selection, which takes into account the item selection history for each test taker. The proposed reward distribution penalizes the agent during the training process if it picks an item, which already has been administered for a certain test taker. Also, it penalizes the agent if the agent selects an item, for which the exposure rate limit is exceeded.

Another issue that might occur with adaptive testing is the content balancing problem. The presented approach can solve this by implementing a specific mask for the items or by giving different rewards for sequences of items. For example, if it assembles a test from an item pool with *n* domains and *M* items for each domain, and there is a constraint that exactly *m* of items from each domain have to be administered. Then we can define the following reward distribution function: give the reward *R* for the correct classification, and subtract value *d* for each unadministered domain, or *d/m* for each missing item from domain.

Any additional constraints can be taken into account by tuning the reward distribution. For instance, for different types of assessment, a different outcome might be desired, e.g. one wants higher accuracy, or higher precision or recall. This can be done by varying rewards during the training process, e.g., after classifying a small sample of test takers, a (batch) additional reward, depending on the desired outcomes, could be implemented. For instance, if in particular batch the number of type II errors exceed limits, then the reward can be reduced by a specific value.

In summary, the proposed method for adaptive item selection in computerized classification testing is efficient for tests requiring to make a confident classification decision with as few items as possible. Moreover, the proposed method can solve problems, such as overexposure and underexposure, with content-distribution constraints, through the reward allocation. However, the design of CACT with ACM model requires simulation research to ensure that the test is implemented to be as efficient as possible.

### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# Part V Technological Developments in Educational Measurement

# **Chapter 18 Feasibility and Value of Using a GoPro Camera and iPad to Study Teacher-Student Assessment Feedback Interactions**

### **Fabienne van der Kleij, Lenore Adie and Joy Cumming**

**Abstract** The importance of effective feedback for learning is widely recognised. However, much of the previous research on assessment feedback has paid limited attention to the nature of feedback interactions, and teacher and student perspectives. Innovations in technology have opened up possibilities to capture such interactions for research and professional development purposes. This study investigated the feasibility and value of using easily accessible equipment for practitioner use and research purposes. Specifically, the study focussed on use of a GoPro camera and an Apple iPad in capturing one-to-one teacher-student feedback interactions, and subsequent individual video-stimulated recall (VSR) for self-reflection. Six teacherstudent pairs in three Australian secondary schools participated in the research. Evidence collected included technical data and characteristics of the devices, teacher and student experiences, and researchers' experiences and reflections. While the iPad and GoPro were both useful for capturing interactions, the iPad was more useful and practicable for VSR interviews. Findings suggest that such technology has potential for use by teachers and students to improve reflection and feedback interaction, and thus to enhance student learning. The researchers identified several technical issues that occurred in a range of settings and make suggestions to overcome these issues.

### **18.1 Introduction**

The importance of effective feedback for learning is widely evidenced in international education research and policy (Stobart 2008; Wiliam 2011). The purpose of feedback is to provide information, to both teachers and students, to improve teaching and learning. While feedback can take many forms (Van der Kleij et al. 2015), feedback within classroom practice is generally realised through written and oral communications and interactions between teachers and students. The significance of

F. van der Kleij (B) · L. Adie · J. Cumming

Institute for Learning Sciences & Teacher Education, Australian Catholic University, Brisbane, Australia

e-mail<EMAIL_ADDRESS>
<sup>©</sup> The Author(s) 2019

B. P. Veldkamp and C. Sluijter (eds.), *Theoretical and Practical Advances*

*in Computer-based Educational Measurement*, Methodology of Educational Measurement and Assessment, https://doi.org/10.1007/978-3-030-18480-3\_18

effective feedback has been identified in research (Hattie 2009). However, research has also identified the variability of feedback effectiveness (Winstone et al. 2017). An effective feedback interaction between teachers and students requires development of some common understanding of goals and quality learning, and engagement by the students in progressing their learning (Black and Wiliam 2009). Thus, the quality of feedback interactions has become a major focus of education research and teacher professional development.

This chapter reports on a study that investigated the feasibility and value of using easily accessible equipment for practitioner use and research purposes. Specifically, we aimed to generate detailed insights into the feasibility of using a GoPro camera and an Apple iPad to capture one-to-one feedback interactions between teachers and students, used for subsequent individual video-stimulated recall (VSR) for selfreflection (Lyle 2003). We were interested in the comparison between these devices for practitioner use as well as for research purposes. Our research purpose also was to explore designs whereby teachers and students would be able to collect research evidence independent of a research team presence, using technology that could be readily available in schools. The overall purpose of such research is to develop effective but simple processes for use by both teachers and students to engage with feedback to improve learning. Important outcomes of this study are insights into possibilities and limitations of using these devices, possible ideas for improvement, and potential for upscaling to whole classroom contexts.

### *18.1.1 The Value of Video Feedback*

Researchers have noted the advantages of using video for communicating feedback between teachers and students (e.g. Crook et al. 2012) and in peer feedback settings (Hung 2016; Lenters and Grant 2016). For example, Lenters and Grant (2016) noted the value of iPad video recordings as rich media for the communication of peer and teacher feedback. Teachers and students reported that use of these devices aided in smooth communication of feedback as intended, as meaning and intention often became distorted through only written feedback. One student in the study by Lenters and Grant noted the particular value of viewing body language in feedback communication as helping understand the intention behind feedback content.

The major advantage of using video in educational research is the opportunity to capture the full record of an interaction (Stigler et al. 2000), providing rich sources for reflection upon review. Reviewing video records of events has proven to be useful to support teacher self-reflection with the potential to encourage changes in teaching practices (Gröschner et al. 2015; Harlin 2014) and enhance teaching quality (Tripp and Rich 2012). The power of video lies in its possibilities to allow teachers to watch the recordings and view their feedback practices from a different perspective (Charteris and Smardon 2013; Harlin 2014; Jordan 2012; Van den Bergh et al. 2014).

Digital technology allows for video recordings to be immediately available for review and reflection. While the act of being videoed may cause teachers and students to initially behave differently, research has found that this effect fades within minutes (Rowe 2009) with the benefits of the technology outweighing this limitation. One method that has particular value for self-reflection, going beyond simply reviewing an interaction, is use of recorded video as a stimulus for video-stimulated recall (VSR) (Lyle 2003). This method is suitable to gain insight into teachers' and students' thinking and feelings in feedback interactions, as the video stimulus helps them recall their thoughts and reflect on their own and each other's behaviour and actions (Van der Kleij et al. 2017).

VSR has also proven to be useful for use in feedback research involving teachers (Van den Bergh et al. 2014) and students (Hargreaves 2012, 2013). While video technology has been used to capture teacher-student interactions and reflections, the use of video technology as stimulus for both *teacher* and *student* reflection on the same feedback interactions is less well documented. This is the focus of our research design, with our study examining the potential of both GoPro camera and iPad technologies for research suitability as well as potential independent use.

GoPro cameras are compact action cameras, designed to capture a variety of activities. The camera can be positioned in a broad range of ways, for example on a head mount, chest mount, handheld monopod or tripod. In classroom settings, this opens up possibilities to record learner actions and interactions from unique angles (Hummel 2015). iPads are multi-purpose tablet devices, which can serve many educational purposes, such as a platform for educative games and simulations (Murphy 2016). Although the use of tablet devices such as iPads is becoming increasingly common in classrooms, their uses for capturing teacher-student interactions through its video functionality are not well documented.

### **18.2 Method**

### *18.2.1 Participants and Context*

The study occurred during a six-month period across 2015 and 2016. It involved six teacher-student pairs (Pairs A–F) in Year 9 (students aged 12–13) from three Australian secondary schools, with two pairs participating in each school. All participants were volunteers. The investigation focused on feedback interactions in three subject areas in order to capture a range of potential contexts: Science, English, and Health and Physical Education (HPE). For a more detailed description of the study participants, see Van der Kleij et al. (2017).

### *18.2.2 Data Collection Instruments and Procedures*

In this study we used a GoPro Hero 4 silver, which has a display screen at the back for ease of camera positioning and reviewing of the captured material (contrary to previous models). We used an iPad Air2 (64 GB) and used the default camera application.

Following ethics approval, school principals were contacted to request participation by two teacher-student pairs in their school. Teachers were asked to select a piece of student work to be discussed in a one-to-one feedback session with the student; this feedback conversation was the focal point of this study. The research involved teachers in different disciplines and allowed for a range of assessment formats and modes. However, the assessment design had to be based on the Australian Curriculum Achievement Standards evident in a formal marking rubric or criteria overview. The researchers did not prompt the teachers or students about the importance of feedback. Written consent was obtained from school principals, teachers, students and parents/guardians.

The data collection procedures for this study included three stages. In Stage 1, video data of 10–15 min feedback interactions were collected using a GoPro and iPad simultaneously from a similar position for comparison of usability of the two devices. Two researchers were present during the feedback sessions to support the operation of the technology.

Previously, we had investigated use of the GoPro camera as a wearable item for the teacher and student, a common use for this technology. However, occupational health and safety (OHS) requirements meant the camera could not be head-mounted and would need to be chest-mounted which may still cause OHS issues. As a chestmounted camera would not capture interactions, and mindful of the OHS requirements, we investigated only the static camera. One of the teachers commented that replaying the video of a GoPro camera that was not in a static position when recording can result in motion sickness. Thus, using the GoPro in a static position appeared to be the most feasible option for this study.

Stage 2 involved VSR interviews with each student and teacher individually using the iPad. In other research using video-stimulated recall, the focal instances that participants have been asked to comment on have been identified by the researcher or the participant, or both (Rowe 2009). For example, in Hargreaves' (2012, 2013) study involving primary education students, the focal instances were identified by the researcher. In our study, a researcher asked the participant to pause the video at points they identified as relevant, and reflect on those instances. By interviewing both the teacher and student separately, we were able to directly compare their reflections on the feedback interaction, and the significance of different aspects of the feedback for each. The VSR sessions were audio recorded and took approximately 20–30 min. Video recordings and VSR sessions were transcribed for research analysis.

We requested for the feedback conversations and the VSR interviews to take place in a relatively quiet space, where no other students would be captured on video, for example, a small office, the library, or a quiet corner of a classroom. However, on some occasions this was not possible due to the organisation of a high school timetable and the types of feedback sessions that were recorded. As a result, the data were collected in a range of different settings. This, of course, reflects the reality of diverse settings for teacher-student interactions in schools.

Our initial intention was to have 50% of the participants watch the GoPro video on the iPad, and 50% watch the iPad video on the iPad. However, a preliminary outcome was that switching the GoPro Wi-Fi connections between the remote control and iPad was time-consuming, and the battery life of the GoPro appeared insufficient to do this. For this reason, all participants were shown only the videos taken by the iPad, on the iPad.

Stage 3 involved a short teacher questionnaire regarding their experiences using the GoPro and iPad for capturing the feedback conversation and self-reflection (See Appendix 1). The researchers also recorded written observations of the technical set-up as well as the dynamics of the teacher-student interaction, and took notes of any technical issues that emerged or issues identified by the teacher or student during the data collection process related to the use of the GoPro or iPad.

The positioning of the cameras differed across observations, depending on the data collection setting. Teachers and students were asked to sit as they normally would in a one-to-one feedback interaction. Video resolution of the GoPro can be modified; it was set at the same resolution as the iPad (1920 × 1080 pixels) to allow for direct comparison of the image quality. Videos were recorded on medium screen width in order to minimise the fish eye effect. We used a Wi-Fi remote to start and stop the recording. We used the rear camera on the iPad, placed diagonally behind the GoPro, as the aperture in the video function required the device to be farther back than the GoPro.

No additional microphones were used to enhance the quality of the sound as the aim was to test the default quality of these two video technologies. However, a special case was necessary so that the GoPro could be mounted on a tripod without covering the microphone. No external speakers were used to enhance the volume of the videos during the VSR sessions. The simplicity of the set-up and the minimalist equipment requirement were essential to test the ease to which these devices could be independently used by teachers for both research and pedagogical purposes within a variety of educational contexts.

### *18.2.3 Analysis*

Feasibility of the GoPro camera and iPad was evaluated based on the following criteria:

Two members of the research team operated the equipment for recording the feedback interactions. However, ideally teachers and students should be able to independently collect the video data. Ease of use was therefore an important criterion.

In order to analyse the feasibility of the two devices for capturing one-to-one feedback interactions we analysed three sources of data, taking account of the different contexts in which these data had been collected:

### **18.3 Results**

### *18.3.1 Technical Results*

The feedback interactions took place in different types of rooms that were representative of the diversity of teaching spaces that could be encountered in schools and in a research project. As a consequence, the quality of the video footage obtained differed widely across the four different settings. As the study progressed, some procedures were revised slightly as a result of learnings from data collection experiences

**Fig. 18.1** Camera set up school 1

at the previous school. Table 18.1 provides an overview of the camera set up and characteristics of the data collection instances, along with issues identified.

The quality of the visuals was satisfactory for both devices across all recordings. In School 1 (see Fig. 18.1), an issue was that the researchers' small tablet tripod did not allow for filming at an appropriate angle, therefore a stack of books of approximately 10 cm in height was used for better camera positioning. This camera angle was still not optimal in the second video with a slightly taller teacher. After data collection in School 1 a height adjustable floor stand with a 360° flexible angle for the iPad was purchased to allow for greater flexibility for data collection in Schools 2 and 3.

The flexible floor stand was valuable as it allowed for greater flexibility for camera positioning of the iPad. The GoPro tripod was easily adjustable in height and angle, and the use of the remote helped keep the camera in position.

The audio volume of the video was acceptable for most of the GoPro recordings, but it must be noted that we used a special case that does not cover the microphone. At times, the volume of the videos recorded on the iPad was too low. This was especially the case when the iPad had to be positioned farther away from the participants, and in noisy environments and large spaces.

In School 2, videoing of the first feedback interaction took place in an HPE hall with open sides. The table tennis table of the participating teacher and student (Fig. 18.2) was slightly separated by a heavy-duty curtain from the rest of the tables on which the other students played.

This was a noisy environment for videoing; as a result the sound quality for both the GoPro and iPad videos was poor, and it was not possible to transcribe all participant dialogue. However, the teacher and student were still able to use the video in the

**Table 18.1** Description of data collection: setting, positioning and identified issues

**Fig. 18.2** Camera set up School 2, Conversation C

VSR sessions, which occurred in the staffroom adjacent to the hall and a sports field. While this was also a noisy environment at times, as students were playing sports outside and other teachers were also working in the office, the quality of VSR audio was satisfactory for research transcription.

The feedback interactions and VSR interviews with the second teacher-student pair in School 2 took place in a large and mostly quiet meeting room (Fig. 18.3). However, the recording did pick up the background noises when students were moving between classes and congregating outside the meeting room.

The volume of the video recording was not quite loud enough, despite conducting the videoing in a quiet room. This may have been caused by the size of the room and the distance of the iPad to the participants when recording the video.

In School 3, the feedback interactions and VSR interviews took place in a small and narrow room with windows on both sides on the long sides of the room, which posed challenges with respect to positioning the cameras. The GoPro was positioned using its tripod on the table opposite the teacher and student. The iPad had to be in the far corner of the room, very close to the wall, in order to capture the image of

**Fig. 18.3** Camera set up School 2, Conversation D

both the teacher and student (Fig. 18.4). There was some outside noise as students moved between classes during the video recording.

Battery life of the two devices was compared during data collection at School 1. Upon commencement of data collection both the iPad and GoPro were fully charged. After recording two feedback conversations (approximately 19 min in total) the GoPro battery was 1/3 full. The use of the remote control and Wi-Fi function caused the battery to run low quickly. The iPad had retained 85% of battery after filming two feedback conversations and replaying of videos to two teachers and two students. Thus, the battery life of the iPad was noticeably better than that of the GoPro.

Teacher survey responses (Appendix 1) further complemented the technical results. The survey first asked whether teachers thought the GoPro is a suitable device to capture teacher-student interactions. Two teachers were neutral, one teacher strongly agreed, one teacher disagreed, and the other two teachers did not respond to this question. The reason one teacher strongly disagreed was based on personal experience rather than this project, and due to the fact that when the GoPro is not in a static position when recording, replaying the video can result in motion sickness. The teachers who did not respond to this question had focused their attention on using the iPad, ignoring the recording on the GoPro.

**Fig. 18.4** Camera set up School 3 (in conversation E the teacher and student positions were reversed)

Five teachers strongly agreed and one teacher agreed that the iPad is a suitable device to capture teacher-student interactions (Question 2). Four teachers strongly agreed and two teachers agreed that the length of the video observation was appropriate (Question 3). The fourth question focused on the quality of the sound when replaying the video. Three teachers strongly agreed, two teachers agreed, and one teacher disagreed that the sound quality was good. The different contexts for recording may account for these differences in response. Five teachers strongly agreed and one teacher agreed that the quality of the visuals when replaying the video was good (Question 5). None of the teachers indicated having experienced any issues while being recorded by the GoPro or iPad.

### *18.3.2 Teacher and Student Experiences*

As stated, the following data were analysed using the Winstone et al. (2017) descriptive model of feedback recipience. This model encapsulates the purpose of feedback, the interpersonal communication variables of the feedback sender and receiver, and the factors that affect recipience of feedback.

### **<IP_ADDRESS> Participant Reflections on the Experience of Being Video-Recorded**

The experience of being video-recorded differed for each of the participants. Some participants took a few minutes to relax and forget about the cameras, while others did not seem to be aware of them. In response to a survey question about issues experienced while being videoed and during the VSR, one teacher reported: "Only nerves and embarrassment! I'm sure I'll get over it with more experience" (Teacher A). Feelings of nervousness appeared to be temporary in both teachers and students. The results align with findings by Rowe (2009) that while the video camera may initially distort the normal interactions between teachers and students, it is soon forgotten even in the short space of a 10–15 min feedback session. This finding adds credibility to the use of video research data of classroom interactions.

During the VSR sessions, four teachers (A, D, E and F) reflected on the experience of being video-recorded and watching themselves back on video (see also Van der Kleij et al. 2017). For example, Teacher A said:

It was very nerve-wracking at the beginning. I didn't expect it to be … But it was. It was like, 'Whoa.' I even slowed down my talking towards the end. At first I was like, 'Blah, blah, blah, blah.' Maybe having a camera around more often will relax you more… and get over it.

As the teacher watched the video there is an evident awareness of a tendency to talk rather than listen. While the teacher attributed this response to being videoed, there is recognition of the need to slow down 'talking' to communicate feedback. In addition, later in the VSR, the teacher further noted that the student's facial expression in the video indicated confusion. It is possible that these insights may trigger a response to slow down in future feedback interactions with a student or with the class. In this instance there is a possibility that the VSR may stimulate a visual memory to slow down communication and check on receipt and understanding of feedback.

Teacher F also indicated initial nervousness about being video-recorded as he reflected on what he had learnt from watching the video:

I'm learning that I ask very long-winded questions, but I think I was a little bit sort of—it sounds silly—but nervous at the start, like just, 'Is she going to say anything?' So you're sort of trying to put it in—put it in her mouth—not put it in her mouth, but like, I don't know, instigate it maybe a bit too much. No—yeah it's—it's all right, it's good. She's an easy student to talk to, I think the manner—my manner and everything's okay, it's just weird when you see yourself doing that [laughs] … I was cringing there for the first part, to be honest.

In this instance, the teacher is cognisant of his own characteristics as feedback provider and the interpersonal variables that are critical to effective dialogue. Student F also noted that Teacher F was very nervous:

I can tell in this video Mr [teacher] is really nervous. I can tell—in the sheets he's drawing scribbles, he was, he was just drawing. I could tell he was nervous, but he also kind of like spread humorous, a humorous aura around. And that just honestly—I'm nervous as well, I don't like interviews, I just don't like it—but it kind of made me less nervous, because he's being funny, he's being himself as a teacher.

The student also identified the importance of the feedback provider's behaviour and how this affected her feedback recipience connecting "a humorous aura" to being "less nervous".

Teacher D identified an instance in the video where the experience of being videorecorded made her feel as though she was in a high-stakes situation:

I wanted it to be an accurate feedback so it would be useful for him. And doing it on the spot and with him, and with you guys filming, was quite high stakes at that moment because I was like I don't want to say the wrong thing here, I don't want to be too liberal, too conservative, I want it to be accurate.

Teacher D indicated her concern with the alignment of her feedback to the curriculum standards and ensuring the feedback was useful to the student. She focussed on her manner of feedback delivery, the characteristics of her feedback message and consideration of subsequent student engagement with her feedback in the context of being videoed.

In contrast, students appeared to be less self-conscious than teachers; only two students (Students B and F) commented on the experience of being video-recorded. Although the majority of teacher participants experienced nervousness due to the presence of the cameras and/or researchers (Teachers A, C and F), only Student F verbally expressed nervousness. However, in some cases the teacher did note nervousness in a student. For instance, in response to the question "How did you find watching the video? Did you learn anything from it?", Teacher E replied: "She was giving very good feedback but she just looked more nervous on the film than what I felt she was. Just to be more aware of that … I do feel like it went well".

When replaying the video Teacher E commented on the student's affective state, which, as she stated, she had not been aware of during the feedback session. As a result of the VSR, the teacher noted that this was an aspect of her practice that she wanted to take into consideration in future feedback interactions. The teacher's response also shows her recognition of the student as feedback provider of information that can be used by the teacher to improve her practice.

### **<IP_ADDRESS> Participant Reflections on the Usefulness of Using Video for Recording and Reflecting on Feedback**

Replaying the video and reflecting on the feedback conversations proved to have considerable educational value. See Van der Kleij et al. (2017) for a detailed analysis and overview of the nature of teacher and student VSR comments. Student reflections mainly focused on reviewing content and strategies for improvement, and reflecting on the characteristics of the feedback message and manner of feedback delivery (Winstone et al. 2017). The nature of the teacher reflections varied more broadly, focusing for instance on characteristics and behaviour of themselves as a feedback sender and the characteristics and behaviour of the student as receiver of feedback, the alignment of feedback to assessment criteria and standards, as well as characteristics of the feedback message and manner of feedback delivery.

The teachers' and students' comments on participation in the video-recorded feedback interaction identified their ability to critique their own performance and reflect more deeply on the others' behaviours. Teachers expressed having gained insights from watching the video about their feedback practice, for example, needing to emphasise the positive, double check for student understanding, and provide time for students to identify quality elements and areas of concern within their own work. For example, Teacher A noted that her feedback was highly corrective in nature, and did not sufficiently recognise the student's strengths. Teachers B, D and F identified instances where they gave the student feedback or asked a question but did not give the student a chance to respond. Teacher C also noted that he might have spoken too much:

It's nice to sit back and see patterns develop, from an outside view. I think there were a couple of times there where I might have spoken too much instead of like let's play roll on. But at the same time it was good to see where I jumped in and offered feedback … it's amazing to see how much change can happen when you isolate someone for ten minutes.

Teacher C has connected the importance of the context, in this case opportunity to engage in one-to-one feedback, even if only for a short (10-min) period of time and the proactive recipience of feedback to cause changes in student behaviour and learning.

Through watching the video, teachers were able to identify the student's emotions in the feedback interaction thus raising their awareness of the affective impact of their feedback. For example, Teacher A observed embarrassment in Student A, and reflected: "I wonder why she looks embarrassed. I'll ask her why did she look embarrassed—because I think because we pored over and over and over it and she's like, 'Oh yeah, I know that.' Or is it just because she didn't realise?" In this interaction, Student A did not contribute much beyond a single response as acknowledgement of the feedback (e.g. "yeah") while Teacher A provided extensive feedback on all aspects of the student's assessment response. Teacher A's VSR reflection suggests an emerging awareness that feedback involves interpersonal communication rather than a one-way transmission of information.

For students, the degree of self-reflection in VSR responses appeared to be related to the way in which the teacher had structured the feedback context (see Van der Kleij et al. 2017). Most of the students identified that they valued reviewing the feedback as it gave them the opportunity to review key points that were missed in quite densely informative conversations. For instance, Student A noted that there was a lot of information to take in during the feedback session, and reviewing the video was helpful. Student D identified that replaying the video was helpful: "I could have missed something while I was listening to her". The students' comments showed an awareness of the manner of feedback delivery (e.g. the quantity of feedback), the characteristics of the message (e.g. the density of information), and the interaction with their own characteristics as feedback receivers. They acknowledged that processing and acting on feedback was important but that they required further time and opportunity to revisit feedback.

Although the research involved only one-on-one feedback interactions, the teachers reported the usefulness of these conversations for their teaching as well as student learning. When asked whether she had found the experience useful, Teacher D replied: "Yes, terrifying [laughing] but very useful". In the survey, four teachers commented on the usefulness of the experience when asked if they had any other comments. Teacher A noted: "I think it's really worthwhile and am going to try recording my feedback again!" Teacher A recognised the value of the VSR for her reflective practice and the potential for enhancing her feedback practice. Teacher F wrote: "I thought the process was worthwhile and contained elements I could use in my professional practice". Teacher F related her learning from the VSR to her classroom teaching practices more broadly. The teachers' comments indicate that the VSR process supported their reflective practices that moved beyond the one-on-one feedback context to consideration of whole class interactions.

### *18.3.3 Researcher Experiences and Reflections*

Overall, both researchers involved in the video process noted use of the technologies to be smooth and relatively problem free. However, there were specific aspects that required attention dependent on the different locations. When setting up the iPad it is important to take into account the aperture, as the image automatically zooms in when switching to video mode. For this reason, the iPad was positioned farther back than the GoPro. However, this sometimes resulted in low volume in the video recordings. Another important note was that the iPad automatically reverts to camera mode, so it needs to be set on video mode when setting up the camera, and needs to be reset on video mode shortly before recording commences.

In this study the teachers were requested to have 10–15 min feedback conversations. The actual length of the conversations varied between 6 and 18 min with an average 12 min 46 s. Our observation was that in some cases the length of the feedback session and subsequent VSR session was too long for students to review. In the lengthier feedback sessions, there was often too much information for students to remember and then act on; after watching the video, regardless of length, the students recalled no more than three main points from the feedback interaction to address in their work.

Use of the technology was observed to cause only minor technical difficulties for the participants. The video could be started by tapping a play button in the middle of the screen, but had to be paused by tapping a small pause button in the right top corner of the screen. While none of the participants required explanations on how to play and pause the video on the iPad in the VSR sessions, in a number of instances the participants accidentally touched the screen in a place that caused the video to replay from the beginning. This was a minor issue as the iPad did not show the time elapsed when replaying the video, and it then took time to retrieve the right point of play in the video to resume the review of the feedback conversation.

Another technical difficulty relevant for research involved time stamping the instances when the teacher and students paused the video to comment on the feedback interaction, as the time elapsed did not show in the video player (this feature has become available in more recent updates, but the time still only displays for about a second). It was possible to hear the video in the background of the interview, but time-stamping the video pauses was a time-consuming activity involving two researchers to check the accuracy of this record.

### **18.4 Discussion**

This study trialled the technical and experiential feasibility of using a GoPro camera and iPad to capture one-to-one feedback interactions between teachers and students for reflection on feedback interactions, to inform the design of future research projects. Both technologies are easy to access, and iPads, in particular, with a broader range of educational uses are common in schools. The iPad, as a multifeatured device, presented several advantages as well as disadvantages. The advantages included ease of replaying video, screen size for replaying video, ease of sharing or transferring videos, battery life and familiarity to teachers and students. The main disadvantage of the iPad was that the distance required for videoing resulted in low audio quality. This could be addressed by using either a fish eye lens to enable closer camera positioning, or by using an additional microphone to capture audio. Another potential disadvantage of the iPad, although it did not occur during our data collection, could be the video recording stopping during times when connected to the Internet and a phone call occurs. The solution to this would be to have the iPad on airplane mode when recording videos. On the other hand, limitations of the GoPro as standalone technology for this type of research mainly related to the small size of the screen to replay the video. While the GoPro camera was compact and easy to mount, this form of research required a tablet or PC to facilitate VSR processes. As a practical consideration, the GoPro required spare batteries as the recording used much charge.

One critical feature for our research purposes was to know when the teacher and student paused the video in order to compare the different instances they identified as critical in the feedback conversation. However, the elapsed time did not show in the iPad's video player, so while records were made of key words spoken at the time, we had to listen carefully to the feedback conversation in the background of the interview recording to identify these times. In future research we recommend that researchers take note of video pause times.

It has been reported in previous research using VSR that some teacher participants were not willing to watch their video recordings in discussion with the researcher, because they were not satisfied with their behaviour (Rowe 2009). This was not the case in our study, although some teachers and students appeared initially uncomfortable watching themselves on video. However, their interest in reviewing the feedback conversations, especially for the teachers, appeared to overcome this reaction. Reviewing video evidence enabled students and teachers to observe reactions through facial features and body language. The visual evidence was seen to provide new information for reflection. During feedback conversations both teacher and student eyes were mostly focused on the student work or scoring rubric. Through use of VSR, the participants were able to focus on the social interaction.

The teachers' comments revealed their attention to multiple dimensions of the feedback conversation including characteristics of the feedback message, the manner of feedback delivery, their own and student characteristics and behaviour and the interaction of these variables (Winstone et al. 2017). The VSR allowed teachers and students to review both verbal and physical responses to the feedback. The teachers were critical of their own behaviour as feedback provider, especially when they identified that they had monopolised the conversation rather than allowing more time for student response, or when the characteristics of the feedback message mainly focused on correcting mistakes rather than also identifying strengths in the student's work. Replaying the video also enabled the teachers to more clearly identify the student's emotional reaction in the feedback interaction. Teachers' survey responses indicated that use of video technology independently to capture teacher-student interactions had potential to improve these interactions and enhance their feedback practice in general, because of an increased awareness of their current feedback practices. To a lesser degree, but still significant, most of the students articulated that they valued reviewing the feedback as it gave them the opportunity to hear again key points that were missed in quite densely informative conversations.

The teachers in the study commented that this form of one-to-one feedback was not common practice, though at the conclusion of the feedback session, they noted the value for student learning, and for their feedback and classroom teaching practice. Since the VSR was not difficult to set up for one-to-one feedback sessions, it would add minimal time to the feedback interaction, especially if the teacher was conducting organised conferencing sessions. The issue in this case is not related to the technology but rather the purposeful scheduling of time. Based on our findings, we hypothesise that the methods used in this study can be replicated by teachers to enhance classroom feedback practices. Engaging in VSR has the potential to enhance proactive recipience of feedback (Winstone et al. 2017) by making both teachers and students aware of their verbal and physical contributions and responses to the feedback interaction.

Extending the research from teacher-student interactions to capturing whole classroom interactions would require additional technologies. Previous video research on formative assessment in a whole classroom situation (Gotwals et al. 2015) used a camera at the back of the classroom and a cordless microphone worn by the teacher. In this type of set up the researchers reported not always being able to capture students' voices in student-student or whole class interactions when the teacher was not nearby. While the GoPro camera can be positioned at different angles, a similar issue identified by Gotwals et al. (2015) may be the inability of the GoPro microphone to pick up student conversations. Similarly, additional technology, for example, a SwivlTM with several wireless markers with microphones, is required for the iPad to capture voices accurately when not in proximity to the camera. When iPads are positioned near students to capture conversations, the use of a fish eye lens for the iPad will enable participant images to be captured on screen.

### **18.5 Conclusion**

This study trialled two technologies that are easy for schools to access regarding the quality of the sound and image and ease of use including battery life, in a range of educational contexts. The focus was teacher-student feedback interactions for the purpose of improving learning. The findings showed that while the iPad and the GoPro camera were both useful for capturing the feedback interactions, the iPad was more useful for video-stimulated recall interviews. With schools increasingly purchasing this type of technology, this finding suggests that this technology has potential for independent use by teachers and students for reflection to improve feedback practices. Through the use of these simple technologies, our project has been able to identify the relationship between teacher and student perceptions of feedback conversations. Most importantly, for our research purposes, it has potential for optimal data collection by enabling larger research data sets to be collected in a range of locations to illuminate the 'black box' (Black and Wiliam 1998) of classroom interactions and assist improvement of teaching and student learning. Additional technology is needed to upscale the use of the two devices to capture and facilitate reflection on whole-classroom interactions, but overall iPads seem more suitable for this type of use.

### **Appendix 1. Teacher questionnaire**

To what extent do you agree or disagree with the following statements?

### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Chapter 19 Game-Based Spoken Interaction Assessment in Special Need Children**

**Jos Keuning, Sanneke Schouwstra, Femke Scheltinga and Marleen van der Lubbe**

**Abstract** The purpose of the present study was to explore whether it is possible to collect high-quality data about children's spoken interaction skills using the *Fischerspiel* board game as an entertaining, non-threatening means to evoke conversations between children in special elementary education. The game was administered to a total of 681 eleven- and twelve-year-old children with varying educational needs. The quality of the conversations between the children was evaluated with a specially designed observation form. The observation forms were filled in by trained test leaders and four independent expert raters. Video recordings showed that almost all children were willing to participate in the game, even the children who usually barely speak in class. Moreover, the game provided more than sufficient data to assess different dimensions of spoken interaction skills. Analyses further showed that the observation form functioned well and provided reliable scores. A group effect was nevertheless present and two test leaders deviated largely from the expert raters. These test leaders may have been insufficiently equipped to cope with the task. Application of Automatic Speech Recognition (ASR) technology in a (computer-based) spoken interaction assessment might ease the task and increase rating quality.

### **19.1 Introduction**

Oral language performance is a good predictor of various school outcomes such as mathematics (Chow and Jacobs 2016; Fuchs et al. 2005) and reading and writing (Dickinson et al. 2010; Hart and Risley 2003; Kent et al. 2014; Nation and Snowling 2004; Dougherty 2014). Deficits in oral language skills may also underlie difficulties

F. Scheltinga University of Amsterdam, Amsterdam, The Netherlands

M. van der Lubbe Inspectie van het Onderwijs, Utrecht, The Netherlands

© The Author(s) 2019 B. P. Veldkamp and C. Sluijter (eds.), *Theoretical and Practical Advances in Computer-based Educational Measurement*, Methodology of Educational Measurement and Assessment, https://doi.org/10.1007/978-3-030-18480-3\_19 361

J. Keuning (B) · S. Schouwstra Cito, Arnhem, The Netherlands e-mail<EMAIL_ADDRESS>
in text comprehension and writing (Shanahan 2006; Naucler and Magnusson 2002). At school, language is necessary to understand instruction, to communicate about content and to demonstrate understanding. Moreover, language ability is related to social behavior and poor language skills affect social behavior negatively (Chow and Wehby 2018). Within the educational context, oral language is generally referred to as listening and speaking. Listening is as a receptive skill: it is the process of recognizing words and understanding messages expressed by others. Speaking is a productive skill: it is the process of transmitting ideas and information orally in a variety of situations. Both skills are used alternately in spoken interaction. Conversation partners must not only be able to convey and understand the content, but in addition, they must also be able to manage, produce and interpret verbal and nonverbal communication. While there is much experience in monitoring the listening and speaking skills of children, it is less clear how to monitor the children's spoken interaction skills. The present study focuses on an even more unexplored area: spoken interaction assessment in special need children.

### *19.1.1 Measuring Spoken Interaction Skills*

Spoken interaction includes three content domains, namely Language form, Language content and Language usage (Bloom and Lahey 1978; Lahey 1988). Language form refers to the grammar of language, that is, to the phonological, morphological and syntactic rules that determine how sounds, words and sentences are formed, respectively. Language content refers to the ability to make up an own (non-)fictional story with a clear line of thought. Finally, in Language usage it is about the understanding and use of communicative functions and conversation rules such as expressing and interpreting emotion, making conversation, maintaining a topic of conversation, taking turns or asking others for information (McLaughlin 1998). When measuring spoken interaction skills it is important that all three content domains are covered. In addition, O'Malley and Pierce (1996) state that a spoken interaction assessment should: (a) test a child's competence as authentically as possible, (b) include an evidence model which shows how responses are scored, analyzed and interpreted, (c) take a limited amount of time to administer. In practice, it is difficult to meet all aforementioned criteria at the same time. Moreover, an extra complicating factor in assessing spoken interaction is the influence of the context and conversation partner. An interplay does exist between individual language ability and contextual factors (Chapelle 1998). Furthermore, the course of a conversation is determined in interaction and co-construction (Kramsch 1986).

Taylor and Galazci (2011) suggested to control context and to use detailed scripts to challenge children at their own level in a meaningful and standardized manner. These suggestions were implemented in a recent large-scale assessment in the Netherlands by having three children jointly conduct a conversation assignment (Van Langen et al. 2017). The assignment was conducted in the context of a national charity action: the children had to make up a goal and activity with which they could

#### **Table 19.1** Spoken interaction assignments for one-to-one settings

outcomes are a bit uncertain

#### 2. Role-play

− The performances in mainstream and special need education are often similar; this is not what is expected, validity might be an issue

4. Referential communication task

raise money. In order to finance the preparation of their idea they were awarded a (fictitious) amount of 50 euro. The children had to agree on the project planning and the corresponding financial framework. The topics to be discussed were shown on a paper for the children as a guide during the assignment. In addition, each child received his own conversation guide as input for the conversation. The conversation guide contained information that was specifically intended for that child. For example, each child was expected to have his or her own contribution at various points in the conversation. The conversation guides encouraged each child to actively contribute to the conversation at minimally two moments. Although not every child received information on all discussion topics, they always did have the opportunity to contribute. The child could, for example, give an opinion, give a reaction to the opinion of others, join this opinion or make a compromise. In this manner, the conversation could proceed as naturally as possible.

The charity assignment functioned well when administered to mainstream elevenand twelve-year-old children (see Van Langen et al. 2017), but it is unlikely that the assignment is also appropriate for special need children. According to special education teachers, the context is, for instance, too abstract for children with lower cognitive abilities. Many other spoken interaction assignments are possible, all with specific advantages and disadvantages. Tables 19.1, 19.2 and 19.3 show which assignments can be administered in a one-to-one setting, a small-group setting or a classroom setting, respectively.

**Table 19.2** Spoken interaction assignments for small-group settings

1. Group storytelling (e.g. about the weekend)

+ Alignment on conversation partners and Taking turns are covered dependent on the role of the child in the group

− Some children may contribute a lot while others do not; the group affects the chance an individual child gets to show what he or she can do

− Administration and scoring are not standardized

2. Group assignment (e.g. development of a project plan)

+ Alignment on conversation partners and Taking turns are covered dependent on the role of the child in the group

− Some children may contribute a lot while others do not; the group affects the chance an individual child gets to show what he or she can do

**Table 19.3** Spoken interaction assignments for classroom settings

1. Lecture or book discussion

+ The child can show extensively what he or she can do

− Alignment on conversation partners and Taking turns are not well covered

− Administration not standardized

2. Observation in a natural situation such as a circle discussion

+ A familiar situation for the children

− Administration and scoring are not standardized

− Some children may contribute a lot while others do not

− Not all children may feel safe in a circle discussion

### *19.1.2 Assessment in Special Elementary Education*