Here, we adopted a one tailed, two-sample t-test for significance test and set
the significance level to 0.05.

Figure 3.16: The improvement in the number of unique key-terms contained in
submitted essays when $linking$ interface is used. Learning performance is
measured in the study of Stat2.1x. The 95% confidence intervals and
significance test results are also provided.

The first row of Table 3.6 and the first bar in Fig. 3.16 show that, overall,
subjects are able to mention a greater number (12%) of key-terms when using
the $linking$ interface (cf. 4.39 $vs$. 4.91), and the difference is
statistically significant. Looking over the rest of Table 3.6 and Fig. 3.16,
we observe that there is a similar trend to that in the information search
scenario, where the $linking$ interface yields improvement over each cohort of
subjects, and in four out of the six cases (i.e., subjects with no prior
knowledge in statistics, without prior exposure to MOOCs, and with/without a
bachelor’s degree or higher) the differences pass the significance test.
Furthermore, it seems that the novices also benefit more from linking than
advanced learners (e.g., all the three naïve cohorts show statistically
significant improvement).

These results reveal another aspect of benefit linking may provide. In the
course materials, there are usually many learning pieces relevant to a topic;
some of them are complementary and some might be redundant. With the
visualized inter-material relation, complementary content can be identified
easily and thus be better utilized to reinforce learning. For example, while
watching the lecture video, subjects can refer to the aligned slides to
understand the lecture at the concept level, as well as to the linked textbook
sections or posts for detailed discussions. The identical learning content can
also be skipped easily. Furthermore, the visualization helps learners better
plan their learning path within the limited-length learning session, and avoid
exploring irrelevant or secondary content to the assigned topics. These
features made possible by visualizing linking can also be interpreted as the
guidance which leads learners navigating through learning content when
accomplishing assigned tasks. Therefore, subjects, especially novices, can
access knowledge more efficiently in the learning session, and retain more
key-terms when they write down what they can remember. In section 3.5 we will
provide more evidence to our claim here.

We also investigated whether this aspect of benefit can be generalized to
various course subjects in a more realistic condition. Hence, similar to the
information search scenario, we further studied a different MOOC, 6.00x, as
well as explored in an expanded material set (i.e., forum discussions were
additionally used) and data annotation pipeline (i.e., teaching assistants
were recruited as annotators).

Table 3.7 summarizes learner performances in the concept retention scenario in
the study of 6.00x. Similarly, performance is evaluated by the average number
of unique key-terms in the submitted essays, and measured within various
cohorts (rows 1 to 7) using different interfaces (column 1: $null$; column 2:
$linking$). In this study, the same quality control mechanism using plagiarism
checking is employed to filter out the noise from spammers. Furthermore, to
focus on the benefit brought by linking, in Fig. 3.17 we plot the performance
difference when various interfaces were deployed (i.e., number of key-terms
when $linking$ is assigned subtracted by the number when the $null$ interface
is used). In addition to the differences, the 95% confidence intervals (the
error bars) as well as the significant test result (marked with red asterisk
if significant) are also indicated in the figure.

Table 3.7: Learner performance in the concept retention scenario in the study of 6.00x. Similar to the study of Stat2.1x, performance is evaluated by the number of unique key-terms in submitted essays and measured within various cohorts using different interfaces. | Number of unique key-terms
---|---
null | linking
Overall | 8.07 | 8.56
Python | Yes | 8.64 | 9.09
No | 7.64 | 8.20
MOOCs | Yes | 8.37 | 8.55
No | 7.93 | 8.56
$\geq$Bachelor | Yes | 8.60 | 9.13
No | 7.21 | 7.91
Figure 3.17: The improvement in the number of unique key-terms when the
$linking$ interface is deployed. Learning performance is measured in the study
of 6.00x. The 95% confidence intervals and significance test results are also
provided.

From Table 3.7 and Fig. 3.17, first we can observe that when the $linking$
interface was deployed, subjects in each cohort mention a greater number of
key-terms. This observation is identical to the results in Stat2.1x. However,
we can find that the improvement is statistically significant in fewer cohorts
(i.e., in the entire group of subjects, subjects without prior exposure to
MOOCs, and subjects without a bachelor’s degree). We might be observing fewer
cohorts showing significant improvement because of the advanced topics used in
this user study. As discussed above, our stratification of learners, which is
inherited from the study of Stat2.1x, might not be able to tell novices from
advanced learners. Therefore learners in each cohort are too diverse to
perform consistently. This claim is supported by the much larger standard
deviations (i.e., longer error bars) observed in Fig. 3.17 than the ones in
Fig. 3.16.

These results also strengthen our previous claim that, when linking is shown
in the interface, learners can access information more efficiently and retain
more key-terms in their summary of assigned topics. The improvement not only
suggests another benefit of linking in learning, but also implies the
possibility of applying our framework in various course subjects. However,
these observations only support our hypothesis that linking is helpful in
learning, but cannot explain why. Without knowing the reason, we cannot
utilize this linking pedagogy in suitable conditions. Thus, in the next
section, we analyze the learners’ click log in order to provide an
explanation.

###  3.5 Click log analysis

From the user study results discussed above, we can summarize that, by
presenting the linking among learning materials, learners can access course
materials more efficiently and perform better in learning tasks. However, we
are also curious to discover how the linking changes learners’ navigation
behavior and why the change yields improvement in accomplishing our learning
tasks. Therefore, we examine the click log333In our study of 6.00x, in
addition to the submitted answers (i.e., the selected learning object or the
essay summarizing the assigned topic), we also record how learners interact
with our interfaces. Whenever a learner initiates an event to search a query
or click on any learning object, the event along with the triggered time is
stored in our server. generated when learners attempted the tasks. We utilize
the log to extract the following three metrics:

  * •

Number of search queries used in each learning task.

  * •

Number of learning objects444Note that because of the way we recorded the
event, here the definition of a learning object is slightly different from the
rest of this thesis. A learning object in this section refers to a lecture
video, a page of lecture slides, a textbook section, or a discussion thread.
surveyed in each task.

  * •

Time spent (measured in seconds) in each surveyed learning object.

In each learning scenario (i.e., search and retention), we compute the three
metrics averaging over the tasks using each of the two interfaces (i.e.,
$null$ and $linking$). The results are summarized in Table 3.8.

Table 3.8: In information search and concept retention scenario of the study of 6.00x, the three metrics (number of search queries used to accomplish a task, number of learning objects surveyed in each task, and the spent time in each learning object) when various interfaces were deployed are computed. The averages ($\mu$) and standard deviations ($\sigma$) of the three metrics are listed here. | Information search | Concept retention
---|---|---
null | linking | null | linking
#Search queries ($\mu$, $\sigma$) | (2.9, 2.9) | (2.7, 2.8) | (1.6, 1.2) | (1.4, 0.9)
#Learning objects ($\mu$, $\sigma$) | (10.9, 9.7) | (7.7, 7.5) | (11.8, 8.7) | (7.8, 6.6)
Spent time per object ($\mu$, $\sigma$) | (32.3, 73.4) | (35.1, 64.9) | (46.0, 88.4) | (70.6, 115.8)

Comparing first the numbers in the two scenarios, we observe that in the
information search scenario, learners tend to use more search queries, survey
more learning objects, and spend less time on each object. Considering the
average time learners have to spend in interacting with the interfaces in the
two scenarios (6.6 minutes in information search in average and 10 minutes in
concept retention), the difference in numbers of queries and learning objects
between the two scenarios is even larger. We surmise that the discrepancy
results from the nature of the two scenarios. In the search scenario, learners
have to only identify the objects which contain information for answering the
assigned problems; however, they need to decide which search queries to use.
As for the retention scenario, learners have to digest and remember
information in the content, but it is obvious that they should use the
assigned topics or relevant terms as the queries. Thus, in the search
scenario, learners are inclined to survey more queries and learning objects,
but spend less time on each of the queries or objects.

Then we juxtapose the metrics of each interface within the two scenarios. We
find that, as compared to using the $null$ interface, when $linking$ is
deployed, experimental subjects tend to use fewer search queries (information
search: 2.7 $vs$. 2.9, concept retention: 1.4 $vs$. 1.6), survey fewer objects
(information search: 7.7 $vs$. 10.9, concept retention: 7.8 $vs$. 11.8), but
spend more time on each object (information search: 35.1 $vs$. 32.3, concept
retention: 70.6 $vs$. 46.0). We believe this observation explains our user
study results. The observation suggests that, when linking is visualized,
learners are able to identify those learning contents which are more
informative for the assigned topics or problems; in contrast, when $null$ is
deployed, learners have to try more queries and learning objects. Thus, with
our $linking$ interface, learners can spend more time in understanding
relevant information for the search or retention tasks, and achieve better
performance. The observation here and the user study results can also be
related by reduced cognitive load, which has a positive effect on learning
[70]. When linking is presented, it is easier for learners to filter out less
useful learning objects, which alleviates learners’ cognitive load in
understanding the materials.

To support our conjecture, we further investigate how relevant the learning
objects learners surveyed are to their assigned tasks. In order to measure the
relevance, we utilize the labeled valid learning pieces which were used in
evaluating whether the selected learning content is correct in the information
search scenario. With the click log recorded in this search scenario, we
measure the percentage of surveyed learning objects which contain at least one
valid learning piece for the assigned problem.

The mean and standard deviation of percentage is 0.33 and 0.26 for tasks using
the $null$ interface, and 0.50 and 0.33 for tasks using the $linking$ one. We
can observe that when $linking$ is deployed, learners tend to survey more
objects containing valid learning pieces. This fact supports our previous
claim that with the relation among learning materials presented, learners are
able to filter out learning objects which are less likely to contain useful
content, and focus on informative objects. Thus, better learning outcomes are
achieved.

To further illustrate how linking can help learners identify useful
information, we visualize two sampled search paths recorded from two subjects
when they were completing assigned tasks using different interfaces. We
sampled the two paths based on the following criteria. First, in the paths,
subjects surveyed the same number of learning objects as the interface average
(i.e., 10 objects for the path recorded in the task using the $null$ interface
and 7 objects for the $linking$ one). Second, learners surveyed the same
number of objects that contain at least one valid learning piece as the
interface average (i.e., 3 informative objects for both interfaces). Third,
the two paths were recorded from tasks assigned with the same question.

In Fig. 3.18, we show the question the two sampled search paths corresponded
to. This question is about the normal distribution. In Fig. 3.19 and 3.20, two
search paths observed in tasks using the $null$ and $linking$ interface
respectively are presented. In these paths, screenshots and titles of surveyed
learning objects are listed according to the visited order; the material type
of each object is also indicated. Furthermore, the titles of objects
containing valid learning pieces are put in red; the titles of other objects
are in cyan.

Figure 3.18: The question asked in the tasks where we recorded the two sampled
search paths. Figure 3.19: The sampled search path recorded when a subject
used the $null$ interface to complete the assigned task. In this path this
subject surveyed 10 objects and 3 of them contain valid learning pieces. The 3
objects are indicated with their titles in red; the titles of the rest objects
are put in cyan. These numbers equal the average of the $null$ interface.
Figure 3.20: The sampled search path recorded when a subject used the
$linking$ interface to complete the same task as in Fig. 3.19. In this path
this learner surveyed 7 objects and 3 of them contain valid learning pieces.
These numbers equal the average of the $linking$ interface.

From these two paths, we first notice that the learner who was assigned with
the $linking$ interface has significantly better survey quality. Most objects
this learner selected are relevant to the topic of the normal distribution. In
contrast, the search path of the learner using the $null$ interface seems
unplanned. This observation agrees with our previous claim that learners can
benefit from linking since they are able to more easily filter out less
informative materials and focus on the useful ones. Furthermore, we note that
the learner using the $linking$ interface switched between different types of
materials more frequently. This fact also strengthens our assertion that when
linking is presented, learners are capable of utilizing complementary
information from various types of materials to reinforce their learning.

In this section, we analyzed the click log recorded in the user study of
6.00x, in order to explain why linking yields better learning performance in
our experiment. We found that, when the linking among materials is presented
to learners, they survey fewer queries and objects as well as spending more
time in each object. Moreover, the surveyed object is more informative and
therefore learners can perform better in tasks. However, we should note that
the differences in metrics computed here are not statistically significant.
Hence, further stratification of samples is required to obtain stronger
evidence.

###  3.6 Conclusions

This chapter explores our first research question: can manually generated
linking help learning? We start by defining two types of linking: homologous
and heterologous linking. We formulate the annotation of these two linking
types as alignment and binary classification problems respectively, and
demonstrate how the annotation can be done by researchers, online workers, and
course staff. Then, we implement a $linking$ interface that can present
learning materials and the annotated linking among them simultaneously. After
that, we conduct a large-scale user research study with the two selected STEM
courses, statistics and programming language, to investigate the question.

Our user research shows that, this $linking$ interface enables learners to
search for desired learning content more efficiently and retain more concepts
more readily. By analyzing the click log recorded when learners using various
interfaces in the user research, we observe that, presenting both content and
linking at the same time helps learners focus on informative learning
materials, and thus potentially reduces learners’ cognitive load. These
results support the notion that manual linking can indeed improve learning
outcomes.

##  Chapter 4 Can we link automatically?

In this chapter, we investigate methods to link courseware automatically. We
showed in the previous chapter that linking can assist learners navigating
through course materials, and help learners find supportive learning content
when they are in need, e.g., confused. The visualization of relation among
content also provides guidance to mitigate learners’ cognitive load. Hence,
learning experience and outcome can be enhanced.

However, from our experience in developing a linking system, annotating
relation information requires deep and comprehensive understanding of the
course subject, and the labeling itself is time-consuming. Moreover, even
though the linking is generated for the current class, obtaining the relation
annotation for future offerings requires a lot of redundant work - since in a
new class offering, more forum posts come in (but many of them are
duplicated), and several lectures may be re-organized, we have to maintain and
repeat labeling on the updated materials. Thus, implementing the educational
content linking framework manually is cost-prohibitive, not efficient, and not
scalable. In order to allow the proposed framework to be more easily deployed
and in a more general condition, especially when the size of learning
materials is enormous, we investigate whether linking of learning content can
be generated by machines.

In the following investigation of automated linking methods, we focus on three
issues:

  * •

How to design an automated linking method?

  * •

How closely can the automatically generated linking approach to human
annotation?

  * •

Can automated linking still benefit learners?

For the first issue, we will discuss how to formulate linking generation as a
prediction/classification problem, as well as how to solve the problem with
machine learning and human language technologies. For the second issue, we
will compare machine generated linking with ground truth established in
advance by a human (i.e., the linking we collected in the previous chapter).
The comparison can be made easily and shed light on the capability of
implemented automated methods. However, such evaluation is not perfectly
precise, since it measures the similarity between two generated linkings,
instead of our ultimate goal: learning outcome improvement. Furthermore,
typically there are many configurations of linking which might benefit
learners. Computing the similarity to a gold standard is a biased metric which
ignores all the other beneficial possibilities. Therefore, we also explore the
third issue by conducting a user study on the automated linking, with a
pipeline similar to that designed in the previous chapter. Although conducting
a user study is more costly and time consuming, it evaluates directly the
benefits of machine generated linking on learners.

We explore the three issues on the same two MOOCs: Stat2.1x and 6.00x. In our
experiment, we find that, although there are some differences between the
machine and human generated linking, they only result in slight negative
effects on learning. Moreover, the interface driven by automatic linking
(denoted as the auto linking interface in the following) still allows learners
to achieve better performance in their tasks. Furthermore, we analyze the
difference pattern, and conclude that anecdotally most disagreements between
human annotators and our contributed linking algorithm do not make much
difference to learners. In the rest of this chapter, we describe our
exploration in detail.

The study discussed above was conducted by comparing an interface presenting
linking information to a baseline that implements the conventional strategy
for delivering learning materials online (i.e., the null interface). Since
MOOC is an emerging research field in education, researchers and educators
have constantly been integrating innovative pedagogies into the design of
MOOC. We are curious how effective our proposed linking framework is as
compared to these state-of-the-art techniques. Therefore, in this chapter we
conduct another user study to compare our linking/auto linking interface to
the interface currently deployed on edX (denoted as the edx interface). Our
results show that learners still achieve better performance in the explored
learning tasks with our manually and automatically generated linking.

###  4.1 Problem formulation

To design our automated algorithm, we choose to focus on the natural language
content in course materials and formulate the linking generation task as a
sequential tagging problem. Natural language is an integral part of education
for knowledge transferal; thus we believe an algorithm based on the
understanding of natural language in learning content can be more
generalizable to different courses. The sequential tagging formulation is as
follows: for nodes on the trunk of a linking tree, we view them as a sequence
of documents, and represent them as a sequence of input feature vectors
$\textbf{x}=(\textbf{x}_{1},\textbf{x}_{2},...,\textbf{x}_{\mathrm{T}})$.
Determining which supplementary objects should be linked to these nodes can be
interpreted as predicting a sequence of labels
$\textbf{y}=(Y_{1},Y_{2},...,Y_{\mathrm{T}})$ given x. Here $Y_{i}$ represents
the linking configuration of trunk node $i$, e.g., the index of an aligned
slide or whether a given discussion thread is linked to node $i$ or not.

We adopt the sequential tagging formulation, because the ordering and context
is informative when modeling learning materials. Many theories in cognitive
science of learning suggest that, to achieve meaningful learning111The state
where the newly acquired knowledge is fully understood and ready for future
used in different circumstances., humans have to match the information they
learn to how their minds are structured, and integrate the new information
with their prior knowledge and existing cognitive system [51, 6]. Guided by
these theories, learning content is typically structured in a sequentially
dependent manner to help students acquire knowledge. Thus, we surmise that the
contextual dependency can be helpful in linking prediction, e.g., it could be
more probable that neighboring video segments are linked to similar or
identical sets of supplementary learning objects.

There are many other applications of this sequential tagging formulation in
the natural language understanding domain, including part-of-speech (POS)
tagging [112], semantic tagging [75], and machine translation [9]. These
applications work on the granularity of words (i.e., each token is a word) and
attempt to interpret the meaning of each token. Since natural language is
usually interpreted in sequence by a human, modeling the context is also
beneficial in understanding the syntax and semantics. In our formulation, a
learning object, which can be a video segment, a slide, a textbook section, or
a post thread, is adopted as a token, and we model the contextual and lexical
dependency upon this larger unit with similar formulation.

Due to the abundance of applications, many machine learning models were
proposed to solve the problems, including linear-chain CRF, Hidden Markov
Models (HMM), general graphical models, and long short-term memory (LSTM)
recurrent neural networks [56]. In this thesis, we also adopt linear-chain CRF
for our linking problem; HMM can be interpreted as a linear-chain CRF with
generative modeling222Linear-chain CRF is a discriminative model.; as compared
to linear-chain CRF or HMM. A general graphical model removes the limitation
of the Markov property and thus has higher complexity333Higher complexity
means that the underlying model has more free variables and more ability to
represent dependency in data.; LSTM is a model with even higher complexity and
non-linearity, and it is widely used to express complicated data dependencies.

We choose linear-chain CRF for several reasons. First, the size of our corpus
is relatively small as compared to many other natural language applications.
This is due to the difficulty of data annotation: labeling the relations among
learning objects requires deeper understanding of the content as compared to
annotating the POS tags or semantics of each word. Our corpus size cannot
afford the training of complicated models such as LSTM or general graphical
models because of the hazard of overfitting. Second, as compared to non-linear
models such as LSTM, a linear model makes it easier to understand the results.
For instance, each weight in the model can be directly interpreted as the
importance of a corresponding feature. The interpretation can facilitate
subsequent system development. Third, a discriminative model is preferred in
our tasks, since it allows us to incorporate new features into the model
without making unnecessary assumptions about the underlying probabilistic
distribution444A generative model such as HMM is based on a full probabilistic
model. It has to model the probabilistic distribution of all variables,
including the observed (i.e., the features or observations) and unobserved
(i.e., the labels) ones. The distribution can be learned from a corpus from
scratch but the learning usually requires too many data samples. We can also
make assumptions to initialize or limit the distributions to a specific form
for the learning to be feasible (especially when we do not have enough data).
However, making assumptions is error-prone. In contrast, a discriminative
model only models the dependency between observations and the unobserved
variables that should be inferred. In such a model we do not have to learn the
entire distribution or make unnecessary assumptions. Therefore, it is much
easier to augment a discriminative model with new features (i.e., introducing
new variables and dependencies into the model).. This property allows the
automated algorithm to be extended easily to various features, which is
favorable because there are usually many information modalities in course
materials. In the following, we will discuss how to predict linking with CRF
under this formulation.

###  4.2 Sequential tagging with CRF

As we discussed in Section 3.1, we categorize the relations among learning
materials into homologous and heterologous linking, and adopt different
annotation methodologies (i.e., the alignment and classification). We follow
the same categorization and design our automated linking algorithm for the two
types of relation respectively.

In the homologous case, we apply the linear-chain CRF model to solve the
alignment problem. First, we have two types of materials to be linked: one is
the trunk and the other is a set of candidates of leaves. In this thesis, we
designate the trunk as the lecture videos in the course; thus the candidates
in this homologous case are the lecture slides corresponding to each video.
For each lecture $i$, the video transcription sentences form the sequence of
input feature vectors
$\textbf{x}^{(i)}=(\textbf{x}_{t}^{(i)})_{t=1}^{\mathrm{T}}$ in the CRF (i.e.,
$\textbf{x}_{t}^{(i)}$) is the $t^{\mathrm{th}}$ transcription sentence). Then
we apply CRF to predict a sequence of labels
$\textbf{y}^{(i)}=(Y_{t}^{(i)})_{t=1}^{\mathrm{T}}$ given $\textbf{x}^{(i)}$.
Here the value of unobserved variable $Y_{t}^{(i)}$ is the index of slide
aligned to $\textbf{x}_{t}^{(i)}$; that is,
$Y_{t}^{(i)}\in\\{1,2,...,S^{(i)}\\}$ and $S^{(i)}$ is the number of slides
used in lecture $i$. In this way, we transform the alignment task into a
problem of inferring the value (i.e., the label or the index of the aligned
slide) of $Y_{t}$ from observation x, and solve this inference problem with
CRF.

With the linear chain structure of CRF, the model not only learns the
dependence between observation x (i.e., the content in each learning object)
and alignment y, but also the contextual dependence (i.e., dependence between
$Y_{t-1}$ and $Y_{t}$) over the sequence of objects. Hence, the pattern of
order-preserved mapping can be learnt during the model training. The learnt
patterns function as probabilistic rules affecting the prediction of alignment
(e.g., if sentence $t-1$ is aligned to slide $s$, it is more likely that
sentence $t$ is aligned to slide $s$ or $s+1$, but it is impossible for that
sentence to be aligned to slide $s^{\prime}\in\\{1,2,...,s-1\\}$). These rules
model the order-preserving characteristic of homologous linking.

For heterologous linking, we also apply linear-chain CRF to the binary
classification problem. Similar to the homologous case, we also have a
sequence of learning objects from the trunk (i.e., the lecture videos in this
thesis), and another learning object sequence as the candidates of leaves
(i.e., textbook sections or forum posts in the following implementation). The
video transcription is still the input. However, in contrast to using a
sentence as a token, here we adopt a video vignette (i.e., a sequence of
sentences aligned to the same page of slides in the previous alignment task)
as an item555As explained above, we use larger units here to reduce the number
of tokens, since each supplementary object is considered separately in this
problem, which greatly increases the computation time of the model in training
and inference. Furthermore, a video vignette is a more comparable unit to
supplementary objects used here, which are either textbook sections or forum
posts.. Besides, instead of modeling a sequence of candidate objects at a
time, every object is considered separately, and the CRF is employed to
predict binary labels - whether the considered object is linked or not. That
is, in this task the input of CRF is
$\textbf{x}^{(ij)}=(\textbf{x}_{t}^{(ij)})_{t=1}^{\mathrm{T}}$ for lecture $i$
and supplementary object (e.g., textbook section or post) $j$;
$\textbf{x}_{t}^{(ij)}$ is the collection of transcription sentences of
lecture $i$ which are aligned to slide $t$ in that lecture. The CRF predicts a
label sequence $\textbf{y}^{(ij)}=(Y_{t}^{(ij)})_{t=1}^{\mathrm{T}}$, where
$Y_{t}^{(ij)}\in\\{0,1\\}$ represent whether video vignette $t$ is linked to
supplementary object $j$ or not.

With the linear chain architecture, in this binary classification task we
still model the dependency of linking configurations among neighboring
learning objects on the trunk, but with a loosened constraint. Our model here
can still learn the linking pattern from a sequence of video vignettes. For
instance, for neighboring vignettes, they are more likely to share the same
relationship (i.e., linked or not linked) to a supplementary object. However,
since each supplementary object is considered independently, no dependence
across supplementary objects will be learnt. This modeling of data dependency
agrees with our understanding of heterologous linking. As discussed above,
heterologous linking is not order-preserved. Thus, certain learning objects
arranged closely in one material may imply little about the arrangement of
their linked objects in another material. For example, whether learning object
$\mathrm{A}_{i}$ is linked to object $\mathrm{B}_{j}$ might have little to do
with the event that $\mathrm{A}_{i-1}$ is linked to $\mathrm{B}_{j-1}$ due to
the variation of object arrangement across materials666Here A and B represent
two types of learning materials; $i$ and $j$ are the indices of learning
objects in the two materials respectively.. Since such dependency is of little
information in predicting linking, modeling the dependency across
supplementary and trunk objects simultaneously simply increases the risk of
overfitting (due to the increased model complexity) and introduces noise.

In contrast, the linking configuration of a sequence of trunk learning objects
to a supplementary one does correlate. This is because topic continuity in
educational material is crucial for learners to better digest the content. It
is unlikely that the lecturer switches topics abruptly or frequently. Thus,
for instance, whether learning object $\mathrm{A}_{i}$ is linked to object
$\mathrm{B}_{j}$ is dependent on the event that $\mathrm{A}_{i-1}$ is linked
to $\mathrm{B}_{j}$. As compared to our model for homologous tasks, this model
design is more reasonable for heterologous linking.

###  4.3 Feature extraction

Information in MOOC materials is multimodal. Text, vision, audio content, or
even the click-log can be useful for linking. To represent the diverse data in
a uniform way that can be learned by the CRF model, we have to design the
feature function set in equation LABEL:2-eq:2_4. These features should be
informative and can help the inference of labels. In the following, we discuss
the design of features for our linking task.

Lexical similarity features. Since natural language is the integral part of
education and knowledge transferal, we design our first feature set, lexical
similarity features, based on the text content of course materials. These
features are designed based on the assumption that the similarity between two
learning objects is correlated with whether the two objects are linked. In the
alignment task, lexical features can be written as:

$f_{yk}(Y_{t},Y_{t-1},\textbf{x})=\mathrm{cos\\_sim}(\Phi(\textbf{x}_{t+k}),\Phi(y))\textbf{1}_{\\{Y_{t}=y\\}},k~{}\in~{}\\{-K,-K+1,...,K\\}~{}and~{}y~{}\in~{}\\{1,2,...,S\\}$
(4.1)

Here 1 is an indicator function and K is a hyper-parameter deciding the length
of context considered in the model. $\mathrm{Cos\\_sim}(\textbf{x}_{t+k},y)$
is the cosine similarity between the vector representation (defined by $\Phi$)
of video transcription sentence $t+k$ and the supplementary learning object
$y$. In the alignment task, $y$ is the page index of lecture slides. Thus
learning object $y$ is the $y^{\mathrm{th}}$ page in the slides.

As for the binary classification task, the lexical features we extracted are

$f_{k}(Y_{t},Y_{t-1},\textbf{x})=\mathrm{cos\\_sim}(\Phi(\textbf{x}_{t+k}),\Phi(lo)),~{}k~{}\in~{}\\{-K,-K+1,...,K\\}$
(4.2)

Since in this task each supplementary learning object is considered
separately, the lexical features compute the cosine similarity between the
vector representation of this supplementary object $lo$ and video vignette
$t+k$ (i.e., sentences aligned to slide $t+k$ in the previous task).

To compute the cosine similarity, we have to define the vector representation,
$\Phi$, of a document (i.e., a video sentence, a video vignette, or a
supplementary learning object). The first adopted is a bag of words (BoW)
representation. In this representation, we compute the TF-IDF score of each
word in the document, and transform the document to a vector where each
dimension corresponds to the score of a unique word. Second, we adopt the
word2vec representation, which is introduced in detail in Section 2.3.3.
Word2vec is a continuous language model trained with a neural network to
compute the word probability based on the word’s context in the corpus. After
the model is trained, each word is represented as a vector in a continuous
space by collecting the neural network weights corresponding to that word.
With the word level embedding, each word in a document is transformed first to
its word2vec representation, and the document vector is computed by averaging
these word vectors. As opposed to the BoW model, the long-term semantic and
syntactic regularities in language can be learned in the word2vec embedding.
Thus we believe our linking algorithm can understand learning objects from
different aspects with the two vector representations.

Transition features. As discussed above, we intend to learn the contextual
dependence of linking configuration with CRF. Thus, we design the second
feature set - transition features:

$f_{yy^{\prime}}(Y_{t},Y_{t-1},\textbf{x})=\textbf{1}_{\\{Y_{t}=y\\}}\textbf{1}_{\\{Y_{t-1}=y^{\prime}\\}},~{}y,y^{\prime}~{}\in~{}Y$
(4.3)

where $Y$ is the set of labels. The assumption behind these features is that
the inference of a linked object for two consecutive video segments (i.e.,
sentences or vignettes) is dependent. These features are typically used in
applications of CRF to encode temporal dependencies, and here they allow our
CRF model to learn the temporal patterns of linking configuration.

Visual features. Lecture videos are usually the center of MOOCs. In addition
to the human language content such as video transcription, the visual channel
can also provide rich information to understand the materials and infer the
linking. For example, studies show that scene changes in educational videos
affect learners’ watching behavior and usually coincide with structural breaks
of videos [69, 68, 146]. Hence, we design a set of visual features to extract
this useful information:

$f_{y}(Y_{t},Y_{t-1},\textbf{x})=\mathrm{frame\\_distance}(t)\textbf{1}_{\\{Y_{t}=y\\}},~{}y~{}\in~{}Y$
(4.4)

Here we define $\mathrm{frame\\_distance}(t)$ as the Euclidean distance
between video frames corresponding to the beginning and end of video segment
$t$. The time code information of sentences is encoded in video subtitles,
which are typically provided in MOOCs to enhance material accessibility. If
there are no subtitles, we can still obtain the time code by aligning the
audio signal with lecture transcription or perform automatic speech
recognition.

Since video frames are represented by the color of each pixel, we also have to
transform this information to vectors for computing the distance. We
investigate two vector representations: HSV (hue, saturation, and value)
histogram and horizontal projection. HSV histogram is widely used in tasks
such as scene detection and represents a frame by its color distribution
[124]. HSV histogram of a video frame is obtained by first transforming the
RGB color value of each pixel in the frame to the HSV space. Then the three
coordinates (i.e., H, S, and V) of the HSV space are discretized into a number
of bins. The number of pixels in each bin is counted to compute a three-
dimensional histogram, and the histogram is flattened to a vector as the HSV
representation of the frame. HSV histogram is popular since it models the
mechanism of human color perception. Furthermore, as opposed to simply
representing frames as vectors of color of each pixel, the histogram method is
much more robust to noise.

However, the HSV representation fails to capture some distinct characteristics
of educational videos. For instance, in MOOCs, it is usually the case that the
entire video consists of shots of slides or scenes with very similar colors.
The HSV descriptor cannot effectively distinguish among these slides or
scenes. Thus, we implement the second descriptor, horizontal projection. To
extract this descriptor, for a frame with $m$ by $n$ pixels in the HSV space,
we first represent the frame with three $m$ by $n$ matrices corresponding to
the three coordinates (i.e., H, S, and V). For each matrix we add up the
intensity of each row to project the visual content along the horizontal
direction and obtain an $m$ by 1 vector. The horizontal projection of a frame
is the concatenation of vectors from the three matrices. Since many
informative contents in educational videos are presented horizontally (e.g.,
bullets in slides), this tailored descriptor can describe video frames in a
more pedagogically meaningful way.

One thing that should be noted is that in Equation 4.4, these features only
depend on the frame distance and label of segment $t$. This is problematic
since the frame distance between the beginning and end of sentence $t$ has
little to do with the label of the sentence, but is highly dependent on
whether labels of sentence $t$ and $t-1$ are different. With the current label
set, the dependence between label transition and frame distance cannot be
represented by the features.

Thus, we add another label, boundary, to the original label set. In Fig. 4.1,
we illustrate how this additional label works by comparing the same linking
configuration represented with two label sets. The upper panel of Fig. 4.1
corresponds to the original label set described above, where the value of the
label is the page index of the linked slide (alignment task) or whether the
considered supplementary object is linked (binary classification task). In the
lower panel, an additional boundary label (represented as ’bnd’ in the figure)
is also employed to denote the condition where the two consecutive segments
are linked to various slides or have different relations to the considered
object. With the additional boundary label, the dependence between label
transition and frame distance can be encoded in the visual feature functions,
and the extended label set has the same ability to express linking
configuration as the original set. Thus, we adopt the extended label set for
incorporating visual channel information to our automated linking algorithm.

Figure 4.1: Examples of representing the same linking configuration with two
different label sets. In the upper panel, the original label set described in
Section 4.2 is used. A, B, and C denote the aligned slide index; Y and N
denote whether the considered object is linked. In the lower panel, the
original label set along with a boundary label (denoted as ’bnd’) is used.

Another possible solution to encode the dependence between label transition
and frame distance is using feature functions depending on both $Y_{t}$ and
$Y_{t-1}$. However, this solution increases the number of features and makes
the model more likely to overfit when a small training corpus is used. Thus we
choose the extended label set as the solution.

In this thesis, we only use the frame distance features to encode visual
information in course materials. There are other methods to understand the
visual channel, e.g., optical character recognition (OCR), and semantic
understanding of images [50]. However, OCR cannot provide much additional
information on top of other resources such as slides or video transcription.
Current image semantic understanding can only extract shallow information such
as there is a man writing on the black board or the woman is talking. Thus,
although these methods are getting popular recently, we believe they are not
suitable or mature enough for our tasks, and choose not to investigate them
here.

Meta-data features. Some description of learning content is also informative
in inferring linking. Thus we extract meta-data features to encode such
information. In this thesis, two types of meta-data features are used:
position and learner tagging. In the following we describe the two features
respectively.

The following is the equation of position features in the alignment task:

$f_{y}(Y_{t},Y_{t-1},\textbf{x})=\mathrm{exp}(|\frac{t}{\mathrm{T}}-\frac{y}{S}|),~{}y~{}\in~{}\\{1,2,...,S\\}$
(4.5)

In this feature set, the difference between the relative position of a video
transcription sentence and a slide in their original sequence is encoded. The
relative position is computed by dividing the index of the sentence or slide
by the number of sentences or slides in the lecture. The motivation of this
feature set is self-evident: if a sentence is mentioned at the beginning of
the lecture, this sentence is more likely to be aligned to the first couple of
slides.

We also extract similar features for the binary classification task:

$f(Y_{t},Y_{t-1},\textbf{x})=\mathrm{exp}(|\frac{t}{\mathrm{T}}-\frac{index(lo)}{\mathrm{N}}|)$
(4.6)

In this case $t$ is the index of video vignettes in the entire sequence of
lecture videos, and T is the number of video vignettes in the entire course.
index(lo) is the index of the considered supplementary learning object, and N
is the number of supplementary objects in the course (e.g., number of textbook
sections or discussion threads). The textbook and forum is sorted and indexed
based on the section number and thread creation time respectively.

Learner tagging is another set of meta-data features we utilized. In the MOOC
platform where we collected learning content for experiments in this thesis,
learners are allowed to post discussions under each lecture video [36]. We
record this tagging information and create a function $lo(t)$. This function
maps the $t^{\mathrm{th}}$ video vignette to a set of discussion threads which
were posted under the video this vignette belongs to. With this function, we
extract our learner tagging features as follows:

$f(Y_{t},Y_{t-1},\textbf{x})=\textbf{1}_{\\{lo\in lo(t)\\}}$ (4.7)

where $lo$ represents the discussion thread we considered each time. Although
learners sometimes post irrelevant discussions under the video, such as
chatting with each other, we believe this feature set is helpful for our
automated linking system by narrowing down the possibly linked vignettes to
each thread. Note that the tagging information is only available in
discussions with the current MOOC platform; thus the learner tagging features
are only deployed in the task of linking videos and discussions.

As described above, we can observe that features we utilized here are diverse
in their forms and value ranges. For all these feature functions, we only
assume the existence of dependence between certain observations and labels; no
assumption of distributions for these dependencies is required. Thus, with our
CRF models it is easier to add new features and extend the automatic linking
algorithm as compared to using a generative method such as HMM.

###  4.4 Evaluation: similarity to human labeling

We then evaluate our linking algorithm with materials in the two MOOCs
introduced in Section 2.4: Stat2.1x and 6.00x. In these two MOOCs, we have
lecture videos, slides, and textbook in Stat2.1x; as for 6.00x, the previous
three types of materials along with forum posts are used. Thus, we investigate
automated homologous linking between lecture videos and slides in the two
MOOCs. For heterologous case, we study video-to-textbook-section linking in
Stat2.1x, and video-to-textbook-section along with video-to-discussion-thread
linking in 6.00x. Since in the heterologous case the video vignette is used as
the linking unit, in the experiment we take a two-pass procedure: we first
train a CRF to align video sentences to slides. Then we utilize the best
alignment result in the development set to obtain the video vignette (i.e.,
assign the sentences aligned to each slide as one video vignette). With the
video vignettes we then train CRFs to predict linking between vignettes and
textbook sections/forum discussions.

As discussed at the beginning of this chapter, two issues are explored in the
evaluation: 1) how closely can the automatically generated linking approach to
human annotation? 2) Can automated linking benefit learners? In this section,
the first issue is investigated. Here we compare automated linking results to
humans’ labeling collected and discussed in Section 3.1.4, and compute F1
scores to measure the similarity. As for the second issue, we will explore it
in the next section.

In the evaluation, instead of simply splitting the corpus into training and
testing set, we adopt a 5-fold cross validation technique. Specifically, we
partition our materials into five equal sized batches. Every time, we choose
one batch for testing. The remaining four batches are used for training and
hyper-parameter selection. We iterate the training-testing procedure five
times with different batches as test sets, and average the test set F1 scores
to evaluate the model performance. We adopt a cross validation technique here,
because we want to present the entire course with machine predicted linking.
If we split the corpus into training and testing, the machines can only
predict linking in a portion (i.e., the test set) of the courseware - it is
meaningless to predict linking in the training set since the information of
human annotation is already used for training the model.

We first investigate the performance of machine generated linking in Stat2.1x.
In Table 4.1 we summarize the F1 scores of automated linking systems using
various models and features in the homologous (i.e., linking between video
sentences and slides) and heterologous (i.e., linking between video vignettes
and textbook sections) tasks. To obtain comparable evaluation metrics, in both
tasks F1 scores are computed at the sentence level. Therefore in the
heterologous case we first map the vignette-level linking results to the
sentence level before computing the F1 scores.

Table 4.1: The F1 scores (%) of automated linking systems in Stat2.1x using various models (logistic regression and CRF) as well as lexical (BoW stands for bag of words and word2vec for the neural network word embedding) and visual (HSV stands for HSV histogram and HP for horizontal projection) features. Performance of both homologous (i.e., linking between video sentences and slides) and heterologous (i.e., linking between video vignettes and textbook sections) tasks is listed. In the table, the parentheses after word2vec denote that the HP visual features are only deployed in the homologous task. Linking systems (model, feature) | Homologous | Heterologous
---|---|---
Logistic regression | BoW | 74.9 | 29.6
CRF | BoW | 81.3 | 45.2
BoW + HSV | 84.9 | 44.6
BoW + HP | 85.8 | 44.8
CRF | word2vec (+ HP) | 86.2 | 45.1
BoW + word2Vec (+ HP) | 86.7 | 47.2

In this table we study how visual and lexical features affect linking
performance by using various vectorization techniques to encode learning
objects for computing video frame distances and lexical similarities. Here,
HSV histogram (denoted as HSV) and horizontal projection (denoted as HP)
descriptors are investigated for frame representation; bag of words (denoted
as BoW) and neural network embedding (denoted as word2vec) are studied for
representing text content. Since text content is an integral part in education
for knowledge transferring, and BoW is the most widely used text vectorization
in natural language processing, in our study we start with linking systems
using BoW for lexical features. The word2vec and visual features are gradually
introduced to the systems for investigating their potential benefit.

In addition to visual and lexical features, we also investigate the benefit of
using transition features (i.e., modeling the contextual dependency) in
predicting linking. The effect of transition features is studied by comparing
the CRF algorithm with a baseline model, logistic regression. In logistic
regression, the alignment and linking of each video segment is predicted
separately and no contextual dependency is modeled. Thus by comparing logistic
regression and CRF systems using identical visual and lexical feature sets,
the potential benefit of transition features in linking performance can be
studied.

As for the meta-data features, we utilize them as default features in every
linking system we report, since these features have been shown to be
beneficial in general in a variety of applications [54, 122]. In the
experiment here with Stat2.1x, position features are the meta-data features we
deploy in the linking system.

In this table first we can see that the logistic regression model (row 1)
performs significantly worse than other systems. When an identical feature set
(except transition features) is used, CRF outperforms logistic regression by
6.4% in homologous linking and 15.6% in the heterologous task (c.f., row 1 and
2). This observation shows the benefit of formulating linking as a sequential
tagging problem: by treating the video segments as a data sequence and
introducing transition features into the system, we can model the temporal
patterns of linking configuration. As compared to simply modeling the content
similarity in materials such as in logistic regression, the contextual
information allows the model to better understand how the topics in courseware
are organized and dependent on one another. Therefore, the transition features
yield better linking performance in both tasks.

Then on top of lexical features computed from BoW embedding, we investigate
how the additional visual features affect the performance of CRF linking
systems (c.f., row 2 to 4). In homologous linking, both HP and HSV features
yield improvement. In order to explain why visual features are helpful in this
task, we analyze our course materials. We find that lecture videos are usually
dominated by colloquial speech, and about only 23% of sentences in the video
transcription contain key terms (the terms in the textbook glossary) which are
useful for identifying underlying concepts. This number shows that the lexical
information is sparse. Besides, the lexical similarity features are sometimes
noisy at the transition of two slides or topics. For instance, the lecturer
might conclude the previous topic or connect two topics with a story. Thus,
information from verbal content is often insufficient for inferring linking.
In contrast, the forte of our visual features is encoding information
pertinent to scene changes, which can provide complementary clues to the
lexical features for aligning slides and videos. Thus, combining both features
yields better performance. Furthermore, comparing row 3 and 4 we find that HP
outperforms HSV in the linking performance. We believe that HP is more
suitable for our tasks since it is tailored to educational videos where many
contents are presented horizontally.

As for the heterologous task, we find both visual features yield no
improvement. This is presumably because information encoded in these visual
features is mostly about when the scenes are changed, and has little to do
with inferring the linking between textbook sections and video transcription.
Even if there is any useful information, it is very likely that the
information has already been encoded in the alignment we used for obtaining
video vignettes. Thus, providing visual features in this task may simply
introduce irrelevant or redundant information to the model, and thus no
improvement is observed.

We then study the benefit of word2vec embedding on top of our best systems up
to now (i.e., row 4 in the homologous linking and row 2 in the heterologous
task). In row 5 we replace the lexical similarity features computed from BoW
document embedding with features from word2vec embedding. In the table, the
parentheses after word2vec denote that the HP visual features are only
deployed in the homologous task. As compared to the original best systems, a
similar linking performance is achieved (c.f., 85.8% to 86.2% in the
homologous linking and 45.2% to 45.1% in the heterologous task). In row 6, we
integrate the lexical features computed from both BoW and word2vec embeddings,
and further performance improvement is observed (c.f., 86.2% to 86.7% in
homologous linking and 45.1% to 47.2% in heterologous task).

To explain the improvement, we analyze the two lexical feature sets. We found
that values of similarity features computed from BoW embedding are sparse,
i.e., most values are zero. This is presumably because each word in BoW
representation is treated as an atomic unit; each word’s corresponding element
in the vector works independently from others. Thus, if two documents have no
overlapping words, the cosine similarity between them is zero. Similarity
features computed from this embedding encode information about how many
overlapping words two learning objects have, which is highly correlated with
whether the two objects are linked. However, sometimes different terms are
chosen for expressing the same meaning in various objects or materials. The
BoW embedding fails in dealing with such conditions and tends to give false
negative, i.e., two objects are relevant but their similarity is zero or
under-estimated.

On the other hand, we found that similarity values computed from word2vec
embedding are much smoother. In the word2vec representation, words and
documents are represented as vectors in a continuous space, where dimensions
work jointly to encode different semantic or syntactic regularities.
Semantically or syntactically related words could be distributed closely in
the continuous space, and it is possible that the meaning of unseen words is
reconstructed by their relevant terms. The breaches caused by mismatched
wordings in two documents are alleviated. However, more false positives could
be found, i.e., two objects are irrelevant but their similarity is high. Based
on these findings, we believe BoW and word2vec embeddings provide
complementary encodings of text content, and therefore integrating similarity
features computed from the two yields better performance.

Table 4.2: The F1 scores (%) of automated linking systems in 6.00x using various models (logistic regression and CRF) as well as lexical (BoW and word2vec) and visual (HP) features. Performance of both homologous (i.e., linking between video sentences and slides) and heterologous (i.e., linking between video vignettes and textbook sections/discussion threads) tasks is listed. Linking systems (model, feature) | Homologous | Heterologous
---|---|---
| Videos
---
to slides
| Videos to
---
textbook
| Videos to
---
discussions
Logistic regression | BoW | 63.3 | 66.0 | 31.3
CRF | BoW | 71.7 | 69.3 | 32.1
BoW + HP | 73.7 | - | -
BoW + word2Vec (+ HP) | 74.7 | 71.1 | 33.3

We then apply our method to generate linking in 6.00x. In Table 4.2 the
performance of automated linking systems using various models and features is
listed. Here we also investigate the linking between video sentences and
slides for the homologous linking; as for heterologous task, in addition to
the linking between video vignettes and textbook sections, linking between
vignettes and discussion threads is also studied. Similar to the experiment in
Stat2.1x, we adopt sentence level F1 score as the evaluation metric.

In this thesis, we utilize 6.00x to investigate generalizability of the
proposed methods. Thus, in this experiment we only explore the system
configurations which have shown improvement in the experiment of Stat2.1x, and
examine whether the improvement can be generalized to various courses and
materials. Specifically, we study three techniques which have been shown to be
helpful above: 1) modeling contextual dependency with transition features, 2)
adding visual features to detect scene changes in the alignment task, and 3)
integrating lexical similarity features computed from BoW and word2vec
embeddings for encoding complementary information. Here, we also start with
systems using meta-data features777Position features are deployed in all the
three linking tasks, i.e., video segments to slides, textbook, and
discussions. The learner tagging features are also used in linking video
segments and discussions. and lexical features from BoW embedding, and add
other features incrementally for the investigation.

In Table 4.2 we observe that the three techniques also yield improvement.
Comparing rows 1 and 2, CRF outperforms logistic regression consistently in
the three tasks (8.4% in linking videos to slides, 3.3% in linking videos to
textbook, and 0.8% in linking videos to discussions). This improvement shows
that the benefit of modeling the contextual dependency can be generalized in
these tasks. However, improvement in linking between videos and discussions is
relatively small as compared to other tasks. We believe this is because the
relations between videos and discussions are distinct from videos and others.
Lecture videos, slides, and textbook are created by educators and aimed at
transferring knowledge systematically. In contrast, most content in forums is
created by learners for resolving specific confusions. The topic organization
is very different between learner-created and educator-created materials.
Hence, encoding contextual dependency, which is aimed at improving linking
prediction by understanding topic organization in materials, is much less
helpful in linking videos to discussions.

Comparing rows 2 and 3 we can find that the visual features also yield
improvement in this programming course. Note that here we only explore the
horizontal projection descriptor in homologous task, since in our experiment
in Stat2.1x, HP yields better performance in homologous linking and none of
the visual features is helpful in the heterologous task. As compared to the
results observed in Table 4.1, we find that performance enhancement from HP is
smaller (71.7% to 73.7% here and 81.3% to 85.8% previously). We surmise that
this is because of the differences in the video styles used in the two
courses. We found in Stat2.1x that a large portion of lecture videos is simply
shots of slides; however in 6.00x there is also a great deal of live coding
demos and talking head sessions. The demos and talking head sessions introduce
noise into our visual features and make detecting slide changes more
challenging. From these results, we conclude that the benefit of our visual
features can be generalized, but how much improvement the features can bring
in depends on the styles of underlying lecture videos.

We then further integrate the lexical features computed from BoW and word2vec
embeddings. In the table we can observe that the combination of two embeddings
also enhances the linking performance consistently (c.f., 73.7% to 74.7% in
linking videos to slides, 69.3% to 71.1% in linking videos to textbook, and
32.1% to 33.3% in linking videos to discussions). These results imply that the
complementarity of BoW and word2vec embeddings could be a general fact found
in different courseware; thus the integration could yield generalized
improvement over various courses.

From Table 4.1 and Table 4.2 we can observe that F1 scores range widely (the
best result of each linking task ranges from 33.3% to 86.7%). In order to
investigate the reason of this variation, we compare the best F1 scores to the
annotator agreement (described in detail in Section 3.1.4) in each linking
task in Table 4.3. The first two rows summarize the best performance of our
automated linking systems in F1 scores, and the following two rows list the
labeling consistency among annotators evaluated in terms of kappa scores. The
kappa scores can be interpreted as a measurement of how difficult and
ambiguous the underlying linking task is for humans. In this table we find
that machine performance is highly correlated with the ambiguity of the
linking task. This finding is reassuring, especially for the video-to-
discussion task where only 33.3% F1 score is achieved, since a significant
portion of difference between the linking labeled by machines and humans might
simply come from the ambiguity of tasks, which is much less harmful to learner
experience than linking irrelevant learning objects.

Table 4.3: The comparison between the best performance of our automated linking system (evaluated with F1 scores and listed in the first two rows) and the annotator agreement (evaluated with kappa scores and listed in the third and fourth row) in each linking task. | Homologous | Heterologous
---|---|---
| Videos
---
to slides
| Videos
---
to textbook
| Videos
---
to discussions
Automatic linking (F1 scores, %) | Stat2.1x | 86.7 | 47.2 | -
600x | 74.7 | 71.1 | 33.3
Annotator agreement (Kappa scores, %) | Stat2.1x | 86.5 | 59.9 | -
600x | 81.0 | 76.1 | 43.4

In this section, we show that the CRF-based linking method can integrate
information from various features and yield better performance than the
conventional logistic regression method. By modeling the contextual dependency
and combining the complementary BoW and word2vec embeddings, we observe
consistent improvement in all the linking tasks we studied. As for the visual
features, they are only helpful in homologous tasks. In these experiments, we
show the extensibility of CRF to various features. We believe this
characteristic is a great fit to our problem. Since both MOOCs and machine
learning are booming in recent years, the styles of learning materials and
algorithms for understanding content are ever-changing. With the extensibility
our model can grow with new machine learning techniques, pedagogies, and
content by adding corresponding features (e.g., our CRF can take posterior
probabilities or classification results predicted by a neural network as
features). Besides, the improvement across courses and materials to some
extent demonstrates the generalizability of the proposed method, which is
crucial for our framework to be widely applied in various conditions.

In the previous discussion we also examined the correlation between model
performance and annotator agreement. We surmise that it is likely that the low
F1 scores will not harm learner experience by much. In the next section, we
attempt to provide evidence for this conjecture with a user study experiment.

###  4.5 Evaluation: benefit in learning

We then investigate the second issue of evaluation: can linking generated
automatically still benefit learners? For this issue, we conduct a user study
similar to what we did in Section 3.4. We study learners’ performance when
course materials are presented with various strategies: presenting different
types of materials separately (i.e., presented in $null$ interface) or in
linking by humans (i.e., $linking$) or machines (i.e., $auto~{}linking$). If
similar performance improvement as described in Section 3.4 can be observed
after replacing manually labeled linking with the one generated by
machines888In this section, the best linking results achieved in Section 4.4
are used in the interface for conducting a user study., we can conclude that
our automated linking system is beneficial to learners. In the following, we
also study the effect of automated linking on learners by learning tasks:
search and retention.

####  4.5.1 How automated linking affects search

Here we investigate how the automated linking affects learners’ performance in
the information search scenario. Performance is also evaluated with two
metrics: average searching time and average accuracy. In Table 4.4 the
performance in Stat2.1x is summarized. The results of the $null$ and $linking$
interfaces are identical to the ones discussed in Section 3.4.1.
$Auto~{}linking$ corresponds to the condition where learners utilize an
interface presenting automatic linking for accomplishing assigned tasks.
Except for the deployed interfaces, other experimental procedures in the study
of $auto~{}linking$ are identical999We also collected 1,000 HITs for the same
10 questions on Amazon mechanical turk. Since a between-subjects design is
adopted, only online workers who did not participate in our experiment with
$null$ and $linking$ interfaces are allowed in the study. The study of
$auto~{}linking$ was conducted three months after the experiment of $null$ and
$linking$ interfaces was complete. Besides, the same quality control mechanism
is applied here.. Moreover, to examine how automatic linking can benefit
learners, we also focus on the performance difference when various interfaces
were deployed. In Fig. 4.2 the improvement from $linking$ (red bars) and
$auto~{}linking$ (black bars) as compared to $null$ is visualized. The upper
panel corresponds to the time reduction the $linking$ and $auto~{}linking$ can
yield in different learner cohorts; the lower panel shows the accuracy
increase from the two interfaces. Furthermore, the 95% confidence interval of
the difference is also provided.

Table 4.4: Learner performance in the information search scenario in the study of Stat2.1x. Performance is evaluated by the average searching time and average accuracy metrics, and measured within various cohorts using $null$, $linking$, or $auto~{}linking$ interfaces. |  | Average searching time (seconds) | Average accuracy (%)
---|---|---|---
|  | null | linking | auto linking | null | linking | auto linking
Overall | 206 | 152 | 162 | 69.2 | 69.5 | 69.5
Statistics | Yes | 166 | 147 | 154 | 71.1 | 70.5 | 70.0
No | 295 | 160 | 178 | 64.9 | 67.1 | 68.0
MOOCs | Yes | 166 | 139 | 160 | 72.0 | 70.6 | 71.7
No | 225 | 154 | 163 | 68.2 | 68.9 | 68.5
$\geq$Bachelor | Yes | 198 | 163 | 167 | 70.7 | 70.6 | 70.0
No | 208 | 136 | 151 | 67.5 | 68.5 | 68.3
Figure 4.2: The improvement in search time (upper panel) and accuracy (lower
panel) when $linking$ (red bars) or $auto~{}linking$ (black bars) interface is
used, with the baseline of deploying $null$. Learning performance improvement
is measured in the study of Stat2.1x. The 95% confidence intervals (shown as
error bars) and significance test results (marked with red asterisk if the
difference is statistically significant) are also provided.

Similar to the $linking$ interface, in Table 4.4 and Fig. 4.2 we can observe
that an interface driven by machine generated linking also helps learners
finding information with a shorter period of search time in general; as for
the search accuracy, no statistically significant difference can be found.
Among the studied learner cohorts, novice learners (subjects without prior
knowledge in statistics, without prior exposure to MOOCs, and without a degree
higher than bachelor’s) as well as subjects with a degree higher than
bachelor’s show statistically significant improvement in search time
reduction. On the other hand, comparing the improvement yielded by $linking$
and $auto~{}linking$, we find that in each cohort subjects using the $linking$
interface consistently take less amount of time in search.

We also explore how automated linking affects search in 6.00x. The results are
summarized in Table 4.5. In this table, in addition to the learning
performance reported in Table LABEL:3-table:search_res_600, the result of the
user study deploying the $auto~{}linking$ interface is also listed101010In
this study, each experimental procedure except the deployed interface is the
same as in Section 3.4.1. 1,000 HITs for the same 10 questions have been
collected on Amazon mechanical turk. A between-subjects design is adopted. In
this study we conduct the experiment with the three interfaces (i.e., $null$,
$linking$, and $auto~{}linking$) simultaneously. Additionally, the same
quality control mechanism is employed.. Furthermore, we also visualize the
performance difference when various interfaces were deployed in Fig. 4.3. The
improvement (i.e., time reduction in the upper panel and accuracy increase in
the lower) from $linking$ (red bars) and $auto~{}linking$ (black bars) as
compared to $null$ is presented.

Table 4.5: Learner performance in the information search scenario in the study of 6.00x. Performance is evaluated by the average searching time and average accuracy metrics, and measured within various cohorts using $null$, $linking$, or $auto~{}linking$ interfaces. | Average search time (seconds) | Average accuracy
---|---|---
null | linking | auto linking | null | linking | auto linking
Overall | 443 | 349 | 360 | 87.7 | 89.5 | 90.4
Python | Yes | 419 | 323 | 352 | 90.3 | 90.3 | 93.0
No | 463 | 378 | 365 | 85.6 | 88.6 | 88.6
MOOCs | Yes | 427 | 336 | 336 | 88.0 | 89.4 | 91.1
No | 454 | 357 | 371 | 87.6 | 89.5 | 90.0
$\geq$Bachelor | Yes | 472 | 359 | 353 | 89.5 | 91.5 | 92.3
No | 399 | 331 | 370 | 85.1 | 86.2 | 87.4
Figure 4.3: The improvement in search time and accuracy when various
interfaces were used in the study of 6.00x. The improvement from using
$linking$ (red bars) and $auto~{}linking$ (black bars) as compared to $null$
is plotted.

In Table 4.5 and Fig. 4.3, we find that the $auto~{}linking$ interface also
allows learners to complete tasks with less time in most cohorts (except for
subjects without a bachelor’s degree), as compared to the $null$ interface.
Search accuracy is also improved in the entire group of subjects, subjects
with or without experience in Python, subjects with or without previous
exposure to MOOCs, and subjects with a bachelor’s degree or higher. Besides,
comparing the improvements yielded by the $linking$ and $auto~{}linking$
interface, we can observe that they are highly correlated and statistically
significant in mostly the same cohorts (except for the time reduction of
subjects without a bachelor’s degree, the accuracy of subjects with experience
in Python, and the accuracy of subjects with previous exposure to MOOCs).

From the user study results discussed above, we can observe that in most cases
the $auto~{}linking$ interface can still help learners in the search task, but
generally yield slightly less improvement than the $linking$ interface (except
for the accuracy of the study of 6.00x). The observation is interesting: our
results in Section 4.4 show that some of the linking used in the
$auto~{}linking$ interface is very different from the one labeled by humans
(e.g., the linking between videos and discussions in 6.00x); however, based on
our user study experiment, learners seem to be able to benefit from both
interfaces using somewhat different linking annotations.

We believe the user study results support our previous conjecture that even
though there are some discrepancies between linking labeled by humans and
machines, many of the differences could come from the ambiguity of underlying
linking tasks. Therefore, both humans and machines make reasonable linking
decisions, and learners can benefit from both $linking$ and $auto~{}linking$
interfaces. However, from the results we also believe that machines cannot
reach the same depth of understanding of the learning content as humans, and
thus still make some linking errors which confuse learners. Hence, usually
less improvement in the learning performance is measured when we replace
human-labeled linking with the machine generated one. In Section 4.6, we will
continue the discussion about the difference between linking labeled by humans
and machines, as well as look into the difference patterns to explain why
$auto~{}linking$ is still helpful, but next we will investigate our other
learning scenario: concept retention.

####  4.5.2 How automated linking affects information memorization

Here we explore how the automated linking affects learners’ performance in the
concept retention scenario. Performance is measured by the numbers of unique
key-terms in the essays submitted by learners. Table 4.6 compares the
performance when an $auto~{}linking$ interface was deployed to the results
when $linking$ or $null$ was used (the ones reported in Table
LABEL:3-table:retention_res_stat in the experiment of Stat2.1x). The user
study of the $auto~{}linking$ interface follows the same experimental
procedures described in Section 3.4.2, e.g., collecting 1,000 HITs for the
same 10 sampled topics on AMT, adopting a between subjects design, and
applying a plagiarism check for quality control111111This study was also
conducted three months after the experiment of $null$ and $linking$ interfaces
was complete.. Moreover, the performance improvement (increased number of key-
terms) from $linking$ (red bars) and $auto~{}linking$ (black bars) as compared
to $null$ is visualized.

Table 4.6: Learner performance in the concept retention scenario in the study of Stat2.1x. Performance is evaluated by the number of unique key-terms in submitted essays and measured within various cohorts using $null$, $linking$, or $auto~{}linking$ interfaces. |  | Number of unique key-terms
---|---|---
|  | null | linking | auto linking
Overall | 4.39 | 4.91 | 4.83
Statistics | Yes | 4.71 | 5.11 | 5.02
No | 3.98 | 4.60 | 4.50
MOOCs | Yes | 4.83 | 5.14 | 5.07
No | 4.27 | 4.77 | 4.75
$\geq$Bachelor | Yes | 4.73 | 5.23 | 5.04
No | 3.98 | 4.60 | 4.46
Figure 4.4: The improvement in the number of unique key-terms contained by
submitted essays when a $linking$ (red bars) or $auto~{}linking$ (black bars)
interface is used, with the baseline of deploying $null$. Learning performance
is measured in the study of Stat2.1x. The 95% confidence intervals and
significance test results are also provided.

Similar to the results in the information search scenario, automatic linking
also yields performance improvement in the retention task. As compared to
subjects assigned with the $null$ interface, learners who use $auto~{}linking$
mentioned more key-terms in their essays. The improvement is statistically
significant in the entire group of experimental subjects as well as among the
novice learners. Moreover, if we compare the improvement yielded by $linking$
and $auto~{}linking$, we can also find that the latter one has consistently a
smaller increase in the number of key-terms.

The effect of automated linking on the retention task in 6.00x is also
summarized in Table 4.7. Columns of $null$ and $linking$ correspond to the
results discussed in Table LABEL:3-table:retention_res_600. The learning
performance observed in user study where the $auto~{}linking$ interface is
deployed is listed in the column $auto~{}linking$121212In the study, the same
experiment setup is adopted, such as 1,000 HITs for the same 10 topics and a
between subjects design. Besides, the experiment with the three interfaces
(i.e., $null$, $linking$, and $auto~{}linking$) is conducted at the same
time.. Additionally, the performance difference observed when various
interfaces were used is visualized in Fig. 4.5. The increase in the number of
key-terms from using $linking$ (red bars) and $auto~{}linking$ (black bars) as
compared to $null$ is plotted.

Table 4.7: Learner performance in the concept retention scenario in the study of 6.00x. Performance is evaluated by the number of unique key-terms in submitted essays and measured within various cohorts using $null$, $linking$, and $auto~{}linking$ interfaces. | Number of unique key-terms
---|---
null | linking | auto linking
Overall | 8.07 | 8.56 | 8.39
Python | Yes | 8.64 | 9.09 | 8.74
No | 7.64 | 8.20 | 8.14
MOOCs | Yes | 8.37 | 8.55 | 8.63
No | 7.93 | 8.56 | 8.28
$\geq$Bachelor | Yes | 8.60 | 9.13 | 8.77
No | 7.21 | 7.91 | 7.88
Figure 4.5: The improvement in the number of key-terms contained by submitted
essays when various interfaces were deployed in the study of 6.00x. The
improvement from using $linking$ (red bars) and $auto~{}linking$ (black bars)
as compared to $null$ is visualized.

From Fig. 4.5 and Table 4.7, we can find automatic and manual linking perform
similarly. Both of the $linking$ and $auto~{}linking$ interfaces allow
learners in each cohort to mention more key-terms in their summary of assigned
topics. However, at 95% confidence interval, the improvement yielded from the
$auto~{}linking$ interface is not statistically significant in the seven
cohorts studied here.

Similar to what we found in the information search scenario, the user study
result discussed here also shows that, disregarding the difference between the
manual and automatic linking, learners can benefit from both. The results
provide additional evidence for our conjecture that many disagreements between
the manual and automatic linking come from the task ambiguity, and our CRF
model still makes reasonable decisions in linking learning objects. Hence,
only a small degradation in learning performance was found when manual linking
was replaced by an automated one.

###  4.6 Difference pattern analysis

In the previous section, we provide evidence that the automated linking can
still benefit learning, and that the low similarity between manual and
automated linking only causes a small degradation in observed learning
performance. In this section, we attempt to provide explanations for why
automatic linking is also helpful in learning although less, by looking into
the difference pattern between manual and automatic linking.

For the analysis, we choose the task of linking between video vignettes and
discussion threads, which is the task of the lowest F1 score. We compare the
linking predicted by our best automated system to human annotation. In the
comparison, we first sampled 50 discussion threads from those threads which
are linked to different vignettes in human and machine labeling. After
skimming through the sampled threads, we summarize four difference patterns:

  1. 1.

Pattern 1: only annotators linked some vignettes to the thread.

  2. 2.

Pattern 2: only machines linked some vignettes to the thread.

  3. 3.

Pattern 3: both machines and annotators linked some but not the same vignettes
to the thread, and the non-overlapping vignettes belong to the same lecture
video.

  4. 4.

Pattern 4: both machines and annotators linked some but not the same vignettes
to the thread, and the non-overlapping vignettes belong to various lecture
videos.

We categorize the sampled 50 threads into the four patterns in Table 4.8. Here
we find that pattern 1 dominates in numbers. With the categorization, in the
following we analyze each pattern and how it can affect learning experience in
order to explain the user study results.

Table 4.8: The number of threads categorized into the four difference patterns. | Pattern 1 | Pattern 2 | Pattern 3 | Pattern 4
---|---|---|---|---
Number of threads | 31 | 6 | 8 | 5

Pattern 1 includes 62% of the sampled threads. However, we believe that this
pattern has the least negative effect on learning experience among the four.
Because of the way we present learning content and visualize linking, in this
pattern, the $auto~{}linking$ interface simply regresses to the $null$; since
the thread is not linked to any video vignette by machines, this thread is
presented under a separate discussion tab. Although the regression increases
the difficulty to access this thread, the user experience in interacting with
the rest of the learning materials is almost the same. Thus, learners can
still access desired information from the rest of the materials to accomplish
their tasks as they did in the interface driven by manual linking.

To illustrate this difference pattern and give a concrete example, we consider
one specific thread from our set of 31. We present the content of this thread
and its linked vignette in Fig. 4.6. In the left panel of the figure, we see
that the discussion is about how dictionaries enable quick web searches in
Google. In the right panel we observe that our TAs related this discussion to
a vignette131313Note that since a video vignette is defined as the video chunk
aligned to one slide page, the content in a vignette can be well summarized by
the aligned slide. For simplicity, here we show the aligned slide to represent
the vignette. introducing the basic idea of a dictionary, while the machines
linked nothing to this post. Thus, in our $linking$ interface, when learners
are surveying this vignette of dictionary introduction, this discussion about
web search will be rendered under the video, while no discussion will be
presented in the $auto~{}linking$ interface. However, the discussion only adds
fun facts and additional information to the vignette. Without the post, the
concept of a dictionary can still be properly learned from the vignette. This
example implies that the effect of difference pattern 1 on learning can be
negligible.

Figure 4.6: An example of difference pattern 1. The left panel shows a sampled
discussion thread; the right panel presents the vignette linked to the thread
by human annotators (upper right) and our CRF algorithm (lower right). The
condition where none of the vignettes is linked is represented by $\emptyset$.

In addition, we examine the relation between this difference pattern and task
ambiguity. We found that in 19 out of the 31 threads one of the three
annotators141414Note that the manual annotation is obtained by taking majority
voting over the labeling of three TAs. agrees with the machines (i.e., linking
nothing to the thread). From this result we believe that many disagreements
between machines and annotators come from the conservativeness of the
machines: our automated algorithm is inclined not to link videos and threads
when the linking is ambiguous. This fact also supports our belief that this
difference pattern has little negative effect on learning, since learners can
focus on the case when the relation between learning objects is strong.

Six out of the 50 sampled threads are categorized into the second pattern.
This pattern deteriorates learning experience and makes learners confused. In
this case, the machines relate some vignettes which have been decided
irrelevant by annotators to discussion threads. Under these vignettes the
discussion threads are shown in our $auto~{}linking$ interface. Learners have
to spend their cognitive capacity to understand these unrelated threads and
might feel confused about why these threads are presented. Therefore with this
type of difference pattern, learning performance is lowered.

Figure 4.7: An example of difference pattern 2. The left panel shows a sampled
discussion thread; the right panel presents the vignette linked to the thread
by human annotators (upper right) and our algorithm (lower right).

We also give an example of difference pattern 2. In Fig. 4.7, we consider one
specific thread from the set of six on the left; the content of the linked
vignette labeled by humans and machines is presented on the right. The thread
is meaningless discussion, but the machines link it to a vignette describing a
hash function. It is obvious that this type of difference may distract or
confuse learners, and thus deteriorate their learning experience.

We also examine the question of whether all disagreements result from
algorithmic errors or whether some of them result from task ambiguity. We
found that in the case of difference pattern 2 one of the three annotators
agrees with the machine’s linking decision in 2 out of the 6 threads. Hence,
we believe that not all differences in this category are the result of machine
mistakes, and only a portion of the differences causes a decline in the
learners’ performance.

As for the third pattern, 8 out of the 50 sampled threads are classified to
this category. Similar to the first pattern, we also believe this difference
pattern has little negative effect on learners with our interface design. In
this pattern, although the machines and humans link different vignettes to a
thread, these vignettes belong to the same lecture video. With our interface
design, this thread is presented under the same video but only aligned to
different parts of the video scrubber. Therefore, this difference pattern does
not change the user experience by much151515In 4 out of the 8 threads, one of
the three annotators link the same video vignettes as machine does. This fact
also supports our claim that our automated system links reasonable vignettes
and thus it is very likely the disagreement does not affect learners
negatively..

Figure 4.8: An example of difference pattern 3. In this example, the thread
(left panel) is linked by humans and machine to two vignettes (right panel) in
the same lecture video. The two vignettes are closely related to each other,
and the difference in presenting these two ways of linking is minor.

An example of difference pattern 3 is shown in Fig. 4.8. As we can see, the
thread is relevant to both vignettes linked by humans and machine. When the
two various ways of linking are presented in our interface, qualitatively the
difference is minor, which supports our previous claim.

A total of 10% (5 out of 50) of the sampled threads are categorized as the
last pattern. As we discussed about pattern 2, this difference category also
makes the learner confused and deteriorates learning experience, since in
$auto~{}linking$ the discussion threads are linked to lecture videos which are
totally different from the ones labeled by the humans, and here our automated
linking algorithm agrees with any one of the three annotators in only 1 out of
the 5 threads. Thus, we conclude that this difference pattern is less probably
caused by task ambiguity, and it is very likely that the videos only linked in
our automated system are irrelevant to the threads.

Figure 4.9: An example of difference pattern 4. Here, humans and machine link
two vignettes (right panel) from various lecture videos to the same discussion
the thread (left panel).

Fig. 4.9 shows an example of difference pattern 4. In the posts here, learners
discussed how the Python interpreter utilizes symbol tables to keep track of
variable bindings in recursion, which is explained exactly in the vignette
linked by our TAs. In contrast, our CRF algorithm links this thread to a less
relevant vignette belonging to another lecture video describing global
variables. Learners may be distracted by the discussion of the symbol tables
when viewing this video.

From the analysis we can find that, although the similarity between automated
and manual linking is low in some tasks, many of the differences resulting
from the task ambiguity and the linking labeled by both machine and humans are
reasonable, or the differences can be properly presented in our interface.
Such differences could be recovered or ignored easily by learners and thus
have little effect on user experience. Therefore, we also observe considerable
improvement in learners’ performance when the $auto~{}linking$ interface is
deployed. Note that this analysis was done in the task of linking video
vignettes and discussion threads. We believe that the conclusion can be
generalized to the linking of other materials, since similar automated
algorithms were utilized.

###  4.7 Comparison with the edx interface

In studies above, we have shown that, when we present learning content to
learners, if we are able to visualize the relations among content, learners
can achieve better performance in completing assigned learning tasks.
Furthermore, we demonstrated that we can obtain the relations from human
annotators or the proposed CRF linking algorithm. These studies were conducted
by comparing the $linking$ and auto linking interfaces to the $null$
interface, which is a baseline that implements the conventional strategy for
delivering learning materials online. In this section, we investigate whether
our linking framework can provide added value to the interface currently
deployed in MOOC platforms (here we choose the edX website as our baseline).

####  4.7.1 The edx interface

For the user study being deployed in the consistent condition, rather than
using the edX website directly, we also implement $our$ $edx$ interface to
conduct the study on AMT. A screenshot of this interface is presented in Fig.
4.10. Here, we reproduce the design of the interface and the layout of content
from the edX website in order to offer learners a user experience that is
identical to the one they used when engaged in a state-of-the-art MOOC
platform. The only difference between our $edx$ interface and the real
platform is that here we additionally provide the search mechanism for
accessing course materials; this is also done in order to create a consistent
comparison to our $linking$, $auto~{}linking$ and $null$ interfaces.

Figure 4.10: The implemented $edx$ interface that reproduces the design and
layout of the edX website to offer learners a user experience similar to that
of a real MOOC, except that we also give the additional search tool for
accessing course materials. This interface is used to investigate how much
added value our linking framework can provide for the state-of-the-art MOOC
platforms.

In this interface, instructors can upload a deck of lecture slides beneath the
paired lecture video. The associated slides are presented as a link under the
video player for learners to refer to. In the edX website, to motivate
learners engaging in discussions and to organize the enormous forum posting,
learners are allowed to post under a lecture video to specify the relation
between their discussions and the lecture; these posts are directly rendered
under the video for future learners. This functionality is also implemented in
our $edx$ interface.

From the design of the $edx$ interface we can find that this interface can be
interpreted as another realization of educational content linking on different
information levels: lecture slides are linked on the lecture level rather than
the page level as they are in the $linking$ interface; the relation between
discussions and lecture videos is inferred from learners’ choice rather than
the content these materials contain; the textbook is still presented
separately. Thus, as compared to the $null$ interface we build, the $edx$
interface serves as a baseline in the comparative study for a different
purpose. We implement the $edx$ interface in order to investigate how much
added value our linking framework can provide for the state-of-the-art MOOC
platforms. In contrast, we utilize the $null$ interface for exploring the
fundamental research question asked in this thesis: can linking help
learning?.

####  4.7.2 The user study, results, and discussions

To examine the value of the proposed linking framework can add to the state-
of-the-art MOOC platform, we conduct another user study. In the study, again
we published 1,000 HITs on AMT for each of the two learning scenarios, and
learners in this study had to use the $edx$ interface to accomplish tasks. By
comparing the learning performance measured here to the results from the
$null$, $linking$, and $auto~{}linking$ interfaces, we can investigate whether
our educational content linking framework can potentially improve the current
MOOC design. Note that in the study here, except for the deployed interface
for completing tasks, the remaining experimental setup is identical to the
other user studies discussed in this thesis (e.g., the same 10 sampled
questions and topics, the same number of rewards, the same quality control
mechanism, and the between-subjects design). Furthermore, for simplicity in
this chapter we only investigate 6.00x for our study, and the user study was
conducted together with the other three (i.e., $null$, $linking$, and
$auto~{}linking$) interfaces.

Table 4.9 summarizes learner performances (evaluated by the average searching
time and accuracy) in the information search scenario. Columns 1 to 3 and 5 to
7 correspond to the performance measured when the $null$, $linking$, and
$auto~{}linking$ interfaces are utilized. These results are identical to the
ones reported in Table 4.5. The user study result when the $edx$ interface was
deployed is listed in columns 4 and 8. Here, we employ the same dividing
criteria for stratifying learners’ background (i.e., prior knowledge,
experience in MOOCs, and highest degree).

Table 4.9: Learner performance in the information search scenario in the study of 6.00x. In addition to the results reported in Table 4.5, performance (evaluated by the average searching time and average accuracy) measured when the $edx$ interface was used is listed. | Average search time (seconds) | Average accuracy
---|---|---
null | linking | auto linking | edx | null | linking | auto linking | edx
Overall | 443 | 349 | 360 | 401 | 87.7 | 89.5 | 90.4 | 88.6
Python | Yes | 419 | 323 | 352 | 352 | 90.3 | 90.3 | 93.0 | 89.5
No | 463 | 378 | 365 | 443 | 85.6 | 88.6 | 88.6 | 87.9
MOOCs | Yes | 427 | 336 | 336 | 386 | 88.0 | 89.4 | 91.1 | 89.3
No | 454 | 357 | 371 | 409 | 87.6 | 89.5 | 90.0 | 88.2
$\geq$Bachelor | Yes | 472 | 359 | 353 | 393 | 89.5 | 91.5 | 92.3 | 90.8
No | 399 | 331 | 370 | 411 | 85.1 | 86.2 | 87.4 | 85.8

Since our goal is to examine how much added value our educational content
linking framework can bring us, we again visualize the differences of
performance yielded by different pairs of interfaces in Fig. 4.11. In this
figure, the improvement (i.e., time reduction in the upper panel and accuracy
increase in the lower) from $linking$ (red bars) and $auto~{}linking$ (black
bars) as compared to $null$, as well as the improvement from $linking$ (blue
bars) and $auto~{}linking$ (orange bars) as compared to $edx$ is presented. By
looking at the blue and orange bars, we can understand the potential of our
linking framework in improving the content delivery in current MOOC platforms.
The red and black bars are plotted here for comparison to our previous
results, and they are identical to the ones in Fig. 4.3.

Figure 4.11: The improvement in search time and accuracy when various
interfaces were deployed. The improvement from using $linking$ (red bars) and
$auto~{}linking$ (black bars) as compared to $null$, as well as the
improvement from $linking$ (blue bars) and $auto~{}linking$ (orange bars) as
compared to $edx$ is plotted.

We report the observed performance in the concept retention scenario in Table
4.10. In addition to the results discussed in Table 4.7, which are listed in
columns 1 to 3 here, the performance measured for learners who use the $edx$
interface for their tasks is presented in column 4. Additionally, to visualize
the added value our linking framework may yield, we also plot the differences
in the number of key-terms when various interfaces were deployed in Fig. 4.12.

Table 4.10: Learner performance in the concept retention scenario in the study of 6.00x. In addition to the results reported in Table 4.7, the number of unique key-terms in submitted essays measured when the $edx$ interface was used is listed. | Number of unique key-terms
---|---
null | linking | auto linking | edx
Overall | 8.07 | 8.56 | 8.39 | 8.03
Python | Yes | 8.64 | 9.09 | 8.74 | 8.47
No | 7.64 | 8.20 | 8.14 | 7.68
MOOCs | Yes | 8.37 | 8.55 | 8.63 | 8.51
No | 7.93 | 8.56 | 8.28 | 7.76
$\geq$Bachelor | Yes | 8.60 | 9.13 | 8.77 | 8.67
No | 7.21 | 7.91 | 7.88 | 7.23
Figure 4.12: The improvement in the number of unique key-terms contained by
submitted essays when various interfaces were deployed. The bars are pictured
in the same way as in Fig. 4.11.

From the $linking-edx$ and $auto-linking-edx$ bars (i.e., the blue and orange
bars) in Fig. 4.11, we find that our linking interfaces (driven by either
manual or automated linking) allow learners to search for content using less
time but with similar accuracy. Comparing each $linking-edx$ and $auto-
linking-edx$ bar to the corresponding $linking-null$ (i.e., red) and $auto-
linking-null$ (i.e., black) one, less time reduction is observed in general,
and the reduction is significant in fewer groups of subjects. For the
$linking-edx$ and $auto-linking-edx$ bars of subjects with experience in
Python, subjects with previous exposure to MOOCs, and subjects with a
bachelor’s or higher degree, as well as the $auto-linking-edx$ bar of subjects
without previous exposure to MOOCs, the differences are not statistically
significant. However, their counterparts in the $linking-null$ and $auto-
linking-null$ bars are significant. Results here suggest the added value our
$linking$ and $auto~{}linking$ interface may provide for the current MOOC
platforms. Furthermore, in the search accuracy, only the $auto-linking-edx$
bars of the entire group of subjects, subjects with experience in Python, and
subjects without previous exposure to MOOCs show statistically significant
improvement. As compared to the $linking-null$ and $auto-linking-null$ bars,
statistically significant difference is observed in much fewer cohorts. These
observations imply that the $edx$ interface is a better baseline (in terms of
yielding better learning performance) than the $null$ one. The reason is self-
evident: the $edx$ interface implements some ideas of linking, which have been
shown to be able to make course materials more accessible and improve
learning.

Our linking interfaces can also yield more key-terms consistently over each
cohort as compared to the interface reproducing the current MOOC design (Fig.
4.12). Furthermore, in this learning scenario, the $null$ and $edx$ interfaces
seem to perform similarly, and therefore the difference between the $linking-
edx$ and $auto-linking-edx$ bars to the $linking-null$ and $auto-linking-null$
ones in the same cohorts is much less obvious than in the information search
scenario. We surmise that this difference results from the nature of the two
scenarios. By comparing the $null$ and $edx$ interfaces, one of the most
significant changes is that in the $edx$ interface relevant discussions are
stacked beneath the lecture video. Discussions are usually initiated because
of confusions about specific problems or concepts in the learning content.
These discussion threads contain useful information to solve questions asked
in the search scenario. However, information in these posts is fractional and
thus it is challenging to learn about a topic systematically from these posts.
Therefore, the improved navigation over discussions is much more helpful in
our search tasks.

Our results show that our educational content linking framework can
potentially allow learners to find desired learning content faster than the
current MOOC interface design with comparable accuracy; with our proposed
interfaces, subjects can also retain more information after learning with the
assigned topic using the same amount of time. Since the $edx$ interface also
partially implements our linking idea, we surmise that the improvement results
from several differences in the design of interfaces. First, the page-level
alignment between slides and lecture videos can better visualize the structure
and emphasize the sub-goals of videos. Second, the relation between
discussions and videos tagged by learners can be noisy and distract other
learners; considering the topical relevance between content contained in the
two types of materials can control the quality of tagging. Third, some
discussions might be related to only parts of the videos. This information is
not available in the current MOOC platforms. Fourth, it might be worthwhile to
visualize the relations between lecture videos and external resources, such as
a recommended textbook. We believe that, by integrating these features
properly into the current platform design, learning experience in MOOCs can be
enhanced significantly.

###  4.8 Conclusions

This chapter investigates our second research question: can linking be
generated at scale? For this question, we formulate the linking annotation as
a sequential tagging problem, and propose an automatic content linking
algorithm based on CRFs. In the algorithm, to infer the linking a variety of
features are utilized: lexical similarity, transition, visual, and meta-data
features. By analyzing the difference pattern between automated and manual
linking, we find that many differences result from the task ambiguity and have
little negative effect on learners. Hence, in our user research we observe
similar improvement in learning performance when the auto linking interface
replaces the $linking$ one. We conclude that our linking framework can be
realized at scale with an automated algorithm.

Furthermore, we also compare our $linking$ and auto linking interfaces to the
reproduction of the edX website, and explore how these interfaces support
learners in completing assigned tasks. In the user study we also demonstrate
that, as compared to the current design in MOOC platforms, our educational
content linking framework can still help learners find information faster and
retain more concepts. We believe that the linking framework we propose
enriches the current MOOC interface design. We envision engaging learners with
more accessible course materials and better learning experience powered by
content linking.

##  Chapter 5 Conclusions

This dissertation introduced educational content linking: a framework for
organizing course materials to make content more accessible for learners. To
conclude, in this chapter we will summarize the main contributions of this
thesis and discuss possible directions for future work.

###  5.1 Summary and contributions

This thesis contributes to the research community by proposing the framework
of educational content linking. This framework provides better navigation over
learning materials and improves learning experience. Around the framework we
conduct two lines of studies to answer two research questions: 1), if we are
able to link learning materials with humans, would the linking help learners?,
and 2), can the courseware be linked at scale with machine learning
techniques?

With the exploration of the first question, this thesis makes three main
contributions: the linking annotation, the interface design, and the
evaluation.

  * •

The linking annotation. We provide a definition of linking and a formulation
of labeling relations among learning materials as an alignment and binary
classification problem. We design the workflow of annotating linking with
researchers or with collaboration between course staff and online workers.
Linking is an abstract concept and this contribution makes it a reality.

  * •

The interface design. We design an interface to deliver learning content with
the visualization of linking among the materials. The interface can provide
pedagogical benefits through improved content navigation.

  * •

The evaluation. We conduct a large-scale user study with online workers and
two learning scenarios to investigate specific learning mechanisms: search and
retention. We argue that this study can measure the benefits of pedagogical
intervention reliably with reasonable cost, when the underlying learning goals
in the study are clear to participants. The study result shows that the
proposed linking framework can indeed improve learning outcomes in the
investigated search and retention scenarios.

In the second part of this thesis, we investigate the second research
question, and make two main contributions: the automated linking algorithm and
the comparison to a currently deployed MOOC interface.

  * •

The automated linking algorithm. We propose an automated linking algorithm
based on CRFs and multimodal features. In our large-scale user study we
demonstrate that, although there are some differences between the manually and
automatically generated linking, most differences can be properly presented in
our interface or easily ignored by learners, and thus the interface powered by
automated linking can still lead to better learning performance than the
unlinked interface. This result suggests that the proposed linking framework
can be realized at scale with an automatic algorithm based on machine learning
techniques.

  * •

The comparison to a currently deployed MOOC interface. In addition to the
conventional unlinked content delivery, we explore the added value our linking
framework can potentially provide to a currently used MOOC interface. The user
research result suggests that the framework proposed here can possibly enrich
the design of the studied MOOC interface, engage learners with more accessible
learning content, and improve learning outcomes.

###  5.2 Future work

This dissertation shows the potential of educational content linking in
engaging learners with better learning experience. This conceptual idea can be
further verified, refined and applied to various circumstances to improve
learning.

####  5.2.1 Learning platforms of the future

Our user research results demonstrate several possible ways in which linking
can offer learners a better experience. We envision that the future learning
platforms can engage learners with more accessible learning content. While in
this thesis we investigate linking in two learning tasks, search and
retention, there are many other aspects of learning that can potentially
benefit from our proposed framework. For instance, in solving the problem sets
or performing online lab experiments, with better organized learning
materials, learners are more likely to receive proper assistance from the
content when they are confused. Furthermore, our studies were conducted with
online workers. Research implementing the proposed experimental pipeline in a
MOOC environment is valuable to clarify the mechanism by which linking helps
actual learners.

Instead of reproducing our entire implementation of linking, separate
components in our pipeline also inspire directions of design for future
platforms. For example, the automated linking algorithm can be applied to
filter noisy posts and improve the quality of discussions; instructors can
utilize the algorithm to discover relevant learning content to enrich or
reorganize their lectures. Our presentation of lecture videos provides design
implication for better video interaction with visualized structure and
subgoals. The design of our $linking$ interface suggests a neat way to offer
recommended readings. These components lead to new avenues to improved MOOC
platforms.

####  5.2.2 Towards a variety of course subjects and material types

This thesis focuses on two STEM courses: statistics and the Python programming
language. However, as we mentioned in Chapter 1, there have been over 4,000
MOOCs on the Web with subject fields ranging from science and engineering to
humanities and law. In addition to the topics, these MOOCs span different
applied pedagogies, course designs, and methods for content organization or
delivery. "Whether our linking framework can be applied to other subject
fields", "which fields, pedagogies, designs, and content organization can
benefit from linking", and "how various conditions interact with the idea of
linking" are several interesting questions that should be explored. We believe
a wider deployment of this framework in various MOOCs can elucidate these
questions.

With a wider deployment, the involved types of course materials can also vary.
For example, some MOOCs may put strong emphasis on problem sets, while others
may stress on online labs. For the proposed framework to be of more general
use, we also have to answer questions such as "how to extend our
implementation, from linking annotation to visualization, to accommodate these
variations" and "whether and how the conclusions made in this dissertation are
affected by using various materials". Our initial foray of adding forums to
our implementation can be good illustration showing how to investigate these
questions.

####  5.2.3 More sophisticated algorithm for linking at scale

In Chapter 4, we have demonstrated that many differences between manual and
automated linking have little negative effect on learners. However, there is
still a considerable portion of disagreement which may confuse learners. We
believe that a more sophisticated machine learning model can achieve deeper
and more comprehensive understanding of the course materials, and thus
generate linking which is more similar to the human annotated one.

One promising model is the attention-based neural network. This model was
first proposed for machine translation [7]; for each word in the sentence of
the target language, this neural network learns a weight for each word in the
source sentence. The weight represents how relevant the source word is in
predicting the target word. Applying this model to our linking problem, the
target tokens to be predicted are the linking configuration (e.g., the linked
slide index or whether two objects are linked), and the source tokens are two
sequences of learning objects to be linked. This neural network can place
attention on informative learning content when deciding linking configuration;
such a mechanism is similar to how humans generate the linking.

More informative features are also capable of yielding better linking results.
In addition to the information extracted from learning content, such as the
lexical and visual features utilized in our method, user behavior is another
excellent resource for predicting linking. In this thesis, we demonstrate the
usage of learner-generated tags about discussion posts in the linking
algorithm. Aside from these tags, click log and browsing history are very
likely to help linking inference. Kim shows that the aggregation of learners’
video interaction can reveal the underlying video structure and provide
implication for video authoring and interface design [67]. We believe that, by
grouping the browsing history and summarizing patterns of clickstream, we can
understand the relevance among learning materials and extract informative
features for linking prediction.

Beyond the machine learning techniques, crowdsourcing (or learnersourcing) is
an alternative for linking at scale. Li and Mitros proposed a learning module
where learners can collaboratively recommend additional learning objects and
manage the recommended materials for future learners [88]. We envision a
linking system which allows learners and machines to author, edit, and manage
the linking of course materials in a collaborative way.

Furthermore, portability is another issue that should be investigated towards
a scalable linking system. In this thesis, we adopt a 5-fold cross validation
technique to obtain training and testing sets from the same MOOC. This
experimental setup makes people wonder why we need the automated linking,
since the manual labeling of the entire MOOC is available. Hence, more
realistic conditions should be explored; for instance, the automated algorithm
is trained and tested on the same course subjects but different offerings, or
even on different MOOCs.

## References

  * [1] 6.00x introduction to computer science and programming MITx on edX course report - 2013 spring.  http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2382322&download=yes/.  Accessed: 2016-11-05.
  * [2] R. Agrawal, S. Gollapudi, A. Kannan, and K. Kenthapadi.  Studying from electronic textbooks.  In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management, pages 1715–1720. ACM, 2013\.
  * [3] M. Allahbakhsh, B. Benatallah, A. Ignjatovic, H. R. Motahari-Nezhad, E. Bertino, and S. Dustdar.  Quality control in crowdsourcing systems.  IEEE Internet Comput, 17(2):76–81, 2013.
  * [4] Susan A Ambrose, Michael W Bridges, Michele DiPietro, Marsha C Lovett, and Marie K Norman.  How learning works: Seven research-based principles for smart teaching.  John Wiley & Sons, 2010.
  * [5] J. R. Anderson, C. F. Boyle, and B. J. Reiser.  Intelligent tutoring systems.  Science(Washington), 228(4698):456–462, 1985.
  * [6] D. P. Ausubel.  The acquisition and retention of knowledge: A cognitive view.  Springer Science & Business Media, 2012.
  * [7] D. Bahdanau, K. Cho, and Y. Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473, 2014.
  * [8] S. P. Balfour.  Assessing writing in moocs: Automated essay scoring and calibrated peer review (tm).  Research & Practice in Assessment, 8, 2013.
  * [9] Y. Belinkov and J. Glass.  Large-scale machine translation between Arabic and Hebrew: Available corpora and initial results.  arXiv preprint arXiv:1609.07701, 2016.
  * [10] V. Bhat, A. Gokhale, R. Jadhav, J. Pudipeddi, and L. Akoglu.  Min (e) d your tags: Analysis of question response time in stackoverflow.  In Advances in Social Networks Analysis and Mining (ASONAM), 2014 IEEE/ACM International Conference on, pages 328–335. IEEE, 2014.
  * [11] C. Bishop.  Pattern recognition and machine learning (information science and statistics), 1st edn. 2006. corr. 2nd printing edn, 2007.
  * [12] D. M. Blei, A. Y. Ng, and M. I. Jordan.  Latent dirichlet allocation.  Journal of machine Learning research, 3(Jan):993–1022, 2003.
  * [13] B. S. Bloom.  The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring.  Educational researcher, 13(6):4–16, 1984.
  * [14] C. C. Bonwell and J. A. Eison.  Active Learning: Creating Excitement in the Classroom. 1991 ASHE-ERIC Higher Education Reports. ERIC, 1991.
  * [15] Lori Breslow, David E Pritchard, Jennifer DeBoer, Glenda S Stump, Andrew D Ho, and Daniel T Seaton.  Studying learning in the worldwide classroom research into edx’s first mooc.  Research & Practice in Assessment, 8:13–25, 2013.
  * [16] P. Brusilovsky.  Developing adaptive educational hypermedia systems: From design models to authoring tools.  In Authoring tools for advanced technology Learning Environments, pages 377–409. Springer, 2003.
  * [17] P. Brusilovsky, J. Eklund, and E. Schwarz.  Web-based education for all: a tool for development adaptive courseware.  Computer networks and ISDN systems, 30(1):291–300, 1998.
  * [18] P. Brusilovsky and C. Peylo.  Adaptive and intelligent web-based educational systems.  International Journal of Artificial Intelligence in Education (IJAIED), 13:159–172, 2003.
  * [19] C. Callison-Burch.  Fast, cheap, and creative: evaluating translation quality using Amazon’s mechanical turk.  In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1, pages 286–295. Association for Computational Linguistics, 2009.
  * [20] C. Camerer, G. Loewenstein, and M. Weber.  The curse of knowledge in economic settings: An experimental analysis.  The Journal of Political Economy, pages 1232–1254, 1989.
  * [21] P. K. Chan and S. J. Stolfo.  Toward scalable learning with non-uniform class and cost distributions: A case study in credit card fraud detection.  In KDD, volume 98, pages 164–168, 1998.
  * [22] Plagiarism checker.  http://smallseotools.com/plagiarism-checker/.  Accessed: 2016-11-05.
  * [23] J. J. Chen, N. J. Menezes, A. D. Bradley, and T. A. North.  Opportunities for crowdsourcing research on Amazon mechanical turk.  Interfaces, 5(3), 2011.
  * [24] M. Chen, E. D. Lughofer, M. Huang, H. Chiang, P. Wu, and Y. Hsieh.  A multi-strategy machine learning student modeling for intelligent tutoring systems: based on blackboard approach.  Library Hi Tech, 31(2):274–293, 2013.
  * [25] R. Choudhry and K. Garg.  A hybrid machine learning system for stock market forecasting.  World Academy of Science, Engineering and Technology, 39(3):315–318, 2008.
  * [26] R. R. Choudhury, H. Yin, J. Moghadam, and A. Fox.  Autostyle: Toward coding style feedback at scale.  In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion, pages 21–24. ACM, 2016.
  * [27] Derrick Coetzee, Armando Fox, Marti A Hearst, and Bjoern Hartmann.  Chatrooms in moocs: all talk and no action.  In Proceedings of the first ACM conference on Learning@ scale conference, pages 127–136, 2014.
  * [28] Derrick Coetzee, Seongtaek Lim, Armando Fox, Bjorn Hartmann, and Marti A Hearst.  Structuring interactions for large-scale synchronous peer learning.  In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, pages 1139–1152, 2015.
  * [29] O. Comber, R. Motschnig, and Z. Komlenov.  Supporting person-centered learning: Does the choice of the learning management system matter?: A case study with moodle, fronter and cewebs.  In IEEE EDUCON 2010 Conference, pages 885–890, 2010.
  * [30] G. Cong, L. Wang, C. Lin, Y. Song, and Y. Sun.  Finding question-answer pairs from online forums.  In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 467–474. ACM, 2008.
  * [31] Coursera.  https://www.coursera.org/.  Accessed: 2016-11-05.
  * [32] A. Dattolo and F. L. Luccio.  A new concept map model for e-learning environments.  In International Conference on Web Information Systems and Technologies, pages 404–417. Springer, 2008.
  * [33] Jennifer DeBoer, Andrew D Ho, Glenda S Stump, and Lori Breslow.  Changing “course” reconceptualizing educational variables for massive open online courses.  Educational researcher, 43(2):74–84, 2014.
  * [34] H. Drachsler, K. Verbert, O. C. Santos, and N. Manouselis.  Panorama of recommender systems to support learning.  In Recommender systems handbook, pages 421–451. Springer, 2015\.
  * [35] Sidney D’Mello, Blair Lehman, Reinhard Pekrun, and Art Graesser.  Confusion can be beneficial for learning.  Learning and Instruction, 29:153–170, 2014.
  * [36] edX.  https://www.edx.org/.  Accessed: 2016-11-05.
  * [37] J. Eisenstein and R. Barzilay.  Bayesian unsupervised topic segmentation.  In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 334–343. Association for Computational Linguistics, 2008.
  * [38] J. L. Fleiss, B. Levin, and M. C. Paik.  Statistical methods for rates and proportions.  John Wiley & Sons, 2013.
  * [39] C. Flow.  The psychology of optimal experience.  Harper&Row, New York, 1990.
  * [40] W. B. Frakes and C. J. Fox.  Strength and similarity of affix removal stemming algorithms.  In ACM SIGIR Forum, volume 37, pages 26–30. ACM, 2003.
  * [41] Scott Freeman, Sarah L Eddy, Miles McDonough, Michelle K Smith, Nnadozie Okoroafor, Hannah Jordt, and Mary Pat Wenderoth.  Active learning increases student performance in science, engineering, and mathematics.  Proceedings of the National Academy of Sciences, 111(23):8410–8415, 2014.
  * [42] Y. Fujii, K. Yamamoto, N. Kitaoka, and S. Nakagawa.  Class lecture summarization taking into account consecutiveness of important sentences.  In INTERSPEECH, pages 2438–2441. Citeseer, 2008.
  * [43] FutureLearn.  https://www.futurelearn.com/.  Accessed: 2016-11-05.
  * [44] N. Gillani and R. Eynon.  Communication patterns in massively open online courses.  The Internet and Higher Education, 23:18–26, 2014.
  * [45] J. R. Glass, T. J. Hazen, D. S. Cyphers, I. Malioutov, D. Huynh, and R. Barzilay.  Recent progress in the MIT spoken lecture processing project.  In Interspeech, pages 2553–2556, 2007.
  * [46] E. L. Glassman, C. J. Terman, and R. C. Miller.  Learner-sourcing in an engineering class at scale.  In Proceedings of the Second (2015) ACM Conference on Learning at Scale, pages 363–366. ACM, 2015.
  * [47] J. V. Guttag.  Introduction to computation and programming using Python.  Mit Press, 2013.
  * [48] M. Guy, M. d’Aquin, S. Dietze, H. Drachsler, E. Herder, and E. Parodi.  Linkedup: Linking open data for education.  Ariadne, (72), 2014.
  * [49] W. Hämäläinen and M. Vinni.  Comparison of machine learning methods for intelligent tutoring systems.  In International Conference on Intelligent Tutoring Systems, pages 525–534. Springer, 2006.
  * [50] D. Harwath and J. Glass.  Deep multimodal semantic embeddings for speech and images.  In 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pages 237–244. IEEE, 2015.
  * [51] J. Hattie and G. CR Yates.  Visible learning and the science of how we learn.  Routledge, 2013.
  * [52] X. He, R. S. Zemel, and M. Á. Carreira-Perpiñán.  Multiscale conditional random fields for image labeling.  In Computer vision and pattern recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE computer society conference on, volume 2, pages II–695. IEEE, 2004.
  * [53] M. A. Hearst and D. Degler.  Sewing the seams of sensemaking: A practical interface for tagging and organizing saved search results.  In Proceedings of the Symposium on Human-Computer Interaction and Information Retrieval, page 4. ACM, 2013.
  * [54] K. Hirohata, N. Okazaki, S. Ananiadou, and M. Ishizuka.  Identifying sections in scientific abstracts using conditional random fields.  In IJCNLP, pages 381–388, 2008.
  * [55] Gregory W Hislop.  The inevitability of teaching online.  Computer, 42(12):94–96, 2009.
  * [56] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [57] T. Hofmann.  Probabilistic latent semantic indexing.  In Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval, pages 50–57. ACM, 1999.
  * [58] Fiona M Hollands and Devayani Tirthali.  Moocs: Expectations and reality. full report.  Online Submission, 2014.
  * [59] L. Hong and B. D. Davison.  A classification-based approach to question answering in discussion boards.  In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 171–178. ACM, 2009\.
  * [60] Introduction to statistics: descriptive statistics (Stat2.1x).  https://www.edx.org/course/introduction-statistics-descriptive-uc-berkeleyx-stat2-1x/.  Accessed: 2016-11-05.
  * [61] J. Janssen, C. Tattersall, W. Waterink, B. Van den Berg, R. Van Es, C. Bolman, and R. Koper.  Self-organising navigational support in lifelong learning: how predecessors can lead the way.  Computers & Education, 49(3):781–793, 2007.
  * [62] W. S. Junk.  The dynamic balance between cost, schedule, features, and quality in software development projects.  Computer Science Dept., University of Idaho, SEPM-001, 2000.
  * [63] F. Jurcıcek, S. Keizer, M. Gašic, F. Mairesse, B. Thomson, K. Yu, and S. Young.  Real user evaluation of spoken dialogue systems using Amazon mechanical turk.  In Proceedings of INTERSPEECH, volume 11, 2011.
  * [64] S. Kalyuga.  Cognitive load theory: Implications for affective computing.  In Twenty-Fourth International FLAIRS Conference, 2011.
  * [65] D. R. Karger, S. Oh, and D. Shah.  Efficient crowdsourcing for multi-class labeling.  ACM SIGMETRICS Performance Evaluation Review, 41(1):81–92, 2013\.
  * [66] J. Kibler.  Cognitive disequilibrium.  In Encyclopedia of Child Behavior and Development, pages 380–380. Springer, 2011.
  * [67] J. Kim.  Learnersourcing: improving learning with collective learner activity.  PhD thesis, Massachusetts Institute of Technology, 2015.
  * [68] Juho Kim, Philip J Guo, Carrie J Cai, Shang-Wen Li, Krzysztof Z Gajos, and Robert C Miller.  Data-driven interaction techniques for improving navigation of educational videos.  In Proceedings of the 27th annual ACM symposium on User interface software and technology, pages 563–572, 2014.
  * [69] Juho Kim, Shang-Wen Li, Carrie J Cai, Krzysztof Z Gajos, and Robert C Miller.  Leveraging video interaction data and content analysis to improve video learning.  In Proceedings of the CHI 2014 Learning Innovation at Scale workshop, 2014.
  * [70] P. A. Kirschner, J. Sweller, and R. E. Clark.  Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching.  Educational psychologist, 41(2):75–86, 2006.
  * [71] A. Kittur, E. H. Chi, and B. Suh.  Crowdsourcing user studies with mechanical turk.  In Proceedings of the SIGCHI conference on human factors in computing systems, pages 453–456. ACM, 2008.
  * [72] A. Kittur, B. Smus, S. Khamkar, and R. E. Kraut.  Crowdforge: Crowdsourcing complex work.  In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 43–52. ACM, 2011.
  * [73] G. Konidaris, L. P. Kaelbling, and T. Lozano-Perez.  Symbol acquisition for probabilistic high-level planning.  In Proceedings of the Twenty Fourth International Joint Conference on Artificial Intelligence (IJCAI), 2007.
  * [74] I. Kononenko.  Machine learning for medical diagnosis: history, state of the art and perspective.  Artificial Intelligence in medicine, 23(1):89–109, 2001.