the open-ended questions of *which* assessments to pre-train for label-scarce educational problems and *how* to pre-train multiple assessments. In Section \[sec:experiments\], we explore these questions for exam score (a non-interactive educational feature) and review correctness (a sporadic assessment) respectively. Experimental results support that assessment modeling are effective pre-training tasks for the label-scarce educational problems.

Assessment Modeling with Deep Bidirectional Transformer Encoder
---------------------------------------------------------------

![image](figure-Bert.pdf){width="70.00000%"}

While there are several possible options for the architecture of the assessment model, we adopt the deep bidirectional Transformer encoder proposed in [@devlin-etal-2019-bert] for the following reasons. First, [@pandey2019self] showed that the self-attention mechanism in Transformer [@vaswani_2017] is effective for knowledge tracing task. The Transformer-based knowledge tracing model proposed in [@pandey2019self] achieved state-of-the-art performance on several datasets. Second, the deep bidirectional Transformer encoder model and pre-train/fine-tune method proposed in [@devlin-etal-2019-bert] achieved state-of-the-art results on several NLP tasks. While [@devlin-etal-2019-bert] conducted experimental studies on NLP domain, the method is also applicable to other domains with slight modifications.

Figure \[fig:am\_model\] depicts our proposed pre-train/fine-tune approach. In the pre-training phase, we train a deep bidirectional Transformer encoder based assessment model to predict the assessments conditioned on past and future interactions. After the pre-training phase, we replace the last layer of 