ranges from 0 to 990 in a gap of 5. Once a user solves each exercise, *Santa* provides educational feedback to his response such as explanations or commentaries on exercises. *EdNet* is the collection of user interactions of multiple-choice exercises collected over the last four years. The main features of the user-exercise interaction data consists of six columns: user id, exercise id, user response, exercise part, received time and time taken. We describe each column as the following. Firstly, the user (resp. exercise) ID identifies each unique user (resp. exercise). The user response is recorded as 1 if the user response is correct and 0 otherwise. The part the exercise belongs to is recorded as exercise part. Finally, the absolute time when the user received the exercise and the time taken by user to respond are recorded respectively. In the dataset, 627,347 users solved more than one problem. The size of the exercise set is 16,175. The total row count of the dataset is 72,907,005.

### Dataset for Pre-training

For pre-training, we first reconstruct the interaction timeline of each user by gathering the responses of a specific user in the increasing order of absolute time received. For each interaction $I_t$, 