positive outcome (died within four years from the date of filling the questionnaire). To avoid bias toward the large class, in each training episode we sample a patient from one of the classes with equal probability (i.e., we over-sample the small class), so that roughly similar number of patients from each class are seen during training.

**Alternating training** To improve the training stability in our non-stationary setting, we trained the guesser and the agent intermittently, switching between them once in 1000 episodes. This way, during each such 1000 episodes period, when the agent is being trained, the guesser network remains fixed, so that the MDP is in fact (“locally” ) stationary.

**Pre-training the guesser network** We pre-train the guesser G as a classifier, where all features are visible. When setting-up the environment, we initialized the guesser network using the parameters of the pre-trained guesser.

**Early Stopping** Unlike typical RL works, we know the labels of the training data, which allows us to dedicate a portion of the data for validation set and apply an early stopping mechanism. Specifically, every 1000 training episodes we run our agent on the validation set and record its AUC. Training stops when no significant improvements 