They suggest that further supervised pretraining of models can provide significant benefits.
they propose supervised pretraining on contact prediction and remote homology detection, and show it increases performance on secondary structure prediction.