I was going to follow the steps mentioned by jaradc: - if new_loss - previous_loss is ridiculous (however you want to define ridiculous) - step back to previous_loss iteration and reduce learning_rate by some amount