title: Training a BERT and Running out of memory - Google Colab
body: <p>I keep running out of memory even after i bought google colab pro which has 25gb RAM usage. I have no idea why is this happening. I tried every kernel possible (Google colab, Google colab pro, Kaggle kernel, Amazon Sagemaker, Google Cloud Platform). I reduced my batch size to 8, no success whatsoever.</p>
  <p>My goal is to train Bert in Deep Pavlov (with Russian text classification extension) to predict emotion of the tweet. It is a multiclass classification with 5 classes</p>
  <p>Here is my whole code:</p>
  <pre><code>!pip3 install deeppavlov
  import pandas as pd
  train_df = pd.read_csv('train_pikabu.csv')
  test_df = pd.read_csv('test_pikabu.csv')
  val_df = pd.read_csv('validation_pikabu.csv')
  
  from deeppavlov.dataset_readers.basic_classification_reader import BasicClassificationDatasetReader
  # read data from particular columns of `.csv` file
  data = BasicClassificationDatasetReader().read(
     data_path='./',
     train='train_pikabu.csv',
     valid=&quot;validation_pikabu_a.csv&quot;, 
     test=&quot;test_pikabu.csv&quot;,
     x = 'content',
     y = 'emotions'
     )
  
  from deeppavlov.dataset_iterators.basic_classification_iterator import 
  BasicClassificationDatasetIterator
  # initializing an iterator
  iterator = BasicClassificationDatasetIterator(data, seed=42, shuffle=True)
  
  !python -m deeppavlov install squad_bert
  from deeppavlov.models.preprocessors.bert_preprocessor import BertPreprocessor
  bert_preprocessor = BertPreprocessor(vocab_file=&quot;./bert/vocab.txt&quot;,
                                   do_lower_case=False,
                                   max_seq_length=256)
  
  from deeppavlov.core.data.simple_vocab import SimpleVocabulary
  vocab = SimpleVocabulary(save_path=&quot;./binary_classes.dict&quot;)
  iterator.get_instances(data_type=&quot;train&quot;)
  vocab.fit(iterator.get_instances(data_type=&quot;train&quot;)[1])
  
  from deeppavlov.models.preprocessors.one_hotter import OneHotter
  one_hotter = OneHotter(depth=vocab.len, 
                     single_vector=True  # means we want to have one vector per sample
                    )
  
  from deeppavlov.models.classifiers.proba2labels import Proba2Labels
  prob2labels = Proba2Labels(max_proba=True)
  
  from deeppavlov.models.bert.bert_classifier import BertClassifierModel
  from deeppavlov.metrics.accuracy import sets_accuracy
  
  bert_classifier = BertClassifierModel(
   n_classes=vocab.len,
   return_probas=True,
   one_hot_labels=True,
   bert_config_file=&quot;./bert/bert_config.json&quot;,
   pretrained_bert=&quot;./bert/bert_model.ckpt&quot;,
   save_path=&quot;sst_bert_model/model&quot;,
   load_path=&quot;sst_bert_model/model&quot;,
   keep_prob=0.5,
   learning_rate=1e-05,
   learning_rate_drop_patience=5,
   learning_rate_drop_div=2.0
   )
  
  
   # Method `get_instances` returns all the samples of particular data field
   x_valid, y_valid = iterator.get_instances(data_type=&quot;valid&quot;)
   # You need to save model only when validation score is higher than previous one.
   # This variable will contain the highest accuracy score
   best_score = 0.
   patience = 2
   impatience = 0
  
   # let's train for 3 epochs
   for ep in range(3):
  
       nbatches = 0
       for x, y in iterator.gen_batches(batch_size=8, 
                                   data_type=&quot;train&quot;, shuffle=True):
          x_feat = bert_preprocessor(x)
          y_onehot = one_hotter(vocab(y))
          bert_classifier.train_on_batch(x_feat, y_onehot)
          print(&quot;Batch done\n&quot;)
          nbatches += 1
      
          if nbatches % 1 == 0:
              # validating every 100 batches
              y_valid_pred = bert_classifier(bert_preprocessor(x_valid))
              score = sets_accuracy(y_valid, vocab(prob2labels(y_valid_pred)))
              print(&quot;Batches done: {}. Valid Accuracy: {}&quot;.format(nbatches, score))
          
       y_valid_pred = bert_classifier(bert_preprocessor(x_valid))
       score = sets_accuracy(y_valid, vocab(prob2labels(y_valid_pred)))
       print(&quot;Epochs done: {}. Valid Accuracy: {}&quot;.format(ep + 1, score))
       if score &gt; best_score:
            bert_classifier.save()
            print(&quot;New best score. Saving model.&quot;)
            best_score = score    
            impatience = 0
       else:
          impatience += 1
          if impatience == patience:
               print(&quot;Out of patience. Stop training.&quot;)
               break
  </code></pre>
  <p>It runs up to 1 batch and then crushes.</p>