<p>I think your learning rate LR (default for adam : 0.001) is too high, leading to <code>catastrophic forgetting</code> . <strong>Refer: How to Fine-Tune BERT for Text Classification?</strong> <a href="https://arxiv.org/pdf/1905.05583.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1905.05583.pdf</a></p>
<p>Ideally, LR should be of the order of e-5 . Try changing the code as follows and it should work</p>
<pre><code>from keras_radam import RAdam

model.compile(
      RAdam(lr=2e-5),
      loss='binary_crossentropy',
      metrics=['accuracy'],
)
</code></pre>
How to make BERT model converge?
<p>I am trying to use BERT for sentiment analysis but I suspect I am doing something wrong. In my code I am fine tuning bert using <code>bert-for-tf2</code> but after 1 epoch I am getting an accuracy of 42% when a simple GRU model was getting around 73% accuracy. What should I be doing different to effectively use BERT. I suspect I am traning the bert layers from the first batch which may be an issue as the dense layer is randomly initialized. Any advice would be appreciated, Thanks!</p>

<pre><code>import bert-for-tf2 #gets imported as bert but relabeled for clarity
model_name = "uncased_L-12_H-768_A-12"
model_dir = bert.fetch_google_bert_model(model_name, ".models")
model_ckpt = os.path.join(model_dir, "bert_model.ckpt")

bert_params = bert.params_from_pretrained_ckpt(model_dir)
l_bert = bert.BertModelLayer.from_params(bert_params, name="bert")
max_seq_len = 100
l_input_ids = tensorflow.keras.layers.Input(shape=(max_seq_len,), dtype='int32')

bertLayer = l_bert(l_input_ids)
flat = Flatten()(bertLayer)
output = Dense(1,activation = 'sigmoid')(flat)
model = tensorflow.keras.Model(inputs=l_input_ids, outputs=output)
model.build(input_shape=(None, max_seq_len))

bert.load_bert_weights(l_bert, model_ckpt) 

with open('../preprocessing_scripts/new_train_data.txt', 'r') as f:
  tweets = f.readlines()

with open('../preprocessing_scripts/targets.csv', 'r') as f:
  targets = f.readlines()

max_words = 14000
tokenizer = Tokenizer(num_words=max_words)

trainX = tweets[:6000]
trainY = targets[:6000]
testX = tweets[6000:]
testY = tweets[6000:]
maxlen = 100
tokenizer.fit_on_texts(trainX)

tokenized_version = tokenizer.texts_to_sequences(trainX)

tokenized_version = pad_sequences(tokenized_version, maxlen=maxlen)trainY = np.array(trainY,dtype = 'int32')
model.compile(loss="binary_crossentropy",
              optimizer="adam",
              metrics=['accuracy']) 

history = model.fit(x=tokenized_version, y=trainY, batch_size = 32, epochs=1, validation_split = 0.2)
</code></pre>