<p>There different methods for summarizing a text i.e. Extractive &amp; Abstractive.</p>
<blockquote>
<p><strong>Extractive summarization</strong> means identifying important sections of the text and generating them verbatim producing a subset of the
sentences from the original text; while <strong>abstractive summarization</strong>
reproduces important material in a new way after interpretation and
examination of the text using advanced natural language techniques to
generate a new shorter text that conveys the most critical information
from the original one.</p>
</blockquote>
<p>For a transformer based approach you just need an additional attention layer which you can add to an encoder-decoder model or you can use pre-trained transformers (fine tune them maybe) like BERT, GPT, T5, etc.</p>
<p>You can have a look at : <a href="https://huggingface.co/transformers/" rel="nofollow noreferrer">https://huggingface.co/transformers/</a></p>
<p>For Abstractive Summarization T5 works pretty well. Here's a nice and simple example : <a href="https://github.com/faiztariq/FzLabs/blob/master/abstractive-text-summarization-t5.ipynb" rel="nofollow noreferrer">https://github.com/faiztariq/FzLabs/blob/master/abstractive-text-summarization-t5.ipynb</a></p>
<p>For Extractive Summarization you may take a look at : <a href="https://pypi.org/project/bert-extractive-summarizer/" rel="nofollow noreferrer">https://pypi.org/project/bert-extractive-summarizer/</a></p>
<p>There's a paper (<strong>Attention Is All You Need</strong>) that explains transformers pretty well, you may also take a look at it : <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">https://arxiv.org/abs/1706.03762</a></p>
BERT for Text Summarization
<p>I'm trying to build a text summarization model using seq2seq architecture in Keras. I've followed this tutorial <a href="https://keras.io/examples/lstm_seq2seq/" rel="nofollow noreferrer">https://keras.io/examples/lstm_seq2seq/</a> and implemented it with Embeddings layer, which works fine. But now I want to use BERT. Can pretrained BERT embeddings be used in such a task, usually I see text classifiation, but not the encoder-decoder architecture used with BERT.</p>

<p>I access BERT model from TF Hub, and have a Layer class implemented from this tutorial <a href="https://github.com/strongio/keras-bert/blob/master/keras-bert.ipynb" rel="nofollow noreferrer">https://github.com/strongio/keras-bert/blob/master/keras-bert.ipynb</a>, I also tokenize accordingly with BERT tokenizer, below is my model</p>

<pre class="lang-py prettyprint-override"><code>enc_in_id = Input(shape=(None, ), name="Encoder-Input-Ids")
enc_in_mask = Input(shape=(None, ), name="Encoder-Input-Masks")
enc_in_segment = Input(shape=(None, ), name="Encoder-Input-Segment-Ids")
bert_encoder_inputs = [enc_in_id, enc_in_mask, enc_in_segment]

encoder_embeddings = BertLayer(name='Encoder-Bert-Layer')(bert_encoder_inputs)
encoder_embeddings = BatchNormalization(name='Encoder-Batch-Normalization')(encoder_embeddings)
encoder_lstm = LSTM(latent_size, return_state=True, name='Encoder-LSTM')
encoder_out, e_state_h, e_state_c = encoder_lstm(encoder_embeddings)
encoder_states = [e_state_h, e_state_c]

dec_in_id = Input(shape=(None,), name="Decoder-Input-Ids")
dec_in_mask = Input(shape=(None,), name="Decoder-Input-Masks")
dec_in_segment = Input(shape=(None,), name="Decoder-Input-Segment-Ids")
bert_decoder_inputs = [dec_in_id, dec_in_mask, dec_in_segment]

decoder_embeddings_layer = BertLayer(name='Decoder-Bert-Layer')
decoder_embeddings = decoder_embeddings_layer(bert_decoder_inputs)
decoder_batchnorm_layer = BatchNormalization(name='Decoder-Batch-Normalization-1')
decoder_batchnorm = decoder_batchnorm_layer(decoder_embeddings)

decoder_lstm = LSTM(latent_size, return_state=True, return_sequences=True, name='Decoder-LSTM')
decoder_out, _, _ = decoder_lstm(decoder_batchnorm, initial_state=encoder_states)
dense_batchnorm_layer = BatchNormalization(name='Decoder-Batch-Normalization-2')
decoder_out_batchnorm = dense_batchnorm_layer(decoder_out)
decoder_dense_id = Dense(vocabulary_size, activation='softmax', name='Dense-Id')
dec_outputs_id = decoder_dense_id(decoder_out_batchnorm)
</code></pre>

<p>The model builds and after a couple of epochs accuracy rises to 1, and loss drops below 0.5, but the predictions are awful. Since I'm working on a dev set comprised of 5 samples, with max 30 WordPiece tokens and predicting on the same data, I only get the first or maybe two tokens right, then it just repeats the last seen token, or [PAD] token.</p>