<p>This paper (<a href="https://arxiv.org/pdf/1903.07288.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1903.07288.pdf</a>) studied the effect of padding types on LSTM and CNN. They found that post-padding achieved substantially lower accuracy (nearly half) compared to pre-padding in LSTMs, although there wasn't a significant difference for CNNs (post-padding was only slightly worse).</p>
<p>A simple/intuitive explanation for RNNs is that, post-padding seems to add noise to what has been learned from the sequence through time, and there aren't more timesteps for the RNN to recover from this noise. With pre-padding, however, the RNN is better able to adjust to the added noise of zeros at the beginning as it learns from the sequence through time.</p>
<p>I think more thorough experiments are needed in the community for more detailed mechanistic explanations on how padding affects performance.</p>
<p>I always recommend using pre-padding over post-padding, even for CNNs, unless the  problem specifically requires post-padding.</p>
How does choosing between pre and post zero padding of sequences impact results
<p>I'm working on an NLP sequence labelling problem. My data consists of variable length sequences <code>(w_1, w_2, ..., w_k)</code> with corresponding labels <code>(l_1, l_2, ..., l_k)</code> (in this case the task is named entity extraction).</p>

<p>I intend to solve the problem using Recurrent Neural Networks. As the sequences are of variable length I need to pad them (I want batch size >1). I have the option of either pre zero padding them, or post zero padding them. I.e. either I make every sequence <code>(0, 0, ..., w_1, w_2, ..., w_k)</code> or <code>(w_1, w_2, ..., w_k, 0, 0, ..., 0)</code> such that the lenght of each sequence is the same. </p>

<p><strong>How does the choice between pre- and post padding impact results?</strong> </p>

<p>It seems like pre padding is more common, but I can't find an explanation of why it would be better. Due to the nature of RNNs it feels like an arbitrary choice for me, since they share weights across time steps.</p>