<h2>The noise shape</h2>

<p>In order to understand <code>SpatialDropout1D</code>, you should get used to the notion of the <strong>noise shape</strong>. In plain vanilla dropout, each element is kept or dropped independently. For example, if the tensor is <code>[2, 2, 2]</code>, each of 8 elements can be zeroed out depending on random coin flip (with certain "heads" probability); in total, there will be 8 independent coin flips and any number of values may become zero, from <code>0</code> to <code>8</code>.</p>

<p>Sometimes there is a need to do more than that. For example, one may need to drop the <em>whole slice</em> along <code>0</code> axis. The <code>noise_shape</code> in this case is <code>[1, 2, 2]</code> and the dropout involves only 4 independent random coin flips. The first component will either be kept together or be dropped together. The number of zeroed elements can be <code>0</code>, <code>2</code>, <code>4</code>, <code>6</code> or <code>8</code>. It cannot be <code>1</code> or <code>5</code>.</p>

<p>Another way to view this is to imagine that input tensor is in fact <code>[2, 2]</code>, but each value is double-precision (or multi-precision). Instead of dropping the bytes in the middle, the layer drops the full multi-byte value.</p>

<h2>Why is it useful?</h2>

<p>The example above is just for illustration and isn't common in real applications. More realistic example is this: <code>shape(x) = [k, l, m, n]</code> and <code>noise_shape = [k, 1, 1, n]</code>. In this case, each batch and channel component will be kept independently, but each row and column will be kept or not kept together. In other words, the <em>whole</em> <code>[l, m]</code> <em>feature map</em> will be either kept or dropped.</p>

<p>You may want to do this to account for adjacent pixels correlation, especially in the early convolutional layers. Effectively, you want to prevent co-adaptation of pixels with its neighbors across the feature maps, and make them learn as if no other feature maps exist. This is exactly what <code>SpatialDropout2D</code> is doing: it promotes independence between feature maps.</p>

<p>The <code>SpatialDropout1D</code> is very similar: given <code>shape(x) = [k, l, m]</code> it uses <code>noise_shape = [k, 1, m]</code> and drops entire 1-D feature maps.</p>

<p>Reference: <a href="https://arxiv.org/abs/1411.4280" rel="noreferrer">Efficient Object Localization Using Convolutional Networks</a> 
by Jonathan Tompson at al.</p>
How to understand SpatialDropout1D and when to use it?
<p>Occasionally I see some models are using <code>SpatialDropout1D</code> instead of <code>Dropout</code>. For example, in the Part of speech tagging neural network, they use:</p>



<pre class="lang-python prettyprint-override"><code>model = Sequential()
model.add(Embedding(s_vocabsize, EMBED_SIZE,
                    input_length=MAX_SEQLEN))
model.add(SpatialDropout1D(0.2)) ##This
model.add(GRU(HIDDEN_SIZE, dropout=0.2, recurrent_dropout=0.2))
model.add(RepeatVector(MAX_SEQLEN))
model.add(GRU(HIDDEN_SIZE, return_sequences=True))
model.add(TimeDistributed(Dense(t_vocabsize)))
model.add(Activation("softmax"))
</code></pre>

<p>According to Keras' documentation, it says:</p>

<blockquote>
  <p>This version performs the same function as Dropout, however it drops
  entire 1D feature maps instead of individual elements.</p>
</blockquote>

<p>However, I am unable to understand the meaning of <strong>entrie 1D feature</strong>. More specifically, I am unable to visualize <code>SpatialDropout1D</code> in the same model explained in <a href="https://www.quora.com/How-does-the-dropout-method-work-in-deep-learning-And-why-is-it-claimed-to-be-an-effective-trick-to-improve-your-network" rel="noreferrer">quora</a>.
Can someone explain this concept by using the same model as in quora?</p>

<p>Also, under what situation we will use <code>SpatialDropout1D</code> instead of <code>Dropout</code>?</p>