<ol>
<li>Do we need to handle <code>row_pooling_sequence</code> and <code>col_pooling_sequence</code>?</li>
</ol>

<p>Even though the <a href="https://www.tensorflow.org/versions/r0.11/api_docs/python/nn.html#fractional_max_pool" rel="noreferrer"><code>tf.nn.fractional_max_pool</code> documentation</a> says it turns 2 extra tensors which are needed to calculate gradient, I believe we <strong>do not</strong> need to specially handle these 2 extra tensors and add them into gradient calculation operation. The backpropagation of <code>tf.nn.fractional_max_pool</code>in TensorFlow is already registered into the gradient calculation flow by the <a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/nn_grad.py" rel="noreferrer"><code>_FractionalMaxPoolGrad</code> function</a>. As you can see in the <code>_FractionalMaxPoolGrad</code>, the <code>row_pooling_sequence</code> and <code>col_pooling_sequence</code> are extracted by <code>op.outputs[1]</code> and <code>op.outputs[2]</code> and used to calculate gradient. </p>

<pre><code>@ops.RegisterGradient("FractionalMaxPool")
def _FractionalMaxPoolGrad(op, grad_0, unused_grad_1, unused_grad_2):
  """..."""
  return gen_nn_ops._fractional_max_pool_grad(op.inputs[0], op.outputs[0],
                                              grad_0, op.outputs[1],
                                              op.outputs[2],
                                              op.get_attr("overlapping"))
</code></pre>

<ol start="2">
<li>Possible reasons for poorer performance after using <code>fractional_max_pool</code> (in my personal opinions). </li>
</ol>

<p>In the <a href="https://arxiv.org/pdf/1412.6071v4.pdf" rel="noreferrer">fractional max pooling paper</a>, the author used fractional max pooling in a <a href="https://arxiv.org/pdf/1409.6070v1.pdf" rel="noreferrer">spatially-sparse convolutional network</a>. According to his spatially-sparse convolutional network design, he actually extended the image input spatial size by padding zeros. Additionally, fractional max pooling downsizes the input by a factor of <code>pooling_ratio</code> which is often less than 2. These two combined together allowed stacking more convolutional layers than using regular max pooling and hence building a deeper network. (i.e. imagine using CIFAR-10 dataset, the (non-padding) input spatial size is 32x32, the spatial size drops to 4x4 after 3 convolutional layers and 3 max pooling operations. If using fractional max pooling with <code>pooling_ratio=1.4</code>, the spatial size drops to 4x4 after 6 convolutional and 6 fractional max pooling layers). I experimented with building a CNN with 2-conv-layer+2-pooling-layer(regular max pool vs. fractional max pool with <code>pooling_ratio=1.47</code>)+2-fully-connected-layer on MNIST dataset. The one using regular max pooling also produced a better performance than the one using fractional max pooling (down by 15~20% performance). Comparing the spatial size before feeding into fully connected layers, the model with regular max pooling has spatial size of 7x7, the one with fractional max pooling has spatial size of 12x12. Adding one more conv+fractional_max_pool into the latter model (final spatial size dropped to be 8x8) improved the performance to a more comparative level with the former model with regular max pooling. </p>

<p>In summary, I personally think the good performance in the Fractional Max-Pooling paper is achieved by a combination of using spatially-sparse CNN with fractional max-pooling and small filters (and network in network) which enable building a deep network even when the input image spatial size is small. Hence in regular CNN network, simply replace regular max pooling with fractional max pooling does not necessarily give you a better performance.</p>
Fractional Max Pooling in Tensorflow
<p>When using the function <code>tf.nn.fractional_max_pool</code> in Tensorflow, in addition to the output pooled tensor it returns, it also returns a <code>row_pooling_sequence</code> and a <code>col_pooling_sequence</code>, which I presume is used in backpropagation to find the gradient of. This is in contrast to the normal $2 \times 2$ max pooling, which just returns the pooled tensor.</p>

<p>My question is: do we have to handle the row_pooling and col_pooling values ourselves? How would we include them into a network to get backpropagation working properly? I modified a simple convolutional neural network to use fractional max pooling instead of 2 x 2 max pooling without making use of these values and the results were much poorer, leading me to believe we must explicitly handle these. </p>

<p>Here's the relevant portion of my code that makes use of the FMP:</p>

<pre><code>def add_layer_ops_FMP(conv_func, x_input, W, keep_prob_layer, training_phase):

    h_conv = conv_func(x_input, W, stride_l = 1)
    h_BN = batch_norm(h_conv, training_phase, epsilon)
    h_elu = tf.nn.elu(h_BN) # Rectified unit layer - change accordingly

    def dropout_no_training(h_elu=h_elu):
        return dropout_op(h_elu, keep_prob = 1.0)

    def dropout_in_training(h_elu=h_elu, keep_prob_layer=keep_prob_layer):
        return dropout_op(h_elu, keep_prob = keep_prob_layer)

    h_drop = tf.cond(training_phase, dropout_in_training, dropout_no_training)
    h_pool, row_pooling_sequence, col_pooling_sequence = tf.nn.fractional_max_pool(h_drop) # FMP layer. See Ben Graham's paper 

    return h_pool
</code></pre>

<p><a href="https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/api_docs/python/functions_and_classes/shard3/tf.nn.fractional_max_pool.md" rel="nofollow">Link to function on github.</a></p>