<p>So I also found the Tensorflow documentation on weight pruning to be quite <em>sparse</em>, so I spent some quality time with the debugger to figure out how everything works.
<br><br></p>

<h1>How Pruning Schedules Work</h1>

<p>At the most basic level, the Pruning Schedule is simply a function that takes the step as an input and produces a sparsity percentage. That sparsity value is then used to generate a mask, which is used to prune out weights with an absolute value less than then the <em>k - 1</em> value given by the absolute value weight distribution and the sparsity percentage.</p>

<h1>PolynomialDecay</h1>

<p>Class definition: <a href="https://github.com/tensorflow/model-optimization/blob/c2642e5de64bb7709310bd7775de84b4765b359a/tensorflow_model_optimization/python/core/sparsity/keras/pruning_schedule.py#L183" rel="noreferrer">Github Link</a><br>
The comments included with the class definition above helped me understand how the PolynomialDecay scheduler works.</p>

<blockquote>
  <p>Pruning rate grows rapidly in the beginning from initial_sparsity, but then
      plateaus slowly to the target sparsity. </p>
  
  <p>The function applied is</p>
  
  <p>current_sparsity = final_sparsity + (initial_sparsity - final_sparsity)
            * (1 - (step - begin_step)/(end_step - begin_step)) ^ exponent</p>
</blockquote>

<p>By the above equation, when <code>step == begin_step</code> then <code>current_sparsity = initial_sparsity</code>. Thus, the weights will be pruned to the <code>initial_sparsity</code> on the step specified by the <code>begin_step</code> parameter.
<br><br>
I would agree with your assessment, in that you would usually want to start pruning at a lower sparsity than 50%, but I do not have any published research I can cite to backup that claim. You may be able to find more information in the <a href="https://arxiv.org/abs/1710.01878" rel="noreferrer">paper</a> cited with the PolynomialDecay class definition, although I have not had a chance to read it myself.</p>

<h1>ConstantSparsity</h1>

<p>Class definition: <a href="https://github.com/tensorflow/model-optimization/blob/c2642e5de64bb7709310bd7775de84b4765b359a/tensorflow_model_optimization/python/core/sparsity/keras/pruning_schedule.py#L137" rel="noreferrer">Github Link</a><br>
The purpose of this scheduler appears to be pretty limited. With every valid prune step, the <code>target_sparsity</code> is returned. As such, multiple pruning steps are very much redundant. The use case for this scheduler appears to be for a one time prune during training. The ability to prune with this scheduler multiple times is to align it with its parent abstract class and other pruning schedulers.</p>

<h1>Creating Your Own Pruning Scheduler</h1>

<p>If the two above schedulers do not float your boat, the abstract class <a href="https://github.com/tensorflow/model-optimization/blob/c2642e5de64bb7709310bd7775de84b4765b359a/tensorflow_model_optimization/python/core/sparsity/keras/pruning_schedule.py#L23" rel="noreferrer">PruningSchedule</a> exposes an endpoint which makes it very easy to create your own pruning scheduler, as convoluted as it may be. Below is an example I created myself.<br><br>
Disclaimer: this scheduler is a creation of a 19 year-old college student's imagination and has no basis in any published literature. </p>

<pre><code>PruningSchedule = tfmot.sparsity.keras.PruningSchedule

class ExponentialPruning(PruningSchedule):
def __init__(self, rate=0.01, begin_step=0, frequency=100, max_sparsity=0.9):
    self.rate = rate
    self.begin_step = begin_step
    self.frequency = frequency
    self.max_sparsity = max_sparsity

    # Validation functions provided by the parent class
    # The -1 parameter is for the end_step
    # as this pruning schedule does not have one
    # The last true value is a boolean flag which says it is okay
    # to have no end_step
    self._validate_step(self.begin_step, -1, self.frequency, True)
    self._validate_sparsity(self.max_sparsity, 'Max Sparsity')

def __call__(self, step):
    # Sparsity calculation endpoint

    # step is a integer tensor

    # The sparsity returned by __call__ must be a tensor
    # of dtype=tf.float32, so tf.math is required.

    # In the logic below, you can assume that a valid
    # pruning step is passed.

    p = tf.math.divide(
        tf.cast(step - self.begin_step, tf.float32),
        tf.constant(self.frequency, dtype=tf.float32)
    )
    sparsity = tf.math.subtract(
        tf.constant(1, dtype=tf.float32),
        tf.math.pow(
            tf.constant(1 - self.rate, dtype=tf.float32),
            p
        )
    )

    sparsity = tf.cond(
        tf.math.greater(sparsity, tf.constant(self.max_sparsity, dtype=tf.float32)),
        lambda: tf.constant(self.max_sparsity, dtype=tf.float32),
        lambda: sparsity
    )

    # This function returns a tuple of length 2
    # The first value determines if pruning should occur on this step
    # I recommend using the parent class function below for this purpose
    # The negative one value denotes no end_step
    # The second value is the sparsity to prune to
    return (self._should_prune_in_step(step, self.begin_step, -1, self.frequency),
            sparsity)

def get_config(self):
    # A function required by the parent class
    # return the class_name and the input parameters as
    # done below
    return {
        'class_name': self.__class__.__name__,
        'config': {
            'rate': self.rate,
            'begin_step': self.begin_step,
            'frequency': self.frequency,
            'max_sparsity': self.max_sparsity
        }
    }
</code></pre>

<h1>Using a Pruning Scheduler</h1>

<p>If you only would like certain layers to be pruned, rather than all prunable layers, you can call the <code>prune_low_magnitude</code> function on a layer which you are adding to your model.</p>

<pre><code>prune_low_magnitude = tfmot.sparsity.keras.prune_low_magnitude
model = keras.models.Sequential()
...
model.add(prune_low_magnitude(keras.layers.Dense(8, activation='relu', kernel_regularizer=keras.regularizers.l1(0.0001)),
  ExponentialPruning(rate=1/8)))
</code></pre>

<p>Also make sure to pass a <code>UpdatePruningStep</code> instance to the training callbacks:</p>

<pre><code>m.fit(train_input, train_labels, epochs=epochs, validation_data=[test_input, test_labels],
  callbacks=[UpdatePruningStep()])
</code></pre>
initial_sparsity parameter in sparsity.PolynomialDecay() TensorFlow 2.0 magnitude-based weight pruning
<p>I was trying the tutorial <a href="https://www.tensorflow.org/model_optimization/guide/pruning/pruning_with_keras#train_a_pruned_mnist" rel="noreferrer">TensorFlow 2.0 Magnitude-based weight pruning with Keras</a>
and came across the parameter <em>initial_sparsity</em></p>

<pre><code>import tensorflow_model_optimization as tfmot
from tensorflow_model_optimization.sparsity import keras as sparsity
import numpy as np

epochs = 12
num_train_samples = x_train.shape[0]
end_step = np.ceil(1.0 * num_train_samples / batch_size).astype(np.int32) * epochs
print('End step: ' + str(end_step))

pruning_params = {
      'pruning_schedule': sparsity.PolynomialDecay(initial_sparsity=0.50,
                                                   final_sparsity=0.90,
                                                   begin_step=2000,
                                                   end_step=end_step,
                                                   frequency=100)
}
</code></pre>

<p>The tutorial says:</p>

<blockquote>
  <p>The parameter used here means:</p>
  
  <p><strong>Sparsity</strong> PolynomialDecay is used across the whole training process. We start at the sparsity level 50% and gradually train the
  model to reach 90% sparsity. X% sparsity means that X% of the weight
  tensor is going to be pruned away.</p>
</blockquote>

<p>My question is, shouldn't you start with <strong>initial_sparsity</strong> of 0% and then prune 90% of the weights off?</p>

<p>What does starting with <strong>initial_sparsity</strong> of 50% mean? Does this mean that 50% of the weights are pruned to begin with and then 90% sparsity of pruning is achieved?</p>

<p>Also, for <strong>tfmot.sparsity.keras.ConstantSparsity</strong>, the API is as follows:</p>

<pre><code>pruning_params_unpruned = {
    'pruning_schedule': sparsity.ConstantSparsity(
        target_sparsity=0.0, begin_step=0,
        end_step = 0, frequency=100
    )
}
</code></pre>

<blockquote>
  <p>Initializes a Pruning schedule with constant sparsity.</p>
  
  <p>Sparsity is applied in the interval [begin_step, end_step] every
  frequency steps. At each applicable step, the sparsity(%) is constant.</p>
</blockquote>

<p>Does this mean that if a neural network model is already at a sparsity level of 50%, but the <em>target_sparsity = 0.5</em> then will the pruning schedule do:</p>

<ol>
<li>No pruning, since the model is already at a pruned level of 50%</li>
<li>It further prunes 50% of the weights of the already (50% pruned) model</li>
</ol>

<p>You can read about it in <a href="https://www.tensorflow.org/model_optimization/api_docs/python/tfmot/sparsity/keras/PolynomialDecay" rel="noreferrer">PolynomialDecay</a> and in <a href="https://www.tensorflow.org/model_optimization/api_docs/python/tfmot/sparsity/keras/ConstantSparsity" rel="noreferrer">ConstantSparsity</a></p>

<p>Thanks</p>