title: Implementing a saturating autoencoder in theano
body: <p>I am trying to implement an autoencoder, using the regularization method described in this paper: <a href="http://yann.lecun.com/exdb/publis/pdf/goroshin-lecun-iclr-13.pdf" rel="nofollow">"Saturating Auto-Encoders", Goroshin et. al., 2013</a></p>
  
  <p>Essentially, this tries to minimize the difference between the output of the hidden layer, and the flat portions of the non-linear function being used to compute the hidden layer output.</p>
  
  <p>Assuming we are using a step function as the nonlinearity, with the step being at 0.5, a simple implementation might be:</p>
  
  <pre><code>for i in range(len(y)):
      if y[i] &lt; 0.5:
          y_prime[i] = 0
      else:
          y_prime[i] = 1
  </code></pre>
  
  <p>Then, the regularization cost can be simply:</p>
  
  <pre><code>(numpy.abs(y-y_prime).sum()
  </code></pre>
  
  <p>I am trying to implement this functionality in Theano. Have started off with the <a href="http://deeplearning.net/tutorial/dA.html" rel="nofollow">denoising autoencoder code</a> available at the Theano website. Have made some basic modifications to it:</p>
  
  <pre><code>def get_cost_updates(self, learning_rate, L1_reg):
      # I do not want to add noise as of now, hence no input corruption.
      # Directly compute the hidden layer values.
      y = self.get_hidden_values(self.x)
      z = self.get_reconstructed_input(y)
  
      # Also, the original code computes the cross entropy loss function.
      # I want to use the quadratic loss as my inputs are real valued, not
      # binary. Further, I have added an L1 regularization term to the hidden
      # layer values.
  
      L = 0.5*T.sum((self.x-z)**2, axis=1) + L1_reg*(abs(y).sum())
      ... # Rest of it is same as original.
  </code></pre>
  
  <p>The above loss functions puts an L1 penalty on the hidden layer output, which should (hopefully) drive most of them to 0. In place of this simple L1 penalty, I want to use the saturating penalty as given above.</p>
  
  <p>Any idea how to do this? Where do I compute the y_prime? How to do it symbolically?</p>
  
  <p>I am a newbie to Theano, and still catching up with the symbolic computation part.</p>