title: Tensorflow custom activation function
body: <p>I implemented a network with TensorFlow and created the model doing the following in my code:</p>
  
  <pre><code>def multilayer_perceptron(x, weights, biases):
      layer_1 = tf.add(tf.matmul(x, weights["h1"]), biases["b1"])
      layer_1 = tf.nn.relu(layer_1)
      out_layer = tf.add(tf.matmul(layer_1, weights["out"]), biases["out"])
      return out_layer
  </code></pre>
  
  <p>I initialize the weights and the biases doing:</p>
  
  <pre><code>weights = {
      "h": tf.Variable(tf.random_normal([n_input, n_hidden_1])),
      "out": tf.Variable(tf.random_normal([n_hidden_1, n_classes]))
      }
  
  biases = {
      "b": tf.Variable(tf.random_normal([n_hidden_1])),
      "out": tf.Variable(tf.random_normal([n_classes]))
      }
  </code></pre>
  
  <p>Now I want to use a custom activation function. Therefore I replaced <code>tf.nn.relu(layer_1)</code> with a custom activation function <code>custom_sigmoid(layer_1)</code> which is defined as:</p>
  
  <pre><code>def custom_sigmoid(x):
      beta = tf.Variable(tf.random.normal(x.get_shape[1]))
      return tf.sigmoid(beta*x)
  </code></pre>
  
  <p>Where <code>beta</code> is a trainable parameter. I realized that this can not work since I don't know how to implement the derivative such that TensorFlow can use it.</p>
  
  <p>Question: How can I use a custom activation function in TensorFlow? I would really appreciate any help.</p>