title: keep_prob value in dropout and getting worst results with dropout
body: <p>According to this link, the value of keep_prob has to be between (0,1]:
  <a href="https://www.tensorflow.org/api_docs/python/tf/nn/dropout" rel="nofollow noreferrer">Tensorflow manual</a></p>
  
  <p>Otherwise I'll get value error:</p>
  
  <pre><code>ValueError: If keep_prob is not in (0, 1] or if x is not a floating point tensor.
  </code></pre>
  
  <p>I'm using the following code for a simple neural network with one hidden layer:</p>
  
  <pre><code>n_nodes_input = len(train_x.columns) # number of input features
  n_nodes_hl = 30     # number of units in hidden layer
  n_classes = len(np.unique(Y_train_numeric)) 
  lr = 0.25
  x = tf.placeholder('float', [None, len(train_x.columns)])
  y = tf.placeholder('float')
  dropout_keep_prob = tf.placeholder(tf.float32)
  
  def neural_network_model(data, dropout_keep_prob):
      # define weights and biases for all each layer
      hidden_layer = {'weights':tf.Variable(tf.truncated_normal([n_nodes_input, n_nodes_hl], stddev=0.3)),
                        'biases':tf.Variable(tf.constant(0.1, shape=[n_nodes_hl]))}
      output_layer = {'weights':tf.Variable(tf.truncated_normal([n_nodes_hl, n_classes], stddev=0.3)),
                      'biases':tf.Variable(tf.constant(0.1, shape=[n_classes]))}
      # feed forward and activations
      l1 = tf.add(tf.matmul(data, hidden_layer['weights']), hidden_layer['biases'])
      l1 = tf.nn.sigmoid(l1)
      l1 = tf.nn.dropout(l1, dropout_keep_prob)
      output = tf.matmul(l1, output_layer['weights']) + output_layer['biases']
  
      return output
  
  def main():
      prediction = neural_network_model(x, dropout_keep_prob)
      cost = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(labels=y,logits=prediction))
      optimizer = tf.train.AdamOptimizer(lr).minimize(cost)
  
      sess = tf.InteractiveSession()
  
      tf.global_variables_initializer().run()
      for epoch in range(1000):
          loss = 0
          _, c = sess.run([optimizer, cost], feed_dict = {x: train_x, y: train_y, dropout_keep_prob: 4.})
          loss += c
  
          if (epoch % 100 == 0 and epoch != 0):
              print('Epoch', epoch, 'completed out of', 1000, 'Training loss:', loss)
      correct = tf.equal(tf.argmax(prediction,1), tf.argmax(y,1))
      accuracy = tf.reduce_mean(tf.cast(correct, tf.float32), name='op_accuracy')
  
      writer = tf.summary.FileWriter('graph',sess.graph)
      writer.close()
  
      print('Train set Accuracy:', sess.run(accuracy, feed_dict = {x: train_x, y: train_y, dropout_keep_prob: 1.}))
      print('Test set Accuracy:', sess.run(accuracy, feed_dict = {x: test_x, y: test_y, dropout_keep_prob: 1.}))
      sess.close()
  
  
  if __name__ == '__main__':
       main()
  </code></pre>
  
  <p>If I use a number in range (0,1] for dropout_keep_prob in the sess.run, the accuracy drops drastically. If I use a number bigger than 1, like 4, the accuracy goes beyond 0.9. 
  Once I use shift+tab in front of tf.nn.dropout(), this is written as part of description:</p>
  
  <pre><code>With probability `keep_prob`, outputs the input element scaled up by
  `1 / keep_prob`, otherwise outputs `0`.  The scaling is so that the expected
  sum is unchanged.
  </code></pre>
  
  <p>which seems to me that keep_prob has to be greater than 1 otherwise nothing would be dropped!</p>
  
  <p>Bottom line, I'm confused. Which part of dropout have I implemented wrong that my results are getting worst and what is a good number for keep_drop?</p>
  
  <p>Thank you</p>