title: How to use hyperopt for hyperparameter optimization of Keras deep learning network?
body: <p>I want to build a non linear regression model using keras to predict a +ve continuous variable.
  For the below model how do I select the following hyperparameters?</p>
  
  <ol>
  <li>Number of Hidden layers and Neurons</li>
  <li>Dropout ratio</li>
  <li>Use BatchNormalization or not</li>
  <li>Activation function out of linear, relu, tanh, sigmoid</li>
  <li>Best optimizer to use among adam, rmsprog, sgd </li>
  </ol>
  
  <p><strong><em>Code</em></strong></p>
  
  <pre><code>def dnn_reg():
      model = Sequential()
      #layer 1
      model.add(Dense(40, input_dim=13, kernel_initializer='normal'))
      model.add(Activation('tanh'))
      model.add(Dropout(0.2))
      #layer 2
      model.add(Dense(30, kernel_initializer='normal'))
      model.add(BatchNormalization())
      model.add(Activation('relu'))
      model.add(Dropout(0.4))
      #layer 3
      model.add(Dense(5, kernel_initializer='normal'))
      model.add(BatchNormalization())
      model.add(Activation('relu'))
      model.add(Dropout(0.4))
  
      model.add(Dense(1, kernel_initializer='normal'))
      model.add(Activation('relu'))
      # Compile model
      model.compile(loss='mean_squared_error', optimizer='adam')
      return model
  </code></pre>
  
  <p>I have considered random gridsearch but instead want to use hyperopt which I believe will be faster. I initially implemented the tuning using <a href="https://github.com/maxpumperla/hyperas" rel="noreferrer">https://github.com/maxpumperla/hyperas</a>. Hyperas is not working with latest version of keras. I suspect that keras is evolving fast and it's difficult for the maintainer to make it compatible. So I think using hyperopt directly will be a better option.</p>
  
  <p>PS: I am new to bayesian optimization for hyper parameter tuning and hyperopt.</p>