title: Hyperparameter Tuning (Keras) a Neural Network Regression
body: <p>We have developed an Artificial Neural Network in Python, and in that regard we would like tune the hyperparameters with GridSearchCV to find the best possible hyperparameters. The goal of our ANN is to predict temperature based on other relevant features, and so far this is the evaluation of the performance of the neural network:</p>
  <pre><code>Coefficient of Determination (R2)    Root Mean Square Error (RMSE)    Mean Squared Error (MSE)    Mean Absolute Percent Error (MAPE)    Mean Absolute Error (MAE)    Mean Bias Error (MBE)
  0.9808840288506496                   0.7527763482280911               0.5666722304516204          0.09142692180578049                   0.588041786518511           -0.07293321963266877
  </code></pre>
  <p>As of now, we have no clue on how to utilize GridSearchCV correctly, and we therefore seek help to move us towards a solution that would satisfy our goal. We have a function that might work, but are not able to apply it correctly to our code.</p>
  <p>This is the hyperparameter tuning function (GridSearchCV):</p>
  <pre><code>def hyperparameterTuning():
      # Listing all the parameters to try
      Parameter_Trials = {'batch_size': [10, 20, 30],
                      'epochs': [10, 20],
                      'Optimizer_trial': ['adam', 'rmsprop']
                      }
  
      # Creating the regression ANN model
      RegModel = KerasRegressor(make_regression_ann, verbose=0)
  
      # Creating the Grid search space
      grid_search = GridSearchCV(estimator=RegModel,
                             param_grid=Parameter_Trials,
                             scoring=None,
                             cv=5)
  
      # Running Grid Search for different paramenters
      grid_search.fit(X, y, verbose=1)
  
      print('### Printing Best parameters ###')
      grid_search.best_params_
  </code></pre>
  <hr />
  <p>Our main function:</p>
  <pre><code>if __name__ == '__main__':
  
      print('--------------')
  
      dataframe = pd.read_csv(&quot;/.../file.csv&quot;)
      
      # Splitting data into training and tesing data
      X_train, X_test, y_train, y_test, PredictorScalerFit, TargetVarScalerFit = splitData(dataframe=dataframe)
      
      # Making the Regression Artificial Neural Network (ANN)
      ann = ANN(X_train=X_train, y_train=y_train, X_test=X_test, y_test=y_test, PredictorScalerFit=PredictorScalerFit, TargetVarScalerFit=TargetVarScalerFit)
  
      # Evaluation of the performance of the Aritifical Neural Network (ANN)
      eval = evaluation(y_test_orig=ann['temp'], y_test_pred=ann['Predicted_temp'])
  </code></pre>
  <p>Our function to split data into training and testing data:</p>
  <pre><code>def splitData(dataframe):
  
      X = dataframe[Predictors].values
      y = dataframe[TargetVariable].values
  
      ### Sandardization of data ###
      PredictorScaler = StandardScaler()
      TargetVarScaler = StandardScaler()
  
      # Storing the fit object for later reference
      PredictorScalerFit = PredictorScaler.fit(X)
      TargetVarScalerFit = TargetVarScaler.fit(y)
  
      # Generating the standardized values of X and y
      X = PredictorScalerFit.transform(X)
      y = TargetVarScalerFit.transform(y)
  
      # Split the data into training and testing set
      X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
  
      return X_train, X_test, y_train, y_test, PredictorScalerFit, TargetVarScalerFit
  </code></pre>
  <p>Our function to fit the model and to utilize the Artificial Neural Network (ANN)</p>
  <pre><code>def ANN(X_train, y_train, X_test, y_test, TargetVarScalerFit, PredictorScalerFit):
  
      model = make_regression_ann()
  
      # Fitting the ANN to the Training set
      model.fit(X_train, y_train, batch_size=5, epochs=100, verbose=1)
  
      # Generating Predictions on testing data
      Predictions = model.predict(X_test)
  
      # Scaling the predicted temp data back to original price scale
      Predictions = TargetVarScalerFit.inverse_transform(Predictions)
  
      # Scaling the y_test temp data back to original temp scale
      y_test_orig = TargetVarScalerFit.inverse_transform(y_test)
  
      # Scaling the test data back to original scale
      Test_Data = PredictorScalerFit.inverse_transform(X_test)
  
      TestingData = pd.DataFrame(data=Test_Data, columns=Predictors)
      TestingData['temp'] = y_test_orig
      TestingData['Predicted_temp'] = Predictions
      TestingData.head()
  
      # Computing the absolute percent error
      APE = 100 * (abs(TestingData['temp'] - TestingData['Predicted_temp']) / TestingData['temp'])
      TestingData['APE'] = APE
  
      # ...
      TestingData = TestingData.round(2)
  
      TestingData.to_csv(&quot;TestingData.csv&quot;)
  
      return TestingData
  </code></pre>
  <p>Our function to make the model of the ANN</p>
  <pre><code>def make_regression_ann():
      # create ANN model
      model = Sequential()
  
      # Defining the Input layer and FIRST hidden layer, both are same!
      model.add(Dense(units=8, input_dim=7, kernel_initializer='normal', activation='sigmoid'))
  
      # Defining the Second layer of the model
      # after the first layer we don't have to specify input_dim as keras configure it automatically
      model.add(Dense(units=6, kernel_initializer='normal', activation='sigmoid'))
  
      # The output neuron is a single fully connected node
      # Since we will be predicting a single number
      model.add(Dense(1, kernel_initializer='normal'))
  
      # Compiling the model
      model.compile(loss='mean_squared_error', optimizer='adam')
  
      return model
  </code></pre>
  <p>Our function to evaluate the performance of the ANN</p>
  <pre><code>def evaluation(y_test_orig, y_test_pred):
  
      # Computing the Mean Absolute Percent Error
      MAPE = mean_absolute_percentage_error(y_test_orig, y_test_pred)
  
      # Computing R2 Score
      r2 = r2_score(y_test_orig, y_test_pred)
  
      # Computing Mean Square Error (MSE)
      MSE = mean_squared_error(y_test_orig, y_test_pred)
  
      # Computing Root Mean Square Error (RMSE)
      RMSE = mean_squared_error(y_test_orig, y_test_pred, squared=False)
  
      # Computing Mean Absolute Error (MAE)
      MAE = mean_absolute_error(y_test_orig, y_test_pred)
  
      # Computing Mean Bias Error (MBE)
      MBE = np.mean(y_test_pred - y_test_orig)  # here we calculate MBE
  
      print('--------------')
  
      print('The Coefficient of Determination (R2) of ANN model is:', r2)
      print(&quot;The Root Mean Squared Error (RMSE) of ANN model is:&quot;, RMSE)
      print(&quot;The Mean Squared Error (MSE) of ANN model is:&quot;, MSE)
      print('The Mean Absolute Percent Error (MAPE) of ANN model is:', MAPE)
      print(&quot;The Mean Absolute Error (MAE) of ANN model is:&quot;, MAE)
      print(&quot;The Mean Bias Error (MBE) of ANN model is:&quot;, MBE)
  
      print('--------------')
  
      eval_list = [r2, RMSE, MSE, MAPE, MAE, MBE]
  columns = ['Coefficient of Determination (R2)', 'Root Mean Square Error (RMSE)', 'Mean Squared Error (MSE)',
             'Mean Absolute Percent Error (MAPE)', 'Mean Absolute Error (MAE)', 'Mean Bias Error (MBE)']
  
      dataframe = pd.DataFrame([eval_list], columns=columns)
  
      return dataframe
  </code></pre>