title: How to specify the correlation coefficient as the loss function in keras
body: <p>I am using keras+tensorflow for the first time. I would like to specify the <a href="https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.stats.pearsonr.html" rel="noreferrer">correlation coefficient</a> as the loss function. It makes sense to square it so that it is a number between 0 and 1 where 0 is bad and 1 is good.</p>
  
  <p>My basic code currently looks like:</p>
  
  <pre><code>def baseline_model():
          model = Sequential()
          model.add(Dense(4000, input_dim=n**2, kernel_initializer='normal', activation='relu'))
          model.add(Dense(1, kernel_initializer='normal'))
          # Compile model
          model.compile(loss='mean_squared_error', optimizer='adam')
          return model
  
  estimators = []
  estimators.append(('standardize', StandardScaler()))
  estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=32, verbose=2)))
  pipeline = Pipeline(estimators)
  kfold = KFold(n_splits=10, random_state=0)
  results = cross_val_score(pipeline, X, Y, cv=kfold)
  print("Standardized: %.2f (%.2f) MSE" % (results.mean(), results.std()))
  </code></pre>
  
  <p>How can I change this so that it optimizes to minimize the squared correlation coefficient  instead?</p>
  
  <hr>
  
  <p>I tried the following:</p>
  
  <pre><code>def correlation_coefficient(y_true, y_pred):
      pearson_r, _ = tf.contrib.metrics.streaming_pearson_correlation(y_pred, y_true)
      return 1-pearson_r**2
  
  def baseline_model():
  # create model
          model = Sequential()
          model.add(Dense(4000, input_dim=n**2, kernel_initializer='normal', activation='relu'))
  #        model.add(Dense(2000, kernel_initializer='normal', activation='relu'))
          model.add(Dense(1, kernel_initializer='normal'))
          # Compile model
          model.compile(loss=correlation_coefficient, optimizer='adam')
          return model
  </code></pre>
  
  <p>but this crashes with:</p>
  
  <pre><code>Traceback (most recent call last):
    File "deeplearning-det.py", line 67, in &lt;module&gt;
      results = cross_val_score(pipeline, X, Y, cv=kfold)
    File "/home/user/.local/lib/python3.5/site-packages/sklearn/model_selection/_validation.py", line 321, in cross_val_score
      pre_dispatch=pre_dispatch)
    File "/home/user/.local/lib/python3.5/site-packages/sklearn/model_selection/_validation.py", line 195, in cross_validate
      for train, test in cv.split(X, y, groups))
    File "/home/user/.local/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py", line 779, in __call__
      while self.dispatch_one_batch(iterator):
    File "/home/user/.local/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py", line 625, in dispatch_one_batch
      self._dispatch(tasks)
    File "/home/user/.local/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py", line 588, in _dispatch
      job = self._backend.apply_async(batch, callback=cb)
    File "/home/user/.local/lib/python3.5/site-packages/sklearn/externals/joblib/_parallel_backends.py", line 111, in apply_async
      result = ImmediateResult(func)
    File "/home/user/.local/lib/python3.5/site-packages/sklearn/externals/joblib/_parallel_backends.py", line 332, in __init__
      self.results = batch()
    File "/home/user/.local/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py", line 131, in __call__
      return [func(*args, **kwargs) for func, args, kwargs in self.items]
    File "/home/user/.local/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py", line 131, in &lt;listcomp&gt;
      return [func(*args, **kwargs) for func, args, kwargs in self.items]
    File "/home/user/.local/lib/python3.5/site-packages/sklearn/model_selection/_validation.py", line 437, in _fit_and_score
      estimator.fit(X_train, y_train, **fit_params)
    File "/home/user/.local/lib/python3.5/site-packages/sklearn/pipeline.py", line 259, in fit
      self._final_estimator.fit(Xt, y, **fit_params)
    File "/home/user/.local/lib/python3.5/site-packages/keras/wrappers/scikit_learn.py", line 147, in fit
      history = self.model.fit(x, y, **fit_args)
    File "/home/user/.local/lib/python3.5/site-packages/keras/models.py", line 867, in fit
      initial_epoch=initial_epoch)
    File "/home/user/.local/lib/python3.5/site-packages/keras/engine/training.py", line 1575, in fit
      self._make_train_function()
    File "/home/user/.local/lib/python3.5/site-packages/keras/engine/training.py", line 960, in _make_train_function
      loss=self.total_loss)
    File "/home/user/.local/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
      return func(*args, **kwargs)
    File "/home/user/.local/lib/python3.5/site-packages/keras/optimizers.py", line 432, in get_updates
      m_t = (self.beta_1 * m) + (1. - self.beta_1) * g
    File "/home/user/.local/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 856, in binary_op_wrapper
      y = ops.convert_to_tensor(y, dtype=x.dtype.base_dtype, name="y")
    File "/home/user/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 611, in convert_to_tensor
      as_ref=False)
    File "/home/user/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 676, in internal_convert_to_tensor
      ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
    File "/home/user/.local/lib/python3.5/site-packages/tensorflow/python/framework/constant_op.py", line 121, in _constant_tensor_conversion_function
      return constant(v, dtype=dtype, name=name)
    File "/home/user/.local/lib/python3.5/site-packages/tensorflow/python/framework/constant_op.py", line 102, in constant
      tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
    File "/home/user/.local/lib/python3.5/site-packages/tensorflow/python/framework/tensor_util.py", line 364, in make_tensor_proto
      raise ValueError("None values not supported.")
  ValueError: None values not supported.
  </code></pre>
  
  <hr>
  
  <p><strong>Update 1</strong></p>
  
  <p>Following the answer below the code now runs. Unfortunately, the <code>correlation_coefficient</code> and <code>correlation_coefficient_loss</code> functions give different values from each other and I am not sure either of them is the same as you would get from 1- <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html" rel="noreferrer">scipy.stats.pearsonr</a>()[0]**2.  </p>
  
  <blockquote>
    <p>Why are loss functions giving the wrong outputs and how can they be
    corrected to give the same values as <code>1 -
    scipy.stats.pearsonr()[0]**2</code> would give?</p>
  </blockquote>
  
  <p>Here is the completely self contained code that should just run:</p>
  
  <pre><code>import numpy as np
  import sys
  import math
  from scipy.stats import ortho_group
  from scipy.stats import pearsonr
  import matplotlib.pyplot as plt
  from keras.models import Sequential
  from keras.layers import Dense
  from keras.wrappers.scikit_learn import KerasRegressor
  from sklearn.model_selection import train_test_split
  from sklearn.preprocessing import StandardScaler
  from sklearn.pipeline import Pipeline
  import tensorflow as tf
  from keras import backend as K
  
  
  def permanent(M):
      n = M.shape[0]
      d = np.ones(n)
      j = 0
      s = 1
      f = np.arange(n)
      v = M.sum(axis=0)
      p = np.prod(v)
      while (j &lt; n-1):
          v -= 2*d[j]*M[j]
          d[j] = -d[j]
          s = -s
          prod = np.prod(v)
          p += s*prod
          f[0] = 0
          f[j] = f[j+1]
          f[j+1] = j+1
          j = f[0]
      return p/2**(n-1)
  
  
  def correlation_coefficient_loss(y_true, y_pred):
      x = y_true
      y = y_pred
      mx = K.mean(x)
      my = K.mean(y)
      xm, ym = x-mx, y-my
      r_num = K.sum(xm * ym)
      r_den = K.sum(K.sum(K.square(xm)) * K.sum(K.square(ym)))
      r = r_num / r_den
      return 1 - r**2
  
  
  def correlation_coefficient(y_true, y_pred):
      pearson_r, update_op = tf.contrib.metrics.streaming_pearson_correlation(y_pred, y_true)
      # find all variables created for this metric
      metric_vars = [i for i in tf.local_variables() if 'correlation_coefficient' in i.name.split('/')[1]]
  
      # Add metric variables to GLOBAL_VARIABLES collection.
      # They will be initialized for new session.
      for v in metric_vars:
          tf.add_to_collection(tf.GraphKeys.GLOBAL_VARIABLES, v)
  
      # force to update metric values
      with tf.control_dependencies([update_op]):
          pearson_r = tf.identity(pearson_r)
          return 1-pearson_r**2
  
  
  def baseline_model():
      # create model
      model = Sequential()
      model.add(Dense(4000, input_dim=no_rows**2, kernel_initializer='normal', activation='relu'))
  #    model.add(Dense(2000, kernel_initializer='normal', activation='relu'))
      model.add(Dense(1, kernel_initializer='normal'))
      # Compile model
      model.compile(loss=correlation_coefficient_loss, optimizer='adam', metrics=[correlation_coefficient])
      return model
  
  
  no_rows = 8
  
  print("Making the input data using seed 7", file=sys.stderr)
  np.random.seed(7)
  U = ortho_group.rvs(no_rows**2)
  U = U[:, :no_rows]
  # U is a random orthogonal matrix
  X = []
  Y = []
  print(U)
  for i in range(40000):
          I = np.random.choice(no_rows**2, size = no_rows)
          A = U[I][np.lexsort(np.rot90(U[I]))]
          X.append(A.ravel())
          Y.append(-math.log(permanent(A)**2, 2))
  
  X = np.array(X)
  Y = np.array(Y)
  
  estimators = []
  estimators.append(('standardize', StandardScaler()))
  estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=100, batch_size=32, verbose=2)))
  pipeline = Pipeline(estimators)
  X_train, X_test, y_train, y_test = train_test_split(X, Y,
                                                      train_size=0.75, test_size=0.25)
  pipeline.fit(X_train, y_train)
  </code></pre>
  
  <p><strong>Update 2</strong></p>
  
  <p>I have given up on the <code>correlation_coefficient</code> function and am now just using the <code>correlation_coefficient_loss</code> one as given by JulioDanielReyes below.  However, either this is still wrong or keras is dramatically overfitting.  Even when I have:</p>
  
  <pre><code>def baseline_model():
          model = Sequential()
          model.add(Dense(40, input_dim=no_rows**2, kernel_initializer='normal', activation='relu'))
          model.add(Dense(1, kernel_initializer='normal'))
          model.compile(loss=correlation_coefficient_loss, optimizer='adam', metrics=[correlation_coefficient_loss])
          return model
  </code></pre>
  
  <p>I get a loss of, for example, 0.6653 after 100 epochs but 0.857 when I test the trained model.</p>
  
  <blockquote>
    <p>How can it be overfitting which such a tiny number of nodes in the
    hidden layer?</p>
  </blockquote>