title: Backprop neural network learns simple functions but doesn't learn sklearn iris dataset
body: <p>I've written a backpropagation neural network in Python using NumPy for the matrix calculations and batch updating. It learns binary functions like XOR well, but when I train it on the the <code>iris</code> dataset (in <code>sklearn.datasets</code>) with one-vs-all (my target function is <code>y = iris.target == 1</code>) it settles on putting all 1's or all -1's on the outputs. I've tried learning rates in [0.01, 5], hidden-layer sizes in [3, 20] nodes, and up to 50k epochs with no improvement.</p>
  
  <p>Below is the important code for the NN. <code>_sigmoid</code> is numpy's tanh function, and <code>_dsigmoid</code> is its derivative. I'd really appreciate any help!</p>
  
  <pre><code>def __init__(self, n_input, n_hidden, n_output):
      self.n_input = n_input + 1
      self.n_hidden = n_hidden
      self.n_output = n_output
      self.w1 = np.random.normal(scale=0.7, size=(self.n_input*self.n_hidden)).reshape(self.n_input, self.n_hidden)
      self.w2 = np.random.normal(scale=0.7, size=(self.n_hidden*self.n_output)).reshape(self.n_hidden, self.n_output)
      self.output_activation = np.zeros(n_output)
      self.hidden_activation = np.zeros(n_hidden)
      self.input_activation = np.zeros(n_input)
  
  def feed_forward(self):
      """
      Update output vector created by feed-forward propagation of input activations
      """
      self.hidden_activation = self._sigmoid(np.dot(self.input_activation, self.w1))
      self.output_activation = self._sigmoid(np.dot(self.hidden_activation, self.w2))
  
  def back_propagate(self, target, alpha):        
      output_error = target - self.output_activation
      output_delta = output_error * self._dsigmoid(self.output_activation)
  
      hidden_error = np.dot(output_delta, self.w2.T)
      hidden_delta = hidden_error * self._dsigmoid(self.hidden_activation)
  
      self.w2 += alpha * (np.dot(self.hidden_activation.T, output_delta))
      self.w1 += alpha * (np.dot(self.input_activation.T, hidden_delta))
  
  def train(self, data, target, alpha, epochs=50):
      m = data.shape[0]
  
      # add bias to input
      X = np.ones((m, self.n_input))
      X[:, 1:] = data
  
      # turn target into a column vector
      target = target[:, np.newaxis]
  
      for epoch in range(epochs):
          self.input_activation = X
          self.feed_forward()
          self.back_propagate(target, alpha)
  
  def predict(self, data):
      m = data.shape[0]
      self.input_activation = np.ones((m, self.n_input))
      self.input_activation[:, 1:] = data
      self.feed_forward()
      return self.output_activation
  </code></pre>