title: Neural network with genetic algorithm question
body: <p>Since this question is going to quite long you can read cliff notes at the bottom, or read the full text here.
  I'm playing around with genetic algorithms and neural networks (i.e. using genetic algorithms to evolve neural networks) but I'm encountering a few problems, in particular I find the evolution extremely slow to converge and often not converging at all towards the desired results! Yet, since there are so many settings (population size, mutation rate, crossover rate, and so on...) I don't know if this is due to some bugs in my code or because I made a poor choice of those aforementioned parameteres. Or maybe I'm expecting results too early when it is normal for this kind of networks to be so slow.</p>
  
  <p>Now some examples:
  Training a net to make the sum of two numbers, providing a training set with 20 examples and a genetic algorithm with the following settings:</p>
  
  <pre><code>#define MUTATION_RATE       0.5
  #define MAX_PERTURBATION    1
  #define POP_SIZE            500
  #define CROSSOVER_RATE      0.7
  #define NUM_TOUR            35
  #define ELITISM             4
  </code></pre>
  
  <p>Where NUM_TOUR is the number of individuals selected to run a tournament and ELITISM is the number of copies I make of the best individual of the current generation to propagate it to the next one.
  With these settings and a network with just 2 input and 1 output neurons I'm able to train it to give the sum of two numbers pretty accurately after 2500 generations (which seems like a hell of a lot to me, but whatever), like:</p>
  
  <pre><code>2 + 2 = 4.01
  </code></pre>
  
  <p>Or</p>
  
  <pre><code>1 + 5 = 5.98
  </code></pre>
  
  <p>Using the same settings, and a network with 2 input, 1 output, and 1 hidden layer of 2 neurons, I try to make it learn how to multiply two numbers: after 2500 generations I don't get anywhere near a good result (like: <code>0 * 10 = 3.7</code>).</p>
  
  <p>I didn't try any other operation since I guess that if I can't come up with a network which learns how to multiply there is either something wrong with my code or with my settings. As I said I'm using tournament selection, the crossover is implemented by chosing each gene with equal probability from one of the two parents, and the mutation by adding or subtracting a value from 0 to <code>MAX_PERTURBATION</code> to the current value.</p>
  
  <p>Is there anything I'm doing glaringly wrong? Can you point me out to any tutorial discussing how to optimize the use of genetic algorithm with neural networks (what kind of crossover works best, what kind of population size is better, also mutation rate, and so on...) Or can give me any tips?</p>
  
  <p>I'm creating a neural network which gets trained with genetic algorithms but it's behaving very poorly, not even being able to learn how to multiply two numbers. Some examples are shown in the complete text.</p>