title: Implementing hybrid Particle Swarm Optimization in CNNs? (And, general questions about the algorithms as a whole)
body: <p>So, I'm kind of new to working with Neural Networks (I use Keras w/ the TensorFlow backend). My background in math spans just deep enough to understand the concept behind Gradient Descent optimization. I'm not confident enough to work with numbers and symbolic math.</p>
  <p>I was recently reading about PSO (another optimization technique called Particle Swarm Optimization). I've been building a CNN to classify lung disease types. So far, I've understood the following:</p>
  <p>Gradient Decent:</p>
  <ul>
  <li>Minimizes the cost function (finds a minimum of the cost function)</li>
  <li>Starts at some randomly initialized position and looks for the steepest gradient</li>
  <li>Cost function must be differentiable (slopes = gradient)</li>
  <li>Usually settles down in one minimum which could be a local or global minimum</li>
  </ul>
  <p>I understand Gradient Descent well but am confused on why PSO is a simpler approach. Here is what I know about PSO:</p>
  <p>Particle Swarm Optimization:</p>
  <ul>
  <li>Minimizes cost function</li>
  <li>Multiple particles start at different locations on this cost function</li>
  <li>Particles look for minimums but each particle is affected by the swarm</li>
  <li>This means particles don't settle into a single local minimum and can move out of minimums based on swarm behavior</li>
  <li>Improves the chance of finding a global minimum</li>
  <li><strong>Cost function DOES NOT have to be differentiable?</strong></li>
  </ul>
  <ol>
  <li><p>Why does this make sense? If the particles (my understanding of a particle is an instance of a model with randomly initialized weights, etc, which means it has a different position on the cost function). This essentially makes more model instances to train vs. gradient descent which trains one. Correct my understanding of a particle if what I just said is utter nonsense...</p>
  </li>
  <li><p>Why does the cost function not have to be differentiable? The particles are looking for a minimum and therefore need to go in direction of the steepest gradient downward.</p>
  </li>
  <li><p>How can one implement PSO in a CNN? I was looking at a library called Pyswarms which left me further frustrated since Pyswarms doesn't seem to be usable as an optimizer for CNNs.</p>
  </li>
  </ol>
  <p>(P.S. I am visualizing a cost function as a 3 variable function).</p>