title: correct implementation of Hinge loss minimization for gradient descent
body: <p>I copied the hinge loss function from <a href="https://code.google.com/p/java-statistical-analysis-tool/source/browse/trunk/JSAT/src/jsat/lossfunctions/HingeLoss.java?r=762" rel="nofollow">here</a> (also LossC and LossFunc upon which it's based. Then I included it in my gradient descent algorithm like so: </p>
  
  <pre><code>  do 
    {
      iteration++;
      error = 0.0;
      cost = 0.0;
  
      //loop through all instances (complete one epoch)
      for (p = 0; p &lt; number_of_files__train; p++) 
      {
  
        // 1. Calculate the hypothesis h = X * theta
        hypothesis = calculateHypothesis( theta, feature_matrix__train, p, globo_dict_size );
  
        // 2. Calculate the loss = h - y and maybe the squared cost (loss^2)/2m
        //cost = hypothesis - outputs__train[p];
        cost = HingeLoss.loss(hypothesis, outputs__train[p]);
        System.out.println( "cost " + cost );
  
        // 3. Calculate the gradient = X' * loss / m
        gradient = calculateGradent( theta, feature_matrix__train, p, globo_dict_size, cost, number_of_files__train);
  
        // 4. Update the parameters theta = theta - alpha * gradient
        for (int i = 0; i &lt; globo_dict_size; i++) 
        {
            theta[i] = theta[i] - LEARNING_RATE * gradient[i];
        }
  
      }
  
      //summation of squared error (error value for all instances)
      error += (cost*cost);       
  
    /* Root Mean Squared Error */
    //System.out.println("Iteration " + iteration + " : RMSE = " + Math.sqrt( error/number_of_files__train ) );
    System.out.println("Iteration " + iteration + " : RMSE = " + Math.sqrt( error/number_of_files__train ) );
  
    } 
    while( error != 0 );
  </code></pre>
  
  <p>But this doesnt work at all. Is that due to the loss function? Maybe how I added the loss function to my code? </p>
  
  <p>I guess it's also possible that my implementation of gradient descent is faulty. </p>
  
  <p>Here are my methods for calculating the gradient and the hypothesis, are these right?</p>
  
  <pre><code>static double calculateHypothesis( double[] theta, double[][] feature_matrix, int file_index, int globo_dict_size )
  {
      double hypothesis = 0.0;
  
       for (int i = 0; i &lt; globo_dict_size; i++) 
       {
           hypothesis += ( theta[i] * feature_matrix[file_index][i] );
       }
       //bias
       hypothesis += theta[ globo_dict_size ];
  
       return hypothesis;
  }
  
  static double[] calculateGradent( double theta[], double[][] feature_matrix, int file_index, int globo_dict_size, double cost, int number_of_files__train)
  {
      double m = number_of_files__train;
  
      double[] gradient = new double[ globo_dict_size];//one for bias?
  
      for (int i = 0; i &lt; gradient.length; i++) 
      {
          gradient[i] = (1.0/m) * cost * feature_matrix[ file_index ][ i ] ;
      }
  
      return gradient;
  }
  </code></pre>
  
  <p>The rest of the code is <a href="https://github.com/h1395010/gradient_diss-1-ent_1-id" rel="nofollow">here</a> if you're interested to take a look. </p>
  
  <p>Below this sentence is what those loss functions look like. Should I use the <code>loss</code> or <code>deriv</code>, are these even correct?</p>
  
  <pre><code>/**
   * Computes the HingeLoss loss
   *
   * @param pred the predicted value
   * @param y the target value
   * @return the HingeLoss loss
   */
  public static double loss(double pred, double y)
  {
      return Math.max(0, 1 - y * pred);
  }
  
  /**
   * Computes the first derivative of the HingeLoss loss
   *
   * @param pred the predicted value
   * @param y the target value
   * @return the first derivative of the HingeLoss loss
   */
  public static double deriv(double pred, double y)
  {
      if (pred * y &gt; 1)
          return 0;
      else
          return -y;
  }
  </code></pre>