title: Is Apache Spark less accurate than Scikit Learn?
body: <p>I've recently been trying to get to know Apache Spark as a replacement for Scikit Learn, however it seems to me that even in simple cases, Scikit converges to an accurate model far faster than Spark does.
  For example I generated 1000 data points for a very simple linear function (z=x+y) with the following script:</p>
  
  <pre><code>from random import random
  
  def func(in_vals):
      '''result = x (+y+z+w....)'''
      result = 0
      for v in in_vals:
          result += v
      return result
  
  if __name__ == "__main__":
      entry_count = 1000
      dim_count = 2
      in_vals = [0]*dim_count
      with open("data_yequalsx.csv", "w") as out_file:
          for entry in range(entry_count):
              for i in range(dim_count):
                  in_vals[i] = random()
              out_val = func(in_vals)
              out_file.write(','.join([str(x) for x in in_vals]))
              out_file.write(",%s\n" % str(out_val))
  </code></pre>
  
  <p>I then ran the following Scikit script:</p>
  
  <pre><code>import sklearn
  from sklearn import linear_model
  
  import numpy as np
  
  data = []
  target = []
  with open("data_yequalsx.csv") as inFile:
      for row in inFile:
          vals = row.split(",")
          data.append([float(x) for x in vals[:-1]])
          target.append(float(vals[-1]))
  
  test_samples= len(data)/10
  
  train_data = [0]*(len(data) - test_samples)
  train_target = [0]*(len(data) - test_samples)
  test_data = [0]*(test_samples)
  test_target = [0]*(test_samples)
  train_index = 0
  test_index = 0
  for j in range(len(data)):
      if j &gt;= test_samples:
          train_data[train_index] = data[j]
          train_target[train_index] = target[j]
          train_index += 1
      else:
          test_data[test_index] = data[j]
          test_target[test_index] = target[j]
          test_index += 1
  
  model = linear_model.SGDRegressor(n_iter=100, learning_rate="invscaling", eta0=0.0001, power_t=0.5, penalty="l2", alpha=0.0001, loss="squared_loss")
  model.fit(train_data, train_target)
  print(model.coef_)
  print(model.intercept_)
  
  result = model.predict(test_data)
  mse = np.mean((result - test_target) ** 2)
  print("Mean Squared Error = %s" % str(mse))
  </code></pre>
  
  <p>And then this Spark script: (with spark-submit , no other arguments)</p>
  
  <pre><code>from pyspark.mllib.regression import LinearRegressionWithSGD, LabeledPoint
  from pyspark import SparkContext
  
  sc = SparkContext (appName="mllib_simple_accuracy")
  
  raw_data = sc.textFile ("data_yequalsx.csv", minPartitions=10) #MinPartitions doesnt guarantee that you get that many partitions, just that you wont have fewer than that many partitions
  data = raw_data.map(lambda line: [float(x) for x in line.split (",")]).map(lambda entry: LabeledPoint (entry[-1], entry[:-1])).zipWithIndex()
  test_samples= data.count()/10
  
  training_data = data.filter(lambda (entry, index): index &gt;= test_samples).map(lambda (lp,index): lp)
  test_data = data.filter(lambda (entry, index): index &lt; test_samples).map(lambda (lp,index): lp)
  
  model = LinearRegressionWithSGD.train(training_data, step=0.01, iterations=100, regType="l2", regParam=0.0001, intercept=True)
  print(model._coeff)
  print(model._intercept)
  
  mse = (test_data.map(lambda lp: (lp.label - model.predict(lp.features))**2 ).reduce(lambda x,y: x+y))/test_samples;
  print("Mean Squared Error: %s" % str(mse))
  
  sc.stop ()
  </code></pre>
  
  <p>Strangely though, the error given by spark is an order of magnitude larger than that given by Scikit (0.185 and 0.045 respectively) despite the two models having a nearly identical setup (as far as I can tell)
  I understand that this is using SGD with very few iterations and so the results may differ but I wouldn't have thought that it would be anywhere near such a large difference or such a large error, especially given the exceptionally simple data.</p>
  
  <hr>
  
  <p>Is there something I'm misunderstanding in Spark? Is it not correctly configured? Surely I should be getting a smaller error than that?</p>