title: Should I leave the variable as transient?
body: <p>I have been experimenting with Apache Spark trying to solve some queries like top-k, skyline etc.</p>
  
  <p>I have made a wrapper which encloses <code>SparkConf</code> and <code>JavaSparkContext</code> named <code>SparkContext</code>. This class also implements serializable but since <code>SparkConf</code> and <code>JavaSparkContext</code> are not serializable then the class isn't either.</p>
  
  <p>I have a class solving the topK query named <code>TopK</code>, the class implements serializable but the class also has a <code>SparkContext</code> member variable which is not serializable (for the reason above). Therefore I am getting an exception whenever I try to execute a <code>TopK</code> method from within a <code>.reduce()</code> function in an RDD.</p>
  
  <p>The solution I have found is to make <code>SparkContext</code> transient. </p>
  
  <p>My question is: Should I keep the <code>SparkContext</code> variable as transient or am I doing a big mistake?</p>
  
  <p><strong><code>SparkContext</code> class:</strong></p>
  
  <pre><code>import java.io.Serializable;
  import org.apache.spark.SparkConf;
  import org.apache.spark.api.java.*;
  
  public class SparkContext implements Serializable {
  
      private final SparkConf sparConf; // this is not serializable
      private final JavaSparkContext sparkContext; // this is not either
  
      protected SparkContext(String appName, String master) {
          this.sparConf = new SparkConf();
          this.sparConf.setAppName(appName);
          this.sparConf.setMaster(master);
  
          this.sparkContext = new JavaSparkContext(sparConf);
      }
  
      protected JavaRDD&lt;String&gt; textFile(String path) {
          return sparkContext.textFile(path);
      }
  
  }
  </code></pre>
  
  <p><strong><code>TopK</code> class:</strong></p>
  
  <pre><code>public class TopK implements QueryCalculator, Serializable {
  
      private final transient SparkContext sparkContext;
      .
      .
      .
  }
  </code></pre>
  
  <p><strong>Example that throws <code>Task not serializable</code> exception.</strong> <code>getBiggestPointByXDimension</code> won't even get entered because in order for it to be executed in a reduce function the class enclosing it (<code>TopK</code>) must be serializable.</p>
  
  <pre><code>private Point findMedianPoint(JavaRDD&lt;Point&gt; points) {
      Point biggestPointByXDimension = points.reduce((a, b) -&gt; getBiggestPointByXDimension(a, b));
      .
      .
      .
  }
  
  private Point getBiggestPointByXDimension(Point first, Point second) {
          return first.getX() &gt; second.getX() ? first : second;
      }
  </code></pre>