title: How to solve "can't create Multiple Sparkcontext error "?
body: <p>I have two files.<br />
  I have a file in which I am creating  sparkcontext.</p>
  <h2>create_spark.py</h2>
  <p>The code is like this</p>
  <pre><code>spark_conf = (SparkConf().setAppName(app_name)
                                .set(&quot;spark.serializer&quot;, &quot;org.apache.spark.serializer.KryoSerializer&quot;)
                                .set(&quot;spark.task.maxFailures&quot;, &quot;14&quot;)
                                .set(&quot;spark.port.maxRetries&quot;, &quot;50&quot;)
                                .set(&quot;spark.yarn.max.executor.failures&quot;, &quot;14&quot;))
  
  
  
  spark_context = SparkContext(conf=spark_conf)
  sqlContext=HiveContext(spark_context)
  </code></pre>
  <p>Then there is another file which contains all the codes . name it : <code>function_file.py</code></p>
  <p>it has to functions namely: This function just does some operation with data.</p>
  <pre><code>def adjust_name(line):
      if line is not None:
          if &quot;(&quot; in line:
              if &quot;\(&quot; in line:
                  tem1 = line.split(&quot;\(&quot;)
                  return tem1[0]
              else:
                  tem1 = line.split(&quot;(&quot;)
                  return tem1[0]
          else:
              return line
      else:
          return line
  </code></pre>
  <p>Now we are creating udf of the <code>adjust_name</code> function as .</p>
  <pre><code>adjust=udf(adjust_name,StringType())
  </code></pre>
  <p>and we are using this udf in process_sql functions as <br></p>
  <p><strong>and another function that does all the table loading and all . for exampl</strong>e <br></p>
  <pre><code>def process_sql(sqlContext,source_db,processing_db,table_name):
      .
      .
      .df3 = df3.withColumn('org_name',trim(adjust(df3['col_name'])))
      return table_name.
  </code></pre>
  <p>and now in create_spark.py file I am importing function_file as a module.
  and I am calling the process_sql function as</p>
  <pre><code>x= function_file.process_sql(sqlContext,source_db,processing_db,table_name)
  </code></pre>
  <p>all the arguments are defined in advance.
  But I am getting error like :</p>
  <blockquote>
  <p>ValueError: Cannot run multiple SparkContexts at once; existing SparkContext() created by udf at function_file.py</p>
  </blockquote>
  <h2>NOTE: I am restricted to use spark 1.6 only</h2>
  <h3>EDIT : I got a clue , the UDF is creating sparkcontext even before my create_spark.py file is creating .</h3>
  <p><code>:Connecting to Spark and creating context with dim_emp_atsc_test_4_sept spark_context = SparkContext(conf=spark_conf)</code>
  <code> ValueError: Cannot run multiple SparkContexts at once; existing SparkContext(app=process_handler.py, master=yarn-client) created by udf at ..</code></p>