title: Pyspark 'NoneType' object has no attribute '_jvm' error
body: <p>I was trying to print total elements in each partitions in a DataFrame using spark 2.2</p>
  
  <pre class="lang-python prettyprint-override"><code>from pyspark.sql.functions import *
  from pyspark.sql import SparkSession
  
  def count_elements(splitIndex, iterator):
      n = sum(1 for _ in iterator)
      yield (splitIndex, n)
  
  spark = SparkSession.builder.appName("tmp").getOrCreate()
  num_parts = 3
  df = spark.read.json("/tmp/tmp/gon_s.json").repartition(num_parts)
  print("df has partitions."+ str(df.rdd.getNumPartitions()))
  print("Elements across partitions is:" + str(df.rdd.mapPartitionsWithIndex(lambda ind, x: count_elements(ind, x)).take(3)))
  </code></pre>
  
  <p>The Code above kept failing with following error</p>
  
  <blockquote>
  <pre class="lang-python prettyprint-override"><code>  n = sum(1 for _ in iterator)
    File "/home/dev/wk/pyenv/py3/lib/python3.5/site-packages/pyspark/python/lib/pyspark.zip/pyspark/sql/functions.py", line 40, in _
      jc = getattr(sc._jvm.functions, name)(col._jc if isinstance(col, Column) else col)
  AttributeError: 'NoneType' object has no attribute '_jvm'
  </code></pre>
  </blockquote>
  
  <p>after removing the import below</p>
  
  <pre class="lang-python prettyprint-override"><code>from pyspark.sql.functions import *
  </code></pre>
  
  <p>Code works fine</p>
  
  <pre class="lang-python prettyprint-override"><code>skewed_large_df has partitions.3
  The distribution of elements across partitions is:[(0, 1), (1, 2), (2, 2)]
  </code></pre>
  
  <p>What is it causing this error and how can I fix it?</p>