title: pyspark: ValueError: Some of types cannot be determined after inferring
body: <p>I have a pandas data frame <code>my_df</code>, and <code>my_df.dtypes</code> gives us:</p>
  
  <pre><code>ts              int64
  fieldA         object
  fieldB         object
  fieldC         object
  fieldD         object
  fieldE         object
  dtype: object
  </code></pre>
  
  <p>Then I am trying to convert the pandas data frame <code>my_df</code> to a spark data frame by doing below:</p>
  
  <pre><code>spark_my_df = sc.createDataFrame(my_df)
  </code></pre>
  
  <p>However, I got the following errors:</p>
  
  <pre><code>ValueErrorTraceback (most recent call last)
  &lt;ipython-input-29-d4c9bb41bb1e&gt; in &lt;module&gt;()
  ----&gt; 1 spark_my_df = sc.createDataFrame(my_df)
        2 spark_my_df.take(20)
  
  /usr/local/spark-latest/python/pyspark/sql/session.py in createDataFrame(self, data, schema, samplingRatio)
      520             rdd, schema = self._createFromRDD(data.map(prepare), schema, samplingRatio)
      521         else:
  --&gt; 522             rdd, schema = self._createFromLocal(map(prepare, data), schema)
      523         jrdd = self._jvm.SerDeUtil.toJavaArray(rdd._to_java_object_rdd())
      524         jdf = self._jsparkSession.applySchemaToPythonRDD(jrdd.rdd(), schema.json())
  
  /usr/local/spark-latest/python/pyspark/sql/session.py in _createFromLocal(self, data, schema)
      384 
      385         if schema is None or isinstance(schema, (list, tuple)):
  --&gt; 386             struct = self._inferSchemaFromList(data)
      387             if isinstance(schema, (list, tuple)):
      388                 for i, name in enumerate(schema):
  
  /usr/local/spark-latest/python/pyspark/sql/session.py in _inferSchemaFromList(self, data)
      318         schema = reduce(_merge_type, map(_infer_schema, data))
      319         if _has_nulltype(schema):
  --&gt; 320             raise ValueError("Some of types cannot be determined after inferring")
      321         return schema
      322 
  
  ValueError: Some of types cannot be determined after inferring
  </code></pre>
  
  <p>Does anyone know what the above error mean? Thanks!</p>