title: I set my data types in pandas , but while convert to pyspark all data go to string
body: <p>I'm trying to define the data types of my dataframe, so they're good for a table soon...</p>
  <p>So when I convert to spark, it ignores the datasource and sets everything to string</p>
  <p>My code:</p>
  <pre><code>BDtable_FINAL = pd.DataFrame({'data': ['0001-01-01 00:00:00', '2020-02-02 00:00:00', '2021-01-01 00:00:00']})
  
  BDtable_FINAL = BDtable_FINAL[~BDtable_FINAL['data'].isin(['0001-01-01 00:00:00'])]
  datainsert = datetime.now().strftime('%Y%m%d%H%M')
  dateinsert= dateinsert[:8] + '0000'
  BDtable_FINAL.insert(loc=0,column='dateinsert',value=dateinsert)
  
  BDtable_FINAL.astype({
        'dateinsert'  : 'int64',
        'date': 'datetime64'
      })
  
  spark_df = pandas_to_spark(BDtable_FINAL)
  spark_df.printSchema()
  </code></pre>
  <p>I've tried several ways:</p>
  <p>I'm using databricks so I passed the dataframe to koalas and used the native to_spark() function</p>
  <p>I've already tried to set the data directly in spark and in the end it leaves everything as a string, always</p>
  <p>Has anyone had this problem?</p>
  <p>Essa é a minha função pandas_to_spark():</p>
  <pre><code># Auxiliar functions
  def equivalent_type(f):
      if f == 'datetime64[ns]': return TimestampType()
      elif f == 'int64': return LongType()
      elif f == 'int32': return IntegerType()
      elif f == 'float64': return FloatType()
      else: return StringType()
  
  def define_structure(string, format_type):
      try: typo = equivalent_type(format_type)
      except: typo = StringType()
      return StructField(string, typo)
  
  # Given pandas dataframe, it will return a spark's dataframe.
  def pandas_to_spark(pandas_df):
      columns = list(pandas_df.columns)
      types = list(pandas_df.dtypes)
      struct_list = []
      print(columns, types)
      for column, typo in zip(columns, types): 
        struct_list.append(define_structure(column, typo))
      p_schema = StructType(struct_list)
      return sqlContext.createDataFrame(pandas_df, p_schema)
      ```
  
  [![print][1]][1]
  
  
    [1]: https://i.stack.imgur.com/bMM7F.png
  </code></pre>