title: How to save dataframe as shp/geojson in PySpark/Databricks?
body: <p>I have a DataFrame that has WKT in one of the columns. That column can be transformed to geojson if needed.</p>
  <p>Is there a way to save (output to storage) this data as a geojson or shapefile in Databricks/PySpark?</p>
  <p>Example of a DataFrame:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Id</th>
  <th>Color</th>
  <th>Wkt</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1</td>
  <td>Green</td>
  <td>POINT (3 7)</td>
  </tr>
  <tr>
  <td>2</td>
  <td>Yellow</td>
  <td>POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10))</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>The DataFrame can have ~100K rows and more.</p>
  <p>I've tried using <code>Geopandas</code> library, but it doesn't work:</p>
  <pre><code>import geopandas as gpd
  
  # df is as PySpark dataframe
  
  # Covert it to Pandas dataframe
  pd_df = df.toPandas()
  pd_df['geometry']=pd['point_wkt'].apply(wkt.loads)
  
  # Convert it to GeoPandas dataframe
  gdf  = gpd.GeoDataFrame(pd, geometry='geometry')
  
  # The following fails:
  gdf.to_file(os.path.join(MOUNT_POINT,output_folder,&quot;shapefile.shp&quot;))
  </code></pre>
  <p>The error is:</p>
  <pre><code>Failed to create file /mnt/traces/output_folder/shapefile.shp: No such file or directory
  </code></pre>
  <p>The error makes no sense as the folder <code>/mnt/traces/output_folder/</code> does exist, and I've successfully saved the PySpark dataframe as CSV to it.</p>
  <p><code>df.write.csv(os.path.join(MOUNT_POINT,output_folder), sep='\t')</code></p>
  <p>I'm able to save GeoPandas dataframe to shapefile with the above code when running locally, but not on Spark (Databricks).</p>