title: Spark Performance issue - Writing partitions to S3 as individual files
body: <p>I'm running a spark job whose job is to scan a large file and split it into smaller files. The file is in Json Lines format and I'm trying to partition it by a certain column (<em>id</em>) and save each partition as a separate file to S3. The file size is about 12 GB but there are about 500000 distinct values of <em>id</em>. The query is taking almost 15 hours. What can I do to improve performance? Is Spark a poor choice for such a task? Please note that I do have the liberty to making sure that the source as a fixed number of  rows per <em>id</em>.</p>
  <pre><code>import sys
  from pyspark.context import SparkContext
  from awsglue.context import GlueContext
  from pyspark.sql import functions as F
  from pyspark.sql.types import *
  from pyspark.sql.functions import *
  from pyspark.sql.window import Window
  from awsglue.utils import getResolvedOptions
  from awsglue.transforms import *
  from pyspark.sql.functions import udf, substring, instr, locate
  from datetime import datetime, timedelta
  
      
  sc = SparkContext.getOrCreate()
  glueContext = GlueContext(sc)
  spark = glueContext.spark_session
  # Get parameters that were passed to the job
  args = getResolvedOptions(sys.argv, ['INPUT_FOLDER', 'OUTPUT_FOLDER', 'ID_TYPE', 'DATASET_DATE'])
  
  id_type = args[&quot;ID_TYPE&quot;]
  output_folder = &quot;{}/{}/{}&quot;.format(args[&quot;OUTPUT_FOLDER&quot;], id_type, args[&quot;DATASET_DATE&quot;])
  input_folder = &quot;{}/{}/{}&quot;.format(args[&quot;INPUT_FOLDER&quot;], id_type, args[&quot;DATASET_DATE&quot;])
  
  
  INS_SCHEMA = StructType([
      StructField(&quot;camera_capture_timestamp&quot;, StringType(), True),
      StructField(id_type, StringType(), True),
      StructField(&quot;image_uri&quot;, StringType(), True)
  ])
  
  
  data = spark.read.format(&quot;json&quot;).load(input_folder, schema=INS_SCHEMA)
  
  data = data.withColumn(&quot;fnsku_1&quot;, F.col(&quot;fnsku&quot;))
  
  data.coalesce(1).write.partitionBy([&quot;fnsku_1&quot;]).mode('append').json(output_folder)   
  </code></pre>
  <p>I have tried repartition instead of coalesce too.</p>
  <p><em>I'm using AWS Glue</em></p>