title: AWS Glue export to parquet issue using glueContext.write_dynamic_frame.from_options
body: <p>I have the following problem.</p>
  
  <p>The code below is auto-generated by AWS Glue.</p>
  
  <p>It's mission is to data from Athena (backed up by .csv @ S3) and transform data into Parquet. </p>
  
  <p>The code is working for the reference flight dataset and for some relatively big tables (~100 Gb).</p>
  
  <p>However, in most cases it returns the error, which does not tell me much.</p>
  
  <pre><code>import sys
  from awsglue.transforms import *
  from awsglue.utils import getResolvedOptions
  from pyspark.context import SparkConf, SparkContext
  from awsglue.context import GlueContext
  from awsglue.job import Job
  
  ## @params: [JOB_NAME]
  args = getResolvedOptions(sys.argv, ['JOB_NAME'])
  
  conf = (SparkConf()
      .set("spark.driver.maxResultSize", "8g"))
  
  sc = SparkContext(conf=conf)
  glueContext = GlueContext(sc)
  spark = glueContext.spark_session
  job = Job(glueContext)
  job.init(args['JOB_NAME'], args)
  
  datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "XXX", table_name = "csv_impressions", transformation_ctx = "datasource0")
  
  applymapping1 = ApplyMapping.apply(frame = datasource0, mappings = [("event time", "long", "event_time", "long"), ("user id", "string", "user_id", "string"), ("advertiser id", "long", "advertiser_id", "long"), ("campaign id", "long", "campaign_id", "long")], transformation_ctx = "applymapping1")
  
  resolvechoice2 = ResolveChoice.apply(frame = applymapping1, choice = "make_struct", transformation_ctx = "resolvechoice2")
  
  dropnullfields3 = DropNullFields.apply(frame = resolvechoice2, transformation_ctx = "dropnullfields3")
  
  datasink4 = glueContext.write_dynamic_frame.from_options(frame = dropnullfields3, connection_type = "s3", connection_options = {"path": "s3://xxxx"}, format = "parquet", transformation_ctx = "datasink4")
  job.commit()
  </code></pre>
  
  <p>The error message identified by AWS Glue is:</p>
  
  <blockquote>
    <p>An error occurred while calling o72.pyWriteDynamicFrame</p>
  </blockquote>
  
  <p>The log file also contains:</p>
  
  <blockquote>
    <p>Job aborted due to stage failure: ...
        Task failed while writing rows</p>
  </blockquote>
  
  <p>Any idea how to find out the reason for failure?</p>
  
  <p>Or what it could be?</p>