title: How to properly rename columns of dynamic dataframe in AWS Glue?
body: <p>I load JSON data and use relationalize method on dynamic dataframe to flatten the otherwise nested JSON object and saving it into parquet format. The problem is that once saved into parquet format for faster Athena queries, the column names contain dots, which is against the Athena SQL query syntax and thus I am unable to make column specific queries.</p>
  
  <p>In order to tackle this problem I also rename the column names in the Glue job to exclude the dots and put underscores instead. My question is which approach of the two would be better and why? (Efficiency- memory? execution speed on nodes? etc.). </p>
  
  <p>Also given the horrible aws glue documentation I could not come up with a dynamic frame only solution. I have problems getting the column names in dynamic fashion, thus I am utilizing toDF().</p>
  
  <p>1) First approach is around getting the column names from df extracted from dynamic df</p>
  
  <pre><code>relationalize1 = Relationalize.apply(frame=datasource0, transformation_ctx="relationalize1").select("roottable")
      df_relationalize1 = relationalize1.toDF()
      for field in df_relationalize1.schema.fields:
          relationalize1 = RenameField.apply(frame = relationalize1, old_name = "`"+field.name+"`", new_name = field.name.replace(".","_"), transformation_ctx = "renamefield_" + field.name)
  </code></pre>
  
  <p>2) Second approach would be to extract the df from dynamic df and perform the rename field on the pyspark df (instead of dynamic df), to then convert back to dynamic df and save it in parquet format.</p>
  
  <p>Is there a better approach? Can a crawler rename columns? How fast is .fromDF() method? Is there a better documentation on functions and methods than the pdf developer guide?</p>