title: Pyspark Dataframe: Transform many columns
body: <p>I have a pyspark dataframe with 10 columns as read from a parquet file</p>
  <pre><code>df = spark.read.parquet(path)
  </code></pre>
  <p>I want to apply several pre-processing steps to a subset of this dataframe's columns: <code>col_list</code>.</p>
  <p>The following works fine, but apart from a bit ugly, I also have the feeling it is not optimal.</p>
  <pre><code>import pyspark.sql.functions as F
  
  for col in col_list:
      df = df.withColumn(col, F.regexp_replace(col, &quot;.&quot;, &quot; &quot;)
      df = df.withColumn(col, F.regexp_replace(col, &quot;_[A-Z]_&quot;, &quot;&quot;)
  </code></pre>
  <p>and the list goes on with other similar text processing steps.</p>
  <p>So the question is whether the above is as optimal and elegant as it gets and also if/how I can use <code>transform</code> to achieve a sequential execution of the above steps.</p>
  <p>Thanks a lot.</p>