title: Can all Spark SQL Builtin Functions be used directly on a dataframe without executing spark.sql("sql statement on temporary view")?
body: <p>The below code works</p>
  <pre><code>from pyspark.sql.functions import expr
  df = spark.createDataFrame([(2020, 6, 26), (1000, 2, 29), (-44, 1, 1)],['Y', 'M', 'D'])
  df.select(&quot;*&quot;,expr(&quot;make_date(Y,M,D) as lk&quot;)).show()
  </code></pre>
  <p>The below code fails with a message &quot;cannot import name 'make_date' from 'pyspark.sql.functions'&quot;</p>
  <pre><code>from pyspark.sql.functions import expr, make_date
  df = spark.createDataFrame([(2020, 6, 26), (1000, 2, 29), (-44, 1, 1)],['Y', 'M', 'D'])
  df.select(make_date(df.Y,df.M,df.D).alias(&quot;datefield&quot;)).show()
  </code></pre>
  <p>And there are other functions like sum, mean etc that goes well with a dataframe groupBy clause .</p>
  <pre><code>df3.groupBy(df3.hair).agg(sum_(df3.age),max_(df3.age)).show()
  </code></pre>
  <p>I am confused here. Why are some functions not used directly with a dataframe column and only as a sql expression.</p>
  <p>The spark documentation shows all functions executed as a sql statement and not on a dataframe column directly</p>
  <p><a href="https://spark.apache.org/docs/latest/sql-ref-functions-builtin.html" rel="nofollow noreferrer">https://spark.apache.org/docs/latest/sql-ref-functions-builtin.html</a></p>