title: Timestamp Timezone Wrong/Missing in Spark/Databricks SQL Output
body: <p>When converting a timestamp between timezones in databricks/spark sql, the timezone itself seems lost in the end result, and I can't seem to either keep it or add it back.</p>
  <p>I have a bunch of UTC times and am using the <a href="https://docs.databricks.com/sql/language-manual/functions/from_utc_timestamp.html" rel="nofollow noreferrer" title="Documentation of the from_utc_timestamp() Databricks SQL function.">from_utc_timetamp()</a> to convert them to a different timezone based on another field. The result is calculated correctly, but if I output it with a timezone it shows as UTC. It seems the conversion is done correctly but the end result has no timezone stored with it (affirmed by <a href="https://stackoverflow.com/questions/62691202/does-spark-sql-timestamp-datatype-actually-store-timezone">this answer</a>), so it uses the server zone for the timezone in all cases.</p>
  <p><strong>Example</strong>: Using the following SQL:</p>
  <pre><code>createTimestampUTC,
  v.timezone,
  date_format(from_utc_timestamp(createTimestampUTC, v.timezone),&quot;yyyy-MM-dd'T'HH:mm:s Z&quot;) createTimestampLocal,
  </code></pre>
  <p>I get the following:</p>
  <p><a href="https://i.stack.imgur.com/pC59r.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pC59r.png" alt="Result of query" /></a></p>
  <p>You can see that the third column has done the conversions correctly for the timezones, but the output itself still shows as being in UTC timezone.</p>
  <p>Repeating this with a <a href="https://docs.databricks.com/sql/language-manual/sql-ref-datetime-pattern.html" rel="nofollow noreferrer">lowercase z</a> in the <a href="https://docs.databricks.com/sql/language-manual/functions/date_format.html" rel="nofollow noreferrer">date_format</a> function shows the same; namely, the conversions occur but the end result is still treated as UTC.</p>
  <pre><code>createTimestampUTC,
  v.timezone,
  date_format(from_utc_timestamp(createTimestampUTC, v.timezone),&quot;yyyy-MM-dd'T'HH:mm:s z&quot;) createTimestampLocal,
  </code></pre>
  <p><a href="https://i.stack.imgur.com/pRDIT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pRDIT.png" alt="Query result with lowercase z in date_format" /></a></p>
  <p>I can also use an <code>O</code> in the format output instead of a <code>Z</code> or <code>z</code>, but this just gives me GMT instead of UTC; same output basically.</p>
  <p>All the <a href="https://docs.databricks.com/sql/language-manual/sql-ref-syntax-aux-conf-mgmt-set-timezone.html" rel="nofollow noreferrer">databricks documentation</a> or <a href="https://stackoverflow.com/questions/68956623/how-apply-a-different-timezone-to-a-timestamp-in-pyspark">stackoverflow questions</a> I can find seem to treat printing timezones as a matter of setting the spark server time and outputting that way, or doing the conversion <a href="https://stackoverflow.com/questions/61719585/from-utc-timestamp-not-taking-daylight-saving-time-into-account">without keeping the resulting timezone</a>. I'm trying to convert to multiple <em>different</em> timezones though, and to keep the timezone in the output. I need to generate the end result in this format:</p>
  <p><a href="https://i.stack.imgur.com/5JdhT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5JdhT.png" alt="Desired timestamp output format" /></a></p>
  <p>Is there a way to do this? How do I either keep the timezone after the conversion or add it back in the format I need based on the timezone column I have? Given that the conversion works, and that I can output the end result with a +0000 on it, all the functionality to do this seems there, how do I put it together?</p>