title: Enum equivalent in Spark Dataframe/Parquet
body: <p>I have a table with hundreds of millions of rows, that I want to store in a dataframe in Spark and persist to disk as a parquet file. </p>
  
  <p>The size of my Parquet file(s) is now in excess of 2TB and I want to make sure I have optimized this.</p>
  
  <p>A large proportion of these columns are string values, that can be lengthy, but also often have very few values. For example I have a column with only two distinct values (a 20charcter and a 30 character string) and I have another column with a string that is on average 400characters long but only has about 400 distinct values across all entries. </p>
  
  <p>In a relational database I would usually normalize those values out into a different table with references, or at least define my table with some sort of enum type. </p>
  
  <p>I cannot see anything that matches that pattern in DF or parquet files. Is the columnar storage handling this efficiently? Or should I look into something to optimize this further? </p>