title: OneHotEncoding: working in one dataframe, not working in very, very similar dataframe (pyspark)
body: <p>I am applying a OneHotEncoding function to two very similar dataframes. The first dataframe is the following:
  <a href="https://i.stack.imgur.com/Z8kTI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Z8kTI.png" alt="enter image description here" /></a></p>
  <p>When I apply the one hot encoding, everything works fine:
  <a href="https://i.stack.imgur.com/5MLR5.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5MLR5.png" alt="enter image description here" /></a></p>
  <p>However, when I apply the exact same function to this different, but very similar dataframe:
  <a href="https://i.stack.imgur.com/3ZbWv.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/3ZbWv.png" alt="enter image description here" /></a></p>
  <p>The following error occurs:</p>
  <p><a href="https://i.stack.imgur.com/wtwbo.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/wtwbo.png" alt="enter image description here" /></a></p>
  <p>I don't understand why this happens, because dataframe 1 and 2 were both extracted from a previous dataframe (they work as a train and test df for a machine learning application). Both are pyspark.sql dataframes. Can anyone help me?</p>