title: How to resize dataframe by dropping NaN in individual cells?
body: <p>I have a df that looks like this: </p>
  
  <pre><code>column1   column2    column3   column4
  1            2       nan          4
  1            2        3           nan
  1            2       nan          nan
  1            2       nan          nan
  </code></pre>
  
  <p>How do I reshape the dataframe, for every cell with NaN drop it, and if the column has the same value only take one instance of it? </p>
  
  <p>New df should look like this; </p>
  
  <pre><code>column1   column2    column3   column4
  1            2        3          4
  </code></pre>
  
  <p>I have roughly 500 columns with spotty data like this. </p>
  
  <p>Edit: </p>
  
  <p>I used this line of code to move the values with the spotty data into one row. </p>
  
  <pre><code>df = df.apply(lambda x: pd.Series(x.dropna().values))
  </code></pre>
  
  <p>new df looks like this: </p>
  
  <pre><code>column1   column2    column3   column4
  1            2        3            4
  1            2       nan          nan
  1            2       nan          nan
  1            2       nan          nan
  </code></pre>
  
  <p>Then I drop the duplicates:</p>
  
  <pre><code>df = df.drop_duplicates()
  </code></pre>
  
  <p>df looks like this now:</p>
  
  <pre><code> column1   column2    column3   column4
      1            2        3            4
      1            2       nan          nan
  </code></pre>
  
  <p>Not sure why the Nan are not dropping after this point with but the rows are dropping:</p>
  
  <pre><code>pivoted_df = pivoted_df.dropna()
  </code></pre>