title: Attempting to compare different tables and cut out rows based on comparison in Pandas
body: <p>So I have two different tables right now. These tables contain a series of information including one column being a specific date.</p>
  <p>Example:</p>
  <pre><code>[Table 1]
  Unique Identifier (Primary Key) / Date / Piece of Information
  0001 / December 1, 2020 / Apples
  
  [Table 2]
  Unique Identifier (Primary Key) / Date / Piece of Information
  0001 / December 5, 2020 / Oranges
  </code></pre>
  <p>I am trying to compare the two tables if the second table has a date that is AFTER the first table (for the same unique identifier), I would like to write this to a new table. There are a lot of rows in this table, and I need to keep going through the rows. However I can't seem to get this to work. This is what I am doing:</p>
  <pre><code>import pandas as pd
  from pyspark.sql.functions import desc
  from pyspark.sql import functions as F
  
  def fluvoxamine_covid_hospital_progression(fluvoxamine_covids_that_were_also_in_hospital_at_any_time, fluvoxamine_for_hospitalization_analysis_only_outpatients_need_to_dbl_chk):
      df_fluvoxamine_covid_outpatients = pd.DataFrame(fluvoxamine_for_hospitalization_analysis_only_outpatients_need_to_dbl_chk)
      df_fluvoxamine_covid_outpatients.dropDuplicates(['visit_occurrence_id'])
      df_fluvoxamine_covid_outpatients.sort(desc('visit_start_date'))
  
      df_fluvoxamine_converted_hospital = pd.DataFrame(fluvoxamine_covids_that_were_also_in_hospital_at_any_time)
      df_fluvoxamine_converted_hospital.dropDuplicates(['visit_occurrence_id'])
      df_fluvoxamine_converted_hospital.sort(desc('visit_start_date'))
  
      i = 0
  
      if df_fluvoxamine_covid_outpatients.sort('visit_start_date') &lt; df_fluvoxamine_converted_hospital.sort('visit_start_date'):
          i = i + 1
  </code></pre>