title: pandas to_sql when table has constraints and some records violoate constraint
body: <p>I have a ~1M rows pandas DataFrame, <code>df</code>, that I'd like to store in a Postgresql (9.3.1) database.  It seems logical to use <code>df.to_sql</code> on large batches:</p>
  
  <pre><code>df.to_sql('names',con=engine,index=False,if_exists='append',chunksize=10000)
  </code></pre>
  
  <blockquote>
    <p>IntegrityError: (psycopg2.IntegrityError) insert or update on table "names_h" violates foreign key constraint "name_h_fk"<br>
    DETAIL:  Key (name)=(HXD13) is not present in table "valid_names".</p>
  </blockquote>
  
  <p>Raised because there is a constraint on the <code>names</code> table which I'm trying to <code>INSERT</code> -- and &lt;100 of the records in <code>df</code> violate this constraint.</p>
  
  <p>To solve my problem I grabbed all the <code>valid_name</code> records from <code>valid_names</code>:</p>
  
  <pre><code>df_valid = pd.read_sql("SELECT valid_name FROM valid_names",cnxn)
  </code></pre>
  
  <p>and filtered out the <strong><em>bad names</em></strong> from my original <code>df</code>. </p>
  
  <pre><code>final_df = df.loc[df['name'].isin(df_valid.valid_name.tolist())]
  </code></pre>
  
  <p>Now, <code>final_df</code> executes fine and populated my table.</p>
  
  <pre><code>final_df.to_sql('names',con=engine,index=False,if_exists='append',chunksize=10000)
  </code></pre>
  
  <p>So, is there a more efficient method to load records into a database when some (in my case a small fraction) of your records violate a table constraint?  I've thought about a vectorized try except, or df.apply and iterating one by one, but I'm not familiar with what <code>to_sql</code> does under the hood.  Thank You.</p>