title: Only adding non-duplicate values to PostgreSQL table
body: <p>I'm wondering what the best practice would be for avoiding duplication inside of a PostgreSQL database table. The rows in the table are equivalent to entries from a 3rd party API which is called several times throughout the day.</p>
  
  <p>I am only adding entries that are not duplicates from the JSON response. The JSON response can be upwards of 500 objects in the database, lets say:</p>
  
  <pre><code>[{
    "unique_id": 1234,
    "name": "A",
    "wins": 1
  },
  {
    "unique_id": 1235
    "name": "B",
    "wins": 1
  }, 
  ... 
  ];
  </code></pre>
  
  <p><strong>The issue is that the data from the API may be data I've already stored away in the database and when I query the 3rd party endpoint again, I don't want to have the data that has already been inserted added again - only the unique objects in the result.</strong></p>
  
  <p>And I'm aware how I could check if one individual object exists as a row, but when I have upwards of 500 in the 3rd party response and only 50 of them will be unique, what would be the reasonable check for this in one database transaction?</p>
  
  <p>I was tempted to go so far as to <code>SELECT * FROM table_name</code>, add those results to the json response, then remove duplicates that way, but that seems like an awful lot of overhead processing.</p>
  
  <p>Any help would be appreciated.</p>