title: Remove duplicates in NiFi
body: <p>I have a flow QueryDatabaseTable->ConvertRecord->PutElasticseachHttpRecord
  What I am trying to do is fetching the full data from MySQL database and feeding it into Elasticsearch to perform analytics on it using Kibana. However, my data has duplicate columns like below:(Highlighted in black are the only repeating value)</p>
  
  <p>ID,Machine Name,value1,Value2,Date</p>
  
  <p>1, <strong>abc, 10, 34</strong>, 2018-09-27 10:40:10</p>
  
  <p>2, <strong>abc, 10, 34</strong>, 2018-09-27 10:41:14</p>
  
  <p>3,<strong>abc, 10, 34</strong>, 2018-09-27 10:42:19</p>
  
  <p>4, xyz, 12, 45, 2018-09-27 10:45:19</p>
  
  <p>So In my table ID is primary key and Timestamp fields keep updating. What I want to achieve is to fetch only one record for particular <strong>Machine Name</strong>. Example below show what output table I want:</p>
  
  <p><strong>ID,Machine Name,value1,Value2,Date</strong></p>
  
  <p>1, abc, 10, 34, 2018-09-27 10:40:10</p>
  
  <p>4, xyz, 12, 45, 2018-09-27 10:45:19</p>
  
  <p>How can I achieve this in NiFi? The objective is to drop/delete the duplicate columns. If it's possible please tell me which processor to use and what configs to set?</p>
  
  <p><strong>I am getting following error in QueryRecord processor:</strong>
  <a href="https://i.stack.imgur.com/oeG5C.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/oeG5C.jpg" alt="QueryRecord Error"></a></p>
  
  <p><a href="https://i.stack.imgur.com/tCV8x.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/tCV8x.jpg" alt="QueryRecord Error"></a></p>
  
  <p><a href="https://i.stack.imgur.com/wymY7.jpg" rel="noreferrer"><img src="https://i.stack.imgur.com/wymY7.jpg" alt="QueryRecord Error2"></a></p>
  
  <p>Any suggestion is much appreciated.
  Thank You</p>