title: Real-time data streaming using Wikipedia's RecentChanges API
body: <p>I'm lately trying to create a demo on real time streaming using NiFi -> Kafka -> Druid -> Superset. For the purposes of this demo I chose to use <a href="https://www.mediawiki.org/wiki/API:RecentChanges" rel="nofollow noreferrer">Wikipedia's RecentChanges API</a> in order to get asynchronous data of the most recent changes. <br><br>
  I use this <a href="https://en.wikipedia.org/w/api.php?action=query&amp;list=recentchanges&amp;rcprop=title%7Cids%7Csizes%7Cflags%7Cuser%7Cuserid%7Ctimestamp&amp;format=json" rel="nofollow noreferrer">URL</a> in order to get a response of changes. I'm calling the API constanlty in order to not miss any changes. This way I get a lot of duplicates that I do not want.<br><br> Is there anyway to parameterize this API to fix it for example getting all the changes from the previous second and doing that everysecond or something else to tackle this issue. I'm trying to make a configuration for this uing NiFi, if someone has to add something on that part then visit this discussion on <a href="https://community.cloudera.com/t5/Support-Questions/Real-time-data-streaming-using-Wikipedia-API-NiFi-Duplicates/td-p/281102" rel="nofollow noreferrer">Cloudera</a>.</p>