{"action":"opened","issue":{"url":"https://api.github.com/repos/cantino/huginn/issues/1740","repository_url":"https://api.github.com/repos/cantino/huginn","labels_url":"https://api.github.com/repos/cantino/huginn/issues/1740/labels{/name}","comments_url":"https://api.github.com/repos/cantino/huginn/issues/1740/comments","events_url":"https://api.github.com/repos/cantino/huginn/issues/1740/events","html_url":"https://github.com/cantino/huginn/issues/1740","id":183330605,"number":1740,"title":"Functionality Design of the Deduplication Agent","user":{"login":"level20peon","id":14050847,"avatar_url":"https://avatars.githubusercontent.com/u/14050847?v=3","gravatar_id":"","url":"https://api.github.com/users/level20peon","html_url":"https://github.com/level20peon","followers_url":"https://api.github.com/users/level20peon/followers","following_url":"https://api.github.com/users/level20peon/following{/other_user}","gists_url":"https://api.github.com/users/level20peon/gists{/gist_id}","starred_url":"https://api.github.com/users/level20peon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/level20peon/subscriptions","organizations_url":"https://api.github.com/users/level20peon/orgs","repos_url":"https://api.github.com/users/level20peon/repos","events_url":"https://api.github.com/users/level20peon/events{/privacy}","received_events_url":"https://api.github.com/users/level20peon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-10-17T05:49:09Z","updated_at":"2016-10-17T05:49:09Z","closed_at":null,"body":"I would like to know how the Deduplication Agent handles its task. Let's say I have three Agents:\r\n\r\nWebSiteAgent -> Deduplication Agent -> DataOutput Agent\r\n\r\nAnd let's say the WebSiteAgent has its Option \"Keep events\" set to \"1 day\". The WebSiteAgent will create 10 events per day. The Deduplication Agent has its Loopback option set to 1000.\r\n\r\nWill the Deduplication agent output the same (duplicate) events at day two (and so forth), because the original events from the WebSiteAgent don't exist any longer, or will it still work correctly (as in filter out the duplicates) because the hash of the new events at day two (and so forth) will match the hash of the original events?\r\n\r\nI am asking because I get lots of duplicates every day and I am wondering if the issue originates from the fact that I am not storing original events very long once they have been processed. I checked out GIT on July 12th for the last time, but I couldn't see any commits from then which involved the Deduplication Agent, so I guess I'm up to date concerning this specific functionality."}}
{"actor":{"display_login":"level20peon"}}