{"action":"created","issue":{"url":"https://api.github.com/repos/cantino/huginn/issues/1740","repository_url":"https://api.github.com/repos/cantino/huginn","labels_url":"https://api.github.com/repos/cantino/huginn/issues/1740/labels{/name}","comments_url":"https://api.github.com/repos/cantino/huginn/issues/1740/comments","events_url":"https://api.github.com/repos/cantino/huginn/issues/1740/events","html_url":"https://github.com/cantino/huginn/issues/1740","id":183330605,"number":1740,"title":"Functionality Design of the Deduplication Agent","user":{"login":"level20peon","id":14050847,"avatar_url":"https://avatars.githubusercontent.com/u/14050847?v=3","gravatar_id":"","url":"https://api.github.com/users/level20peon","html_url":"https://github.com/level20peon","followers_url":"https://api.github.com/users/level20peon/followers","following_url":"https://api.github.com/users/level20peon/following{/other_user}","gists_url":"https://api.github.com/users/level20peon/gists{/gist_id}","starred_url":"https://api.github.com/users/level20peon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/level20peon/subscriptions","organizations_url":"https://api.github.com/users/level20peon/orgs","repos_url":"https://api.github.com/users/level20peon/repos","events_url":"https://api.github.com/users/level20peon/events{/privacy}","received_events_url":"https://api.github.com/users/level20peon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2016-10-17T05:49:09Z","updated_at":"2016-10-19T10:41:00Z","closed_at":null,"body":"I would like to know how the Deduplication Agent handles its task. Let's say I have three Agents:\r\n\r\nWebSiteAgent -> Deduplication Agent -> DataOutput Agent\r\n\r\nAnd let's say the WebSiteAgent has its Option \"Keep events\" set to \"1 day\". The WebSiteAgent will create 10 events per day. The Deduplication Agent has its Loopback option set to 1000.\r\n\r\nWill the Deduplication agent output the same (duplicate) events at day two (and so forth), because the original events from the WebSiteAgent don't exist any longer, or will it still work correctly (as in filter out the duplicates) because the hash of the new events at day two (and so forth) will match the hash of the original events?\r\n\r\nI am asking because I get lots of duplicates every day and I am wondering if the issue originates from the fact that I am not storing original events very long once they have been processed. I checked out GIT on July 12th for the last time, but I couldn't see any commits from then which involved the Deduplication Agent, so I guess I'm up to date concerning this specific functionality."},"comment":{"url":"https://api.github.com/repos/cantino/huginn/issues/comments/254777100","html_url":"https://github.com/cantino/huginn/issues/1740#issuecomment-254777100","issue_url":"https://api.github.com/repos/cantino/huginn/issues/1740","id":254777100,"user":{"login":"dsander","id":20943,"avatar_url":"https://avatars.githubusercontent.com/u/20943?v=3","gravatar_id":"","url":"https://api.github.com/users/dsander","html_url":"https://github.com/dsander","followers_url":"https://api.github.com/users/dsander/followers","following_url":"https://api.github.com/users/dsander/following{/other_user}","gists_url":"https://api.github.com/users/dsander/gists{/gist_id}","starred_url":"https://api.github.com/users/dsander/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dsander/subscriptions","organizations_url":"https://api.github.com/users/dsander/orgs","repos_url":"https://api.github.com/users/dsander/repos","events_url":"https://api.github.com/users/dsander/events{/privacy}","received_events_url":"https://api.github.com/users/dsander/received_events","type":"User","site_admin":false},"created_at":"2016-10-19T10:41:00Z","updated_at":"2016-10-19T10:41:00Z","body":"> Doesn't it write the hash of the event it saw into the db and the other worker reads that list of hashes before deciding on the new event being a duplicate or not?\r\n\r\nYes, normally it works like this. What I think might happen is a classic race condition. Both workers run the Agent at the same time, which means they get the memory from the database and both write it back with their own changes (not knowing about what the other worker did). The Agent that finished last thereby overrides the changes of the other one that finished milliseconds before.\r\n\r\nDisabling `propagate_immediately` has the effect that the Agent only runs once per minute and processes all events of the last minute. That means the race condition I  am describing can not happen anymore and thus you should not get any duplicates anymore."}}
{"actor":{"display_login":"dsander"}}