{"action":"created","issue":{"url":"https://api.github.com/repos/certtools/intelmq/issues/402","repository_url":"https://api.github.com/repos/certtools/intelmq","labels_url":"https://api.github.com/repos/certtools/intelmq/issues/402/labels{/name}","comments_url":"https://api.github.com/repos/certtools/intelmq/issues/402/comments","events_url":"https://api.github.com/repos/certtools/intelmq/issues/402/events","html_url":"https://github.com/certtools/intelmq/issues/402","id":117414569,"number":402,"title":"need to fix default processing behaviour: no stopping on failures of individual events","user":{"login":"aaronkaplan","id":750019,"avatar_url":"https://avatars.githubusercontent.com/u/750019?v=3","gravatar_id":"","url":"https://api.github.com/users/aaronkaplan","html_url":"https://github.com/aaronkaplan","followers_url":"https://api.github.com/users/aaronkaplan/followers","following_url":"https://api.github.com/users/aaronkaplan/following{/other_user}","gists_url":"https://api.github.com/users/aaronkaplan/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronkaplan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronkaplan/subscriptions","organizations_url":"https://api.github.com/users/aaronkaplan/orgs","repos_url":"https://api.github.com/users/aaronkaplan/repos","events_url":"https://api.github.com/users/aaronkaplan/events{/privacy}","received_events_url":"https://api.github.com/users/aaronkaplan/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/certtools/intelmq/labels/architecture","name":"architecture","color":"bfe5bf"},{"url":"https://api.github.com/repos/certtools/intelmq/labels/feature%20request","name":"feature request","color":"84b6eb"},{"url":"https://api.github.com/repos/certtools/intelmq/labels/needs:%20discussion","name":"needs: discussion","color":"fef2c0"}],"state":"closed","locked":false,"assignee":{"login":"sebix","id":199050,"avatar_url":"https://avatars.githubusercontent.com/u/199050?v=3","gravatar_id":"","url":"https://api.github.com/users/sebix","html_url":"https://github.com/sebix","followers_url":"https://api.github.com/users/sebix/followers","following_url":"https://api.github.com/users/sebix/following{/other_user}","gists_url":"https://api.github.com/users/sebix/gists{/gist_id}","starred_url":"https://api.github.com/users/sebix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sebix/subscriptions","organizations_url":"https://api.github.com/users/sebix/orgs","repos_url":"https://api.github.com/users/sebix/repos","events_url":"https://api.github.com/users/sebix/events{/privacy}","received_events_url":"https://api.github.com/users/sebix/received_events","type":"User","site_admin":false},"assignees":[{"login":"sebix","id":199050,"avatar_url":"https://avatars.githubusercontent.com/u/199050?v=3","gravatar_id":"","url":"https://api.github.com/users/sebix","html_url":"https://github.com/sebix","followers_url":"https://api.github.com/users/sebix/followers","following_url":"https://api.github.com/users/sebix/following{/other_user}","gists_url":"https://api.github.com/users/sebix/gists{/gist_id}","starred_url":"https://api.github.com/users/sebix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sebix/subscriptions","organizations_url":"https://api.github.com/users/sebix/orgs","repos_url":"https://api.github.com/users/sebix/repos","events_url":"https://api.github.com/users/sebix/events{/privacy}","received_events_url":"https://api.github.com/users/sebix/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/certtools/intelmq/milestones/2","html_url":"https://github.com/certtools/intelmq/milestones/Release%20v1.0","labels_url":"https://api.github.com/repos/certtools/intelmq/milestones/2/labels","id":719078,"number":2,"title":"Release v1.0","description":"First stable version.","creator":{"login":"SYNchroACK","id":4184292,"avatar_url":"https://avatars.githubusercontent.com/u/4184292?v=3","gravatar_id":"","url":"https://api.github.com/users/SYNchroACK","html_url":"https://github.com/SYNchroACK","followers_url":"https://api.github.com/users/SYNchroACK/followers","following_url":"https://api.github.com/users/SYNchroACK/following{/other_user}","gists_url":"https://api.github.com/users/SYNchroACK/gists{/gist_id}","starred_url":"https://api.github.com/users/SYNchroACK/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SYNchroACK/subscriptions","organizations_url":"https://api.github.com/users/SYNchroACK/orgs","repos_url":"https://api.github.com/users/SYNchroACK/repos","events_url":"https://api.github.com/users/SYNchroACK/events{/privacy}","received_events_url":"https://api.github.com/users/SYNchroACK/received_events","type":"User","site_admin":false},"open_issues":38,"closed_issues":285,"state":"open","created_at":"2014-07-12T18:19:52Z","updated_at":"2016-07-25T12:54:35Z","due_on":"2016-08-30T22:00:00Z","closed_at":null},"comments":2,"created_at":"2015-11-17T18:07:10Z","updated_at":"2016-07-25T12:54:35Z","closed_at":"2016-07-25T12:54:35Z","body":"(this is related to Tomas' PR)\r\n\r\nAs discussed today with @sebix we need to change the default behaviour of intelmq bots:\r\n\r\n  1. a parser but **MUST** accept only those events that he can parse. Everything else gets dumped (logged) (for potential analysis and later re-injection via the dumptool). After an event got sucessfully parsed, the later bots (experts , output) can rely on valid data. But they can not rely on all the necessary data being there.\r\n  2. an expert bot or output bot **MUST** assume that data it receives is incomplete (though it can assume that it's syntactically and semantically valid). If some key-value pairs in an event are missing, it **MUST** either assume sane default values and complain (log) and continue to pass on the event OR it **MUST** reject the event (dump it - as above) and continue with the next event. It must never stop processing.\r\n  3. administrators of the system MUST have an easy way to figure out how many events were dumped in each processing stage (could this be shown in the intelmq-manager?) and give the possibility to edit the data and reinject it at the right place (-> dump tool)\r\n  4. All of this needs to be documented in a tutorial-style walk through\r\n  5. Add test-cases to each bot for failing data\r\n"},"comment":{"url":"https://api.github.com/repos/certtools/intelmq/issues/comments/234944825","html_url":"https://github.com/certtools/intelmq/issues/402#issuecomment-234944825","issue_url":"https://api.github.com/repos/certtools/intelmq/issues/402","id":234944825,"user":{"login":"aaronkaplan","id":750019,"avatar_url":"https://avatars.githubusercontent.com/u/750019?v=3","gravatar_id":"","url":"https://api.github.com/users/aaronkaplan","html_url":"https://github.com/aaronkaplan","followers_url":"https://api.github.com/users/aaronkaplan/followers","following_url":"https://api.github.com/users/aaronkaplan/following{/other_user}","gists_url":"https://api.github.com/users/aaronkaplan/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronkaplan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronkaplan/subscriptions","organizations_url":"https://api.github.com/users/aaronkaplan/orgs","repos_url":"https://api.github.com/users/aaronkaplan/repos","events_url":"https://api.github.com/users/aaronkaplan/events{/privacy}","received_events_url":"https://api.github.com/users/aaronkaplan/received_events","type":"User","site_admin":false},"created_at":"2016-07-25T12:54:35Z","updated_at":"2016-07-25T12:54:35Z","body":"done."}}
{"actor":{"display_login":"aaronkaplan"}}