{"action":"created","issue":{"url":"https://api.github.com/repos/certtools/intelmq/issues/524","repository_url":"https://api.github.com/repos/certtools/intelmq","labels_url":"https://api.github.com/repos/certtools/intelmq/issues/524/labels{/name}","comments_url":"https://api.github.com/repos/certtools/intelmq/issues/524/comments","events_url":"https://api.github.com/repos/certtools/intelmq/issues/524/events","html_url":"https://github.com/certtools/intelmq/issues/524","id":155435422,"number":524,"title":"Multiple issues with Shadowserver Drone Parser","user":{"login":"dmth","id":4152441,"avatar_url":"https://avatars.githubusercontent.com/u/4152441?v=3","gravatar_id":"","url":"https://api.github.com/users/dmth","html_url":"https://github.com/dmth","followers_url":"https://api.github.com/users/dmth/followers","following_url":"https://api.github.com/users/dmth/following{/other_user}","gists_url":"https://api.github.com/users/dmth/gists{/gist_id}","starred_url":"https://api.github.com/users/dmth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmth/subscriptions","organizations_url":"https://api.github.com/users/dmth/orgs","repos_url":"https://api.github.com/users/dmth/repos","events_url":"https://api.github.com/users/dmth/events{/privacy}","received_events_url":"https://api.github.com/users/dmth/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/certtools/intelmq/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/certtools/intelmq/labels/component:%20bots","name":"component: bots","color":"d4c5f9"}],"state":"closed","locked":false,"assignee":{"login":"sebix","id":199050,"avatar_url":"https://avatars.githubusercontent.com/u/199050?v=3","gravatar_id":"","url":"https://api.github.com/users/sebix","html_url":"https://github.com/sebix","followers_url":"https://api.github.com/users/sebix/followers","following_url":"https://api.github.com/users/sebix/following{/other_user}","gists_url":"https://api.github.com/users/sebix/gists{/gist_id}","starred_url":"https://api.github.com/users/sebix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sebix/subscriptions","organizations_url":"https://api.github.com/users/sebix/orgs","repos_url":"https://api.github.com/users/sebix/repos","events_url":"https://api.github.com/users/sebix/events{/privacy}","received_events_url":"https://api.github.com/users/sebix/received_events","type":"User","site_admin":false},"assignees":[{"login":"sebix","id":199050,"avatar_url":"https://avatars.githubusercontent.com/u/199050?v=3","gravatar_id":"","url":"https://api.github.com/users/sebix","html_url":"https://github.com/sebix","followers_url":"https://api.github.com/users/sebix/followers","following_url":"https://api.github.com/users/sebix/following{/other_user}","gists_url":"https://api.github.com/users/sebix/gists{/gist_id}","starred_url":"https://api.github.com/users/sebix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sebix/subscriptions","organizations_url":"https://api.github.com/users/sebix/orgs","repos_url":"https://api.github.com/users/sebix/repos","events_url":"https://api.github.com/users/sebix/events{/privacy}","received_events_url":"https://api.github.com/users/sebix/received_events","type":"User","site_admin":false},{"login":"dmth","id":4152441,"avatar_url":"https://avatars.githubusercontent.com/u/4152441?v=3","gravatar_id":"","url":"https://api.github.com/users/dmth","html_url":"https://github.com/dmth","followers_url":"https://api.github.com/users/dmth/followers","following_url":"https://api.github.com/users/dmth/following{/other_user}","gists_url":"https://api.github.com/users/dmth/gists{/gist_id}","starred_url":"https://api.github.com/users/dmth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmth/subscriptions","organizations_url":"https://api.github.com/users/dmth/orgs","repos_url":"https://api.github.com/users/dmth/repos","events_url":"https://api.github.com/users/dmth/events{/privacy}","received_events_url":"https://api.github.com/users/dmth/received_events","type":"User","site_admin":false}],"milestone":null,"comments":10,"created_at":"2016-05-18T07:38:57Z","updated_at":"2016-07-19T17:19:13Z","closed_at":"2016-07-19T16:55:59Z","body":"The Shadowserver Drone Parser does not work as expected.\r\nintelmq/bots/parsers/shadowserver/drone_parser.py\r\n\r\n- [x] [l47](https://github.com/certtools/intelmq/blob/master/intelmq/bots/parsers/shadowserver/drone_parser.py#L47) `dictreader=True` is not an allowed parameter anymore. I guess this affects all Shadowserver parsers. This is easy to fix\r\n\r\n- [x] [l72](https://github.com/certtools/intelmq/blob/master/intelmq/bots/parsers/shadowserver/drone_parser.py#L72) If `count = ''` this does not work. This is easy to fix\r\n\r\n- [x]  The data I received has different headers than the specification on [Shadowserver.org ](https://www.shadowserver.org/wiki/pmwiki.php/Services/Botnet-Drone-Hadoop)\r\nReceived Data:\r\n`\"timestamp\",\"ip\",\"port\",\"asn\",\"geo\",\"region\",\"city\",\"hostname\",\"type\",\"infection\",\"url\",\"agent\",\"cc_ip\",\"cc_port\",\"cc_asn\",\"cc_geo\",\"cc_dns\",\"count\",\"proxy\",\"application\",\"p0f_genre\",\"p0f_detail\",\"machine_name\",\"id\",\"naics\",\"sic\",\"cc_naics\",\"cc_sic\",\"sector\",\"cc_sector\",\"ssl_cipher\",\"family\",\"tag\",\"public_source\"`\r\n\r\n- [x] [l66](https://github.com/certtools/intelmq/blob/master/intelmq/bots/parsers/shadowserver/drone_parser.py#L66) The field 'cc' is called 'cc_ip' in my Dataset. Documentation differs from reality.\r\n\r\n- [x] [l62](https://github.com/certtools/intelmq/blob/master/intelmq/bots/parsers/shadowserver/drone_parser.py#L62) assumes that the value in `row['url']` is a valid URL according to the DHO. Unfortunately it is not. It might be required to guess / determine the url. This might be done by deriving it from the fields: `cc_dns`, `application` and `port`. But there might still be some magic involved as those do not always contain a sufficient amount of data to determine the full url.\r\n\r\n**EDIT / Solution** \r\nSee https://github.com/certtools/intelmq/issues/524#issuecomment-226443897\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/certtools/intelmq/issues/comments/233703652","html_url":"https://github.com/certtools/intelmq/issues/524#issuecomment-233703652","issue_url":"https://api.github.com/repos/certtools/intelmq/issues/524","id":233703652,"user":{"login":"aaronkaplan","id":750019,"avatar_url":"https://avatars.githubusercontent.com/u/750019?v=3","gravatar_id":"","url":"https://api.github.com/users/aaronkaplan","html_url":"https://github.com/aaronkaplan","followers_url":"https://api.github.com/users/aaronkaplan/followers","following_url":"https://api.github.com/users/aaronkaplan/following{/other_user}","gists_url":"https://api.github.com/users/aaronkaplan/gists{/gist_id}","starred_url":"https://api.github.com/users/aaronkaplan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aaronkaplan/subscriptions","organizations_url":"https://api.github.com/users/aaronkaplan/orgs","repos_url":"https://api.github.com/users/aaronkaplan/repos","events_url":"https://api.github.com/users/aaronkaplan/events{/privacy}","received_events_url":"https://api.github.com/users/aaronkaplan/received_events","type":"User","site_admin":false},"created_at":"2016-07-19T17:19:13Z","updated_at":"2016-07-19T17:19:13Z","body":"\n> On 19 Jul 2016, at 18:56, Sebastian <notifications@github.com> wrote:\n> \n> ui, I missed this one in cd8b4e4\n\nokay.\nDo you still want to merge it in?\nI'll fix the test cases for shadowserver now...\n\n\n"}}
{"actor":{"display_login":"aaronkaplan"}}