{"action":"created","issue":{"url":"https://api.github.com/repos/spohara79/estreamer/issues/4","repository_url":"https://api.github.com/repos/spohara79/estreamer","labels_url":"https://api.github.com/repos/spohara79/estreamer/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/spohara79/estreamer/issues/4/comments","events_url":"https://api.github.com/repos/spohara79/estreamer/issues/4/events","html_url":"https://github.com/spohara79/estreamer/issues/4","id":192950439,"number":4,"title":"Connection data does not get parsed","user":{"login":"TeePaps","id":3230497,"avatar_url":"https://avatars.githubusercontent.com/u/3230497?v=3","gravatar_id":"","url":"https://api.github.com/users/TeePaps","html_url":"https://github.com/TeePaps","followers_url":"https://api.github.com/users/TeePaps/followers","following_url":"https://api.github.com/users/TeePaps/following{/other_user}","gists_url":"https://api.github.com/users/TeePaps/gists{/gist_id}","starred_url":"https://api.github.com/users/TeePaps/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TeePaps/subscriptions","organizations_url":"https://api.github.com/users/TeePaps/orgs","repos_url":"https://api.github.com/users/TeePaps/repos","events_url":"https://api.github.com/users/TeePaps/events{/privacy}","received_events_url":"https://api.github.com/users/TeePaps/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-12-01T20:13:58Z","updated_at":"2016-12-02T06:27:12Z","closed_at":null,"body":"When I set the client up to get connection events, it doesn't seem to parse the data properly.\r\nThe `estreamer.config` file is configured like so:\r\n\r\n```\r\n[settings]\r\nevent_types=CONNECTION\r\n[flags]\r\nconnection=1\r\nextended_request=1\r\n```\r\n\r\nI also tried `connection_v5=1` because I wasn't sure which one to use\r\n\r\nWhen a **Connection** event comes in (type 70) I get a **MessageHeader** that look like this:\r\n```\r\n<MessageHeader(ver=1, type=4002, length=586, data=<MessageBundle(conn_id=1480621563, seq_num=1, data=[<EventData(type=71, length=554, data=bytearray(b\r\n'X/PW\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x05\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x03\\xeb\\x00\\x00\\x00\\x01\\\r\nx934/X\\xad\\x8d\\xf6\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x9b\\x00\\x00\\x01\\xf2\\x00\\x00\\x00\\x05@\\xae\\xcd\\xac\\x9\r\na\\xc8\\x11\\xe6\\xa6\\xa3\\xaf\\xeay\\x96i\"H\\xea\\x90F\\x9a\\xc8\\x11\\xe6\\xa6\\xa3\\xaf\\xeay\\x96i\"$\\xc4\\xec\\x00\\x8f\\xb2\\x11\\xe6\\xaf\\x05\\xf8\\x03\\xf9)\\xde\\xd9\\x19p`\\\r\nxbe\\x8f\\xb2\\x11\\xe6\\xaeK\\xe3\\xa7\\x1b\\t\\x9b$\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xff\\xff\\xac\\x14\\xd2\\xe6\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xf\r\nf\\xff\\xc0\\xa8\\xd3\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00X/47\\x10\\x00\\x08\\x06\\x00\\x02\\x00\\x00\\xe9\\x90\\x00\\xa1\\x00\\x00\\x11\\x00\\x00\\x00\\x00\\\r\nx00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x01~\\xdbX/PV\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x\r\n00\\x00\\x00\\x00\\x00\\x00V\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x98\\x96\\x7f\\x00\\x00\\x03E\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00w5\\x97E\\x00\\x00\\x00\\x00\\x00\\x00\\x0\r\n0\\x00\\x00\\x00\\x00\\x08\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x08\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x08\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\\r\nx00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0\r\n0\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x08\\x00\\x00\\x00\\x00\\x00\\\r\nx00\\x00\\x08\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x08\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0\r\n0\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\\r\nx00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x08\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x0\r\n0\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\xb5c\\xe6\\x0b\\xb9\\xb1<q\\x\r\nc1\\xc9W\\xb3\\xc6!\\xaf8'))>])>)>\r\n```\r\n\r\nAs opposed to say a **Packet** **MessageHeader**:\r\n```\r\n<MessageHeader(ver=1, type=4002, length=1494, data=<MessageBundle(conn_id=1480622076, seq_num=1, data=[<EventData(type=2, length=1462, timestamp=14794\r\n95767, reserved=0, data=<Packet(device_id=1, event_id=131940, event_second=1479495766, packet_second=1479495766, packet_microsecond=550334, link_type=\r\n1, length=1434, data=bytearray(b\"\\x00b\\xec\\x15\\xfe$\\x00$P\\x1c\\xf8\\x00\\x08\\x00E\\x00\\x05\\x8c;.@\\x00\\x7f\\x06j\\xae\\xac\\x10\\x1akJ\\xd9?;\\xf8\\'\\x00P:\\x07\\x1b\r\n>\\xcd\\xa9\\x1f\\xecP\\x10\\x01\\x02\\xd2\\x12\\x00\\x00GET /pixel;r=1853297836;a=p-49lspY_sxMAyQ;fpan=0;fpa=P0-1033023347-1479493051336;ns=0;ce=1;cm=;je=0;sr=1\r\n920x1080x24;enc=n;dst=1;et=1479495766516;tzo=300;ref=http%3A%2F%2Fabovethelaw.com%2F2016%2F11%2FXXXXn%2F%3Frf%3D1;url=http%3A%2F%2Fabovethelaw.com%2F2016%2F11%2FXXXX%2F%3Frf%3D1;ogl=site_name.Above%20the%20Law%2Clocale.en_US%2Cdescription.This%20is%20absolutely%20horrifying%252E%2Ctitle.The%20P\r\nink%20xxx%3A%20Reports%20Of%vvv%vvv%vvv%20In%vvv%vvv%vvv%vvv%20T%2Curl.http%3A%2F%vvv%252Ecom%2F2016%2F\r\n11%2Fvvvvvvvvv%2Ctype.article%2Cimage.http%3A%2F%vvvvvv%aaaaaa%2Fwp-content%2Fuploads%2F2016%2F11%2FTr\r\nump-Grimace%252Ejpg HTTP/1.1\\r\\nHost: pixel.quantserve.com\\r\\nConnection: keep-alive\\r\\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebK\r\nit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36\\r\\nAccept: image/webp,image/*,*/*;q=0.8\\r\\nReferer: http://abovethelaw.com/2016/11/websiteurlasdfasd/?rf=1\\r\\nAccept-Encoding: gzip, deflate, sdch\\r\\nAccept-Language: en-U\r\nS,en;q=0.8\\r\\nCookie: mc=57fbbc01-b3b24-ab148-05083; d=ELcB7QEBvhaB8Qgt-T0cLhvvg1qLIezUfIxMH\"))>)>])>)>\r\n```\r\n\r\nI saw that `eventdata.py` did not have a class for the record type 71, which is \"Connection Statatistics\"\r\nCisco defines this type as so: [Connection Statistics Data Block 5.4+](http://www.cisco.com/c/en/us/td/docs/security/firesight/540/api/estreamer/EventStreamerIntegrationGuide/RNA-Records.html#46893)\r\n\r\nSo I added the following to `eventdata.py`:\r\n\r\n```\r\nRCD_TYPE_Connection=71\r\nclass Connection(Struct):\r\n\r\n    _fields_ = [\r\n        ('block_type', 'uint32', 155),\r\n        ('block_length', 'uint32', 0),\r\n        ('device_id', 'uint32', 0),\r\n        ('ingres_zone', 'uint8[16]', 0),\r\n        ('egress_zone', 'uint8[16]', 0),\r\n        ('ingress_interface', 'uint8[16]', 0),\r\n        ('egress_interface', 'uint8[16]', 0),\r\n        ('initiator_ip', 'uint8[16]', 0),\r\n        ('responder_ip', 'uint8[16]', 0),\r\n        ('policy_revision', 'uint8[16]', 0),\r\n        ('rule_id', 'uint32', 0),\r\n        ('rule_action', 'uint16', 0),\r\n        ('rule_reason', 'uint16', 0),\r\n        ('initiator_port', 'uint16', 0),\r\n        ('responder_port', 'uint16', 0),\r\n        ('tcp_flags', 'uint16', 0),\r\n        ('protocol', 'uint8', 0),\r\n        ('netflow_source', 'uint8[16]', 0),\r\n        ('instance_id', 'uint16', 0),\r\n        ('conn_counter', 'uint16', 0),\r\n        ('first_pkt_time', 'uint32', 0),\r\n        ('last_pkt_time', 'uint32', 0),\r\n        ('initiator_tx_pkts', 'uint8[8]', 0),\r\n        ('resp_tx_pkts', 'uint8[8]', 0),\r\n        ('initiator_tx_bytes', 'uint8[8]', 0),\r\n        ('resp_tx_bytes', 'uint8[8]', 0),\r\n        ('user_id', 'uint32', 0),\r\n        ('app_prot_id', 'uint32', 0),\r\n        ('url_category', 'uint32', 0),\r\n        ('url_reputation', 'uint32', 0),\r\n        ('client_app_id', 'uint32', 0),\r\n        ('web_app_id', 'uint32', 0),\r\n        ('client_url', StringDataBlock, 0),\r\n        ('netbios_name', StringDataBlock, 0),\r\n        ('client_app_version', StringDataBlock, 0),\r\n        ('monitor_rule_1', 'uint32', 0),\r\n        ('monitor_rule_2', 'uint32', 0),\r\n        ('monitor_rule_3', 'uint32', 0),\r\n        ('monitor_rule_4', 'uint32', 0),\r\n        ('monitor_rule_5', 'uint32', 0),\r\n        ('monitor_rule_6', 'uint32', 0),\r\n        ('monitor_rule_7', 'uint32', 0),\r\n        ('monitor_rule_8', 'uint32', 0),\r\n        ('sec_int_src_dst', 'uint8', 0),\r\n        ('sec_int_layer', 'uint8', 0),\r\n        ('file_event_count', 'uint16', 0),\r\n        ('intrusion_event_count', 'uint16', 0),\r\n        ('initiator_country', 'uint16', 0),\r\n        ('resp_country', 'uint16', 0),\r\n        ('ioc_num', 'uint16', 0),\r\n        ('src_autonomous_sys', 'uint32', 0),\r\n        ('dst_autonomous_sys', 'uint32', 0),\r\n        ('snmp_in', 'uint16', 0),\r\n        ('snmp_out', 'uint16', 0),\r\n        ('src_tos', 'uint8', 0),\r\n        ('dst_tos', 'uint8', 0),\r\n        ('src_mask', 'uint8', 0),\r\n        ('dst_mask', 'uint8', 0),\r\n        ('sec_ctxt', 'uint8[16]', 0),\r\n        ('vlan_id', 'uint16', 0),\r\n        ('referenced_host', StringDataBlock, 0),\r\n        ('user_agent', StringDataBlock, 0),\r\n        ('http_referrer', StringDataBlock, 0),\r\n        ('ssl_cert_fingerprint', 'uint8[16]', 0),\r\n        ('ssl_pol_id', 'uint8[16]', 0),\r\n        ('ssl_rule_id', 'uint32', 0),\r\n        ('ssl_ciph_suite', 'uint16', 0),\r\n        ('ssl_version', 'uint8', 0),\r\n        ('ssl_srv_cert_stat', 'uint8', 0),\r\n        ('ssl_srv_cert_stat', 'uint16', 0),\r\n        ('ssl_actual_action', 'uint16', 0),\r\n        ('ssl_flow_status', 'uint16', 0),\r\n        ('ssl_flow_error', 'uint16', 0),\r\n        ('ssl_flow_msgs', 'uint16', 0),\r\n        ('ssl_flow_flags', 'uint32', 0),\r\n        ('ssl_srv_name', StringDataBlock, 0),\r\n        ('ssl_url_category', 'uint32', 0),\r\n        ('ssl_session_id', 'uint8[32]', 0),\r\n        ('ssl_session_id_len', 'uint8', 0),\r\n        ('ssl_ticket_id', 'uint8[20]', 0),\r\n        ('ssl_ticket_id_len', 'uint8', 0),\r\n        ('net_analysis_policy_revision', 'uint8[16]', 0),\r\n    ]\r\n```\r\nAnd I also added `RCD_TYPE_Connection=71` to config.py\r\n\r\nHowever, the data still shows as a byte array as shown above. Am I missing something?\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/spohara79/estreamer/issues/comments/264383491","html_url":"https://github.com/spohara79/estreamer/issues/4#issuecomment-264383491","issue_url":"https://api.github.com/repos/spohara79/estreamer/issues/4","id":264383491,"user":{"login":"spohara79","id":3719291,"avatar_url":"https://avatars.githubusercontent.com/u/3719291?v=3","gravatar_id":"","url":"https://api.github.com/users/spohara79","html_url":"https://github.com/spohara79","followers_url":"https://api.github.com/users/spohara79/followers","following_url":"https://api.github.com/users/spohara79/following{/other_user}","gists_url":"https://api.github.com/users/spohara79/gists{/gist_id}","starred_url":"https://api.github.com/users/spohara79/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spohara79/subscriptions","organizations_url":"https://api.github.com/users/spohara79/orgs","repos_url":"https://api.github.com/users/spohara79/repos","events_url":"https://api.github.com/users/spohara79/events{/privacy}","received_events_url":"https://api.github.com/users/spohara79/received_events","type":"User","site_admin":false},"created_at":"2016-12-02T06:27:12Z","updated_at":"2016-12-02T06:27:12Z","body":"It could be possible that the UnpackError isn't being raised appropriately, but it just throws it into the byte array anyway.  I found that a number of structures were incorrectly documented in the guide and I had to break down the structure myself.  I don't work directly with these products anymore, so direct testing will be hard for me to do to actually troubleshoot.\r\n\r\nLet me see what I can do and I'll let you know.\r\n"}}
{"actor":{"display_login":"spohara79"}}