{"action":"created","issue":{"url":"https://api.github.com/repos/Exa-Networks/exabgp/issues/810","repository_url":"https://api.github.com/repos/Exa-Networks/exabgp","labels_url":"https://api.github.com/repos/Exa-Networks/exabgp/issues/810/labels{/name}","comments_url":"https://api.github.com/repos/Exa-Networks/exabgp/issues/810/comments","events_url":"https://api.github.com/repos/Exa-Networks/exabgp/issues/810/events","html_url":"https://github.com/Exa-Networks/exabgp/issues/810","id":318909603,"number":810,"title":"Exabgp to collect RouteReflector Changes!","user":{"login":"Jitoxxx","id":18415449,"avatar_url":"https://avatars3.githubusercontent.com/u/18415449?v=4","gravatar_id":"","url":"https://api.github.com/users/Jitoxxx","html_url":"https://github.com/Jitoxxx","followers_url":"https://api.github.com/users/Jitoxxx/followers","following_url":"https://api.github.com/users/Jitoxxx/following{/other_user}","gists_url":"https://api.github.com/users/Jitoxxx/gists{/gist_id}","starred_url":"https://api.github.com/users/Jitoxxx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Jitoxxx/subscriptions","organizations_url":"https://api.github.com/users/Jitoxxx/orgs","repos_url":"https://api.github.com/users/Jitoxxx/repos","events_url":"https://api.github.com/users/Jitoxxx/events{/privacy}","received_events_url":"https://api.github.com/users/Jitoxxx/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-04-30T14:27:04Z","updated_at":"2018-05-02T08:48:37Z","closed_at":null,"author_association":"NONE","body":"\r\n##### Summary\r\n\r\nHello,\r\nI'm looking for a way to collect Route Changes on my Route Reflectors and collecting this data too a JSON format and then sending it to my Logstash/Elasticsearch so I can display this on my webGUI.\r\nAny examples or anyone who could point me in the right direction?\r\nThank you \r\n\r\n\r\n# OS\r\n\r\nLinux\r\n\r\n# Installation\r\n```\r\n pip install exabgp\r\n```\r\n\r\n\r\n# Environment \r\n```\r\nexabgp --di is empty\r\n```\r\n\r\n# Configuration\r\n```\r\n## pprint-json-from-file.py\r\n\r\n#!/usr/bin/env python\r\nimport sys\r\nfrom pprint import pprint\r\nimport json\r\n\r\n\r\ninfile = open(sys.argv[1])\r\nfor line in infile.readlines():\r\n  jsonin = json.loads(line)\r\n  pprint(jsonin)\r\n```\r\n```\r\n## transfrom-exabgp-json-py\r\n#!/usr/bin/env python\r\nimport sys\r\nimport socket\r\nimport json\r\nfrom pprint import pprint\r\n\r\nsock = socket.socket()\r\nsock.connect( ('MYIP', 5150) )\r\n\r\ndef get_attrs(json_in):\r\n    attrs = {}\r\n    attrs['neighbor'] =     json_in['neighbor']['address']['peer']\r\n    attrs['peeras'] =       json_in['neighbor']['asn']['peer']\r\n\r\n    msgtype =               json_in['neighbor']['message'].keys()[0]\r\n    attrs['msgtype'] = msgtype\r\n\r\n    if 'announce' in        json_in['neighbor']['message'][msgtype].keys():\r\n      action = 'announce'\r\n    elif 'withdraw' in      json_in['neighbor']['message'][msgtype].keys():\r\n      action = 'withdraw'\r\n    else:\r\n      print \"funky action in this message, %s\" % json_in\r\n    attrs['action'] = action\r\n\r\n    afisafi  =              json_in['neighbor']['message'][msgtype][action].keys()[0]\r\n    attrs['afi'] = afisafi.split()[0]\r\n    attrs['safi'] = afisafi.split()[1]\r\n\r\n    if action == 'announce':\r\n      nexthop =             json_in['neighbor']['message'][msgtype][action][afisafi].keys()[0]\r\n      attrs['nexthop'] =      nexthop\r\n\r\n      # simple mapping, but substitute hyphens since elastic doesn't like em\r\n      simple_attributes = ['local-preference','origin','med','originator-id','cluster-list']\r\n      for simple_attribute in simple_attributes:\r\n        if json_in['neighbor']['message'][msgtype]['attribute'].has_key(simple_attribute):\r\n          attrs[simple_attribute.replace('-','')] = json_in['neighbor']['message'][msgtype]['attribute'][simple_attribute]\r\n\r\n      # as-path needs to be casted from list to string\r\n      if json_in['neighbor']['message'][msgtype]['attribute'].has_key('as-path'):\r\n        if type(json_in['neighbor']['message'][msgtype]['attribute']['as-path']) == type([]):\r\n          attrs['aspath'] = ' '.join(str(x) for x in json_in['neighbor']['message'][msgtype]['attribute']['as-path'])\r\n        else:\r\n          attrs['aspath'] =   json_in['neighbor']['message'][msgtype]['attribute']['as-path']\r\n\r\n      # ext comms are string/value pairs by exa, only keep strings\r\n      if json_in['neighbor']['message'][msgtype]['attribute'].has_key('extended-community'):\r\n        extcomms = []\r\n        for extcomm in json_in['neighbor']['message'][msgtype]['attribute']['extended-community']:\r\n          extcomms.append(extcomm['string'])\r\n        attrs['extcomms'] = extcomms\r\n\r\n      # regular comms are lists, rewrite to colon=separated\r\n      if json_in['neighbor']['message'][msgtype]['attribute'].has_key('community'):\r\n        comms = []\r\n        for comm in json_in['neighbor']['message'][msgtype]['attribute']['community']:\r\n          comms.append(\"%s:%s\" % (comm[0],comm[1]))\r\n        attrs['community'] = comms\r\n    return attrs\r\n\r\n\r\ndef get_evpn_routes(announce_in):\r\n    announce = {}\r\n    announce['rd'] =           announce_in['rd']\r\n    routetype =                announce_in['code']\r\n    routetypename =            announce_in['name']\r\n    announce['routetype'] =    routetype\r\n    announce['routetypename'] =    routetypename\r\n    if routetype == 1:\r\n      announce['esi'] = announce_in['esi']\r\n      announce['ethernettag'] = announce_in['ethernet-tag']\r\n      announce['label'] = announce_in['label']\r\n    elif routetype == 2:\r\n      announce['esi'] = announce_in['esi']\r\n      announce['mac'] = announce_in['mac']\r\n      announce['ethernettag'] = announce_in['ethernet-tag']\r\n      try:\r\n        announce['ip'] = announce_in['ip']\r\n      except KeyError:\r\n        pass\r\n      announce['label'] = announce_in['label']\r\n    elif routetype == 3:\r\n      announce['ethernettag'] = announce_in['ethernet-tag']\r\n      announce['ip'] = announce_in['ip']\r\n    elif routetype == 4:\r\n      announce['ip'] = announce_in['ip']\r\n      announce['esi'] = announce_in['esi']\r\n    elif routetype == 5:\r\n      pass\r\n    else: # not supported\r\n      pass\r\n    return announce\r\n\r\ndef get_mpls_vpn_routes(announce_in):\r\n    announce = {}\r\n    announce['rd'] =           announce_in['rd']\r\n    announce['label'] = announce_in['label']\r\n    announce['prefix'] = announce_in['nlri']\r\n\r\n    return announce\r\n\r\ndef get_unicast_routes(announce_in):\r\n    announce = {}\r\n    announce['prefix'] = announce_in['nlri']\r\n\r\n    return announce\r\n\r\ndef get_flow_routes(announce_in):\r\n    announce = {}\r\n    announce['rule'] = announce_in['string']\r\n    return announce\r\n\r\nlogfile = open('/tmp/transform.log','a')\r\ncounter = 0\r\nwhile True:\r\n    try:\r\n        line = sys.stdin.readline().strip()\r\n        if line == \"\":\r\n            counter += 1\r\n            if counter > 100:\r\n                break\r\n            continue\r\n        counter = 0\r\n        json_in = json.loads(line)\r\n        json_out_list = []\r\n\r\n        #logfile.write(line)\r\n        attrs = get_attrs(json_in)\r\n        attrs['orig'] = line # only when developing/troubleshooting\r\n        #print \"ATTRS:\",attrs\r\n        routes_to_parse = []\r\n        if attrs['action'] == 'announce':\r\n          routes_to_parse = json_in['neighbor']['message'][attrs['msgtype']][attrs['action']][attrs['afi'] + ' ' + attrs['safi']][attrs['nexthop']]\r\n        elif attrs['action'] == 'withdraw':\r\n          routes_to_parse = json_in['neighbor']['message'][attrs['msgtype']][attrs['action']][attrs['afi'] + ' ' + attrs['safi']]\r\n        else:\r\n          print \"ERROR: we're not supposed to get here, funky action in \", line\r\n        #print \"RTP:\", routes_to_parse\r\n        for route in routes_to_parse:\r\n          newroute = {}\r\n          newroute.update(attrs)\r\n          if attrs['safi'] == 'evpn':\r\n            newroute.update(get_evpn_routes(route))\r\n            json_newroute = json.dumps(newroute)\r\n            json_out_list.append(json_newroute)\r\n          elif attrs['safi'] == 'mpls-vpn':\r\n            newroute.update(get_mpls_vpn_routes(route))\r\n            json_newroute = json.dumps(newroute)\r\n            json_out_list.append(json_newroute)\r\n          elif attrs['safi'] == 'unicast':\r\n            newroute.update(get_unicast_routes(route))\r\n            json_newroute = json.dumps(newroute)\r\n            json_out_list.append(json_newroute)\r\n          elif attrs['safi'] == 'flow':\r\n            newroute.update(get_flow_routes(route))\r\n            json_newroute = json.dumps(newroute)\r\n            json_out_list.append(json_newroute)\r\n          else:\r\n            logfile.write(line)\r\n        #print \"OUT LIST:\",len(json_out_list), pprint(json_out_list)\r\n        msg = '\\n'.join(json_out_list)\r\n        #print \"MSG\", type(msg),len(msg),msg\r\n        sock.sendall(msg + '\\n')\r\n    except KeyboardInterrupt:\r\n        sock.close()\r\n#        logfile.close()\r\n    except IOError:\r\n        # most likely a signal during readline\r\n        sock.close()\r\n#        logfile.close()\r\n\r\nsock.close()\r\n#logfile.close()\r\n```\r\n```\r\n## exabgp.env\r\n\r\n[exabgp.api]\r\nencoder = text\r\nhighres = false\r\nrespawn = false\r\nsocket = ''\r\n\r\n[exabgp.bgp]\r\nopenwait = 60\r\n\r\n[exabgp.cache]\r\nattributes = true\r\nnexthops = true\r\n\r\n[exabgp.daemon]\r\ndaemonize = true\r\npid = ''\r\nuser = 'nobody'\r\n\r\n[exabgp.log]\r\nall = false\r\nconfiguration = true\r\ndaemon = true\r\ndestination = 'exa.log'\r\nenable = true\r\nlevel = INFO\r\nmessage = false\r\nnetwork = true\r\npackets = false\r\nparser = false\r\nprocesses = true\r\nreactor = true\r\nrib = false\r\nroutes = false\r\nshort = false\r\ntimers = false\r\n\r\n[exabgp.pdb]\r\nenable = false\r\n\r\n[exabgp.profile]\r\nenable = false\r\nfile = ''\r\n\r\n[exabgp.reactor]\r\nspeed = 1.0\r\n\r\n[exabgp.tcp]\r\nacl = false\r\nbind = ''\r\ndelay = 0\r\nonce = false\r\nport = 179\r\n```\r\n\r\n```\r\n## collector.conf\r\n\r\nprocess parsed-route-backend {\r\n        run /home/ubuntu/exabgp-4.0.5/routecollector/bin/transform-exabgp-json.py;\r\n\tencoder json;\r\n}\r\n\r\ntemplate {\r\n    neighbor VPN-RR {\r\n\t    router-id xx.xxx.xx.xxx;\r\n        local-address xx.xxx.xx.xxx;\r\n        local-as xxxx;\r\n        peer-as xxxx;\r\n        hold-time 180;\r\n\tfamily {\r\n\t\t\tipv4 mpls-vpn;\r\n\t\t\tipv6 mpls-vpn;\r\n\t\t}\r\n\tapi speaking {\r\n\t\tprocesses [ parsed-route-backend ];\r\n\t\treceive {\r\n\t\t\tparsed;\r\n\t\t\t#keepalive;\r\n\t\t\tupdate;\r\n\t\t}\r\n\t}\r\n   }\r\n}\r\n\r\nneighbor xxx.xxx.xxx.xxx {\r\n\tinherit VPN-RR;\r\n```\r\n\r\n# Program output\r\n```\r\n^C14:25:40 | 10545  | reactor       | ^C received\r\n14:25:40 | 10545  | reactor       | performing shutdown\r\nTraceback (most recent call last):\r\n  File \"/home/ubuntu/exabgp-4.0.5/routecollector/bin/transform-exabgp-json.py\", line 129, in <module>\r\n    attrs = get_attrs(json_in)\r\n  File \"/home/ubuntu/exabgp-4.0.5/routecollector/bin/transform-exabgp-json.py\", line 12, in get_attrs\r\n    attrs['neighbor'] =     json_in['neighbor']['address']['peer']\r\nKeyError: 'neighbor'\r\n14:25:40 | 10545  | process       | terminating process parsed-route-backend\r\n```\r\n## This is the only error i seem to get."},"comment":{"url":"https://api.github.com/repos/Exa-Networks/exabgp/issues/comments/385907854","html_url":"https://github.com/Exa-Networks/exabgp/issues/810#issuecomment-385907854","issue_url":"https://api.github.com/repos/Exa-Networks/exabgp/issues/810","id":385907854,"user":{"login":"Jitoxxx","id":18415449,"avatar_url":"https://avatars3.githubusercontent.com/u/18415449?v=4","gravatar_id":"","url":"https://api.github.com/users/Jitoxxx","html_url":"https://github.com/Jitoxxx","followers_url":"https://api.github.com/users/Jitoxxx/followers","following_url":"https://api.github.com/users/Jitoxxx/following{/other_user}","gists_url":"https://api.github.com/users/Jitoxxx/gists{/gist_id}","starred_url":"https://api.github.com/users/Jitoxxx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Jitoxxx/subscriptions","organizations_url":"https://api.github.com/users/Jitoxxx/orgs","repos_url":"https://api.github.com/users/Jitoxxx/repos","events_url":"https://api.github.com/users/Jitoxxx/events{/privacy}","received_events_url":"https://api.github.com/users/Jitoxxx/received_events","type":"User","site_admin":false},"created_at":"2018-05-02T08:48:37Z","updated_at":"2018-05-02T08:48:37Z","author_association":"NONE","body":"If I used --env exabgp.env\r\n\r\nI get help from all the options Env had and what it does ?? \r\nIt doesn't load the exabgp.env "}}
{"actor":{"display_login":"Jitoxxx"}}