{"action":"created","issue":{"url":"https://api.github.com/repos/snabbco/snabb/issues/886","repository_url":"https://api.github.com/repos/snabbco/snabb","labels_url":"https://api.github.com/repos/snabbco/snabb/issues/886/labels{/name}","comments_url":"https://api.github.com/repos/snabbco/snabb/issues/886/comments","events_url":"https://api.github.com/repos/snabbco/snabb/issues/886/events","html_url":"https://github.com/snabbco/snabb/pull/886","id":149515059,"number":886,"title":"[wip] snabb top --yang: dump RFC7223 interface stats as JSON","user":{"login":"eugeneia","id":4933566,"avatar_url":"https://avatars.githubusercontent.com/u/4933566?v=3","gravatar_id":"","url":"https://api.github.com/users/eugeneia","html_url":"https://github.com/eugeneia","followers_url":"https://api.github.com/users/eugeneia/followers","following_url":"https://api.github.com/users/eugeneia/following{/other_user}","gists_url":"https://api.github.com/users/eugeneia/gists{/gist_id}","starred_url":"https://api.github.com/users/eugeneia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eugeneia/subscriptions","organizations_url":"https://api.github.com/users/eugeneia/orgs","repos_url":"https://api.github.com/users/eugeneia/repos","events_url":"https://api.github.com/users/eugeneia/events{/privacy}","received_events_url":"https://api.github.com/users/eugeneia/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":22,"created_at":"2016-04-19T16:37:31Z","updated_at":"2016-05-31T13:52:48Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/snabbco/snabb/pulls/886","html_url":"https://github.com/snabbco/snabb/pull/886","diff_url":"https://github.com/snabbco/snabb/pull/886.diff","patch_url":"https://github.com/snabbco/snabb/pull/886.patch"},"body":"This picks up on #766 to provide RFC7223 interface statistics as shared memory and updates `snabb top` to be able to access and format the information. In brief:\r\n\r\n* Extends `core.app` to support app-local counters under `counters/$app`\r\n* Adds RFC7223-based counters to Intel10G and VhostUser apps\r\n* Adds `snabb top --counters <app>` to list app counters\r\n* Adds `snabb top --yang` to dump [RFC7223](https://tools.ietf.org/html/rfc7223) model as JSON, including three interface types: hardware (NICs), virtual (VhostUser), link (links)\r\n\r\nOpen questions / to-do:\r\n\r\n* shared memory hierarchy is a bit of a mess, have to think how to structure it in case we want to support other types of app-local shm besides counters\r\n* which counters to provide and how: right now Intel10G does not expose unicast/multicast statistics as it does not provide the required statistics registers per queue (also only packets that actually go over the wire are counted, no NIC-switched packets); RFC7223 does not define a total packets count so rxpackets/txpackets counters are useless. Right now I mostly use the existing data to fill in applicable YANG fields: See VhostUser changes for a more pro-active example.\r\n* JSON does not support 64bit numbers so I convert these values to strings, RFC7223 defines drop counts as 32bit so I truncate those values to the lower 32 bits. Maybe JSON is a bad encoding for this?\r\n\r\nCc @lukego \r\n\r\n**Appendix:**\r\n\r\nYANG/JSON output looks like this:\r\n\r\n```\r\n{\r\n  \"interface-state\": [\r\n    {\r\n      \"type\": \"link\",\r\n      \"name\": \"B_NIC.tx -> B_Filter_in.rx\",\r\n      \"statistics\": {\r\n        \"in-octets\": \"0x00000000000005D2\",\r\n        \"out-octets\": \"0x00000000000005D2\",\r\n        \"discontinuity-time\": \"2016-04-19T15:56:31Z\",\r\n        \"out-discards\": 0\r\n      }\r\n    },\r\n    {\r\n      \"type\": \"link\",\r\n      \"name\": \"A_Virtio.tx -> A_NIC.rx\",\r\n      \"statistics\": {\r\n        \"in-octets\": \"0x00000000000005D2\",\r\n        \"out-octets\": \"0x00000000000005D2\",\r\n        \"discontinuity-time\": \"2016-04-19T15:56:31Z\",\r\n        \"out-discards\": 0\r\n      }\r\n    },\r\n    {\r\n      \"type\": \"link\",\r\n      \"name\": \"B_Virtio.tx -> B_Filter_out.rx\",\r\n      \"statistics\": {\r\n        \"in-octets\": \"0x00000000000002E9\",\r\n        \"out-octets\": \"0x00000000000002E9\",\r\n        \"discontinuity-time\": \"2016-04-19T15:56:49Z\",\r\n        \"out-discards\": 0\r\n      }\r\n    },\r\n    {\r\n      \"type\": \"link\",\r\n      \"name\": \"B_Filter_out.tx -> B_NIC.rx\",\r\n      \"statistics\": {\r\n        \"in-octets\": \"0x000000000000022D\",\r\n        \"out-octets\": \"0x000000000000022D\",\r\n        \"discontinuity-time\": \"2016-04-19T15:56:49Z\",\r\n        \"out-discards\": 0\r\n      }\r\n    },\r\n    {\r\n      \"type\": \"link\",\r\n      \"name\": \"A_NIC.tx -> A_Virtio.rx\",\r\n      \"statistics\": {\r\n        \"in-octets\": \"0x000000000000045A\",\r\n        \"out-octets\": \"0x000000000000045A\",\r\n        \"discontinuity-time\": \"2016-04-19T15:56:31Z\",\r\n        \"out-discards\": 0\r\n      }\r\n    },\r\n    {\r\n      \"type\": \"link\",\r\n      \"name\": \"B_Filter_in.tx -> B_Virtio.rx\",\r\n      \"statistics\": {\r\n        \"in-octets\": \"0x000000000000045A\",\r\n        \"out-octets\": \"0x000000000000045A\",\r\n        \"discontinuity-time\": \"2016-04-19T15:56:31Z\",\r\n        \"out-discards\": 0\r\n      }\r\n    },\r\n    {\r\n      \"type\": \"virtual\",\r\n      \"name\": \"B_Virtio\",\r\n      \"statistics\": {\r\n        \"in-octets\": \"0x0000000054D6F6D8\",\r\n        \"out-multicast\": \"0x0000000000000001\",\r\n        \"in-discards\": 0,\r\n        \"in-multicast\": \"0x0000000000000008\",\r\n        \"out-unicast\": \"0x0000000000EA4562\",\r\n        \"in-unicast\": \"0x00000000004A9D28\",\r\n        \"discontinuity-time\": \"2016-04-19T15:52:02Z\",\r\n        \"out-octets\": \"0x0000000C651B8648\"\r\n      }\r\n    },\r\n    {\r\n      \"type\": \"hardware\",\r\n      \"phys-address\": \"52:54:00:00:00:01\",\r\n      \"name\": \"B_NIC\",\r\n      \"statistics\": {\r\n        \"in-octets\": \"0x0000000000000000\",\r\n        \"out-octets\": \"0x0000000000000000\",\r\n        \"discontinuity-time\": \"2016-04-19T15:52:02Z\",\r\n        \"in-discards\": 0\r\n      }\r\n    },\r\n    {\r\n      \"type\": \"virtual\",\r\n      \"name\": \"A_Virtio\",\r\n      \"statistics\": {\r\n        \"in-octets\": \"0x0000000C81A73BB6\",\r\n        \"out-multicast\": \"0x0000000000000000\",\r\n        \"in-discards\": 0,\r\n        \"in-multicast\": \"0x0000000000000009\",\r\n        \"out-unicast\": \"0x000000000046808F\",\r\n        \"in-unicast\": \"0x0000000000EF0A89\",\r\n        \"discontinuity-time\": \"2016-04-19T15:52:02Z\",\r\n        \"out-octets\": \"0x000000003C3BDBF0\"\r\n      }\r\n    },\r\n    {\r\n      \"type\": \"hardware\",\r\n      \"phys-address\": \"52:54:00:00:00:00\",\r\n      \"name\": \"A_NIC\",\r\n      \"statistics\": {\r\n        \"in-octets\": \"0x0000000000000000\",\r\n        \"out-octets\": \"0x0000000000000000\",\r\n        \"discontinuity-time\": \"2016-04-19T15:52:02Z\",\r\n        \"in-discards\": 0\r\n      }\r\n    }\r\n  ]\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/snabbco/snabb/issues/comments/222694869","html_url":"https://github.com/snabbco/snabb/pull/886#issuecomment-222694869","issue_url":"https://api.github.com/repos/snabbco/snabb/issues/886","id":222694869,"user":{"login":"eugeneia","id":4933566,"avatar_url":"https://avatars.githubusercontent.com/u/4933566?v=3","gravatar_id":"","url":"https://api.github.com/users/eugeneia","html_url":"https://github.com/eugeneia","followers_url":"https://api.github.com/users/eugeneia/followers","following_url":"https://api.github.com/users/eugeneia/following{/other_user}","gists_url":"https://api.github.com/users/eugeneia/gists{/gist_id}","starred_url":"https://api.github.com/users/eugeneia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eugeneia/subscriptions","organizations_url":"https://api.github.com/users/eugeneia/orgs","repos_url":"https://api.github.com/users/eugeneia/repos","events_url":"https://api.github.com/users/eugeneia/events{/privacy}","received_events_url":"https://api.github.com/users/eugeneia/received_events","type":"User","site_admin":false},"created_at":"2016-05-31T13:52:48Z","updated_at":"2016-05-31T13:52:48Z","body":"@alexandergall My gripes is not with `lib.ipc.*` in general, but with its direct use in `intel10g.lua`. E.g. right now there are two statistics interfaces exposed by intel_app in this branch. I want to replace if the MIB interface with a more generic one.\r\n\r\n> I would prefer if we had a more generic framework for exposing data for the purpose of monitoring which is not biased towards any particular northbound interface. I'm sure that you can't assume any kind of mapping between YANG models and SNMP MIBs in general. That would make it necessary to have another layer that would transform the raw data to something specific for each protocol.\r\n\r\nThis is my point of view as well. I propose that we expose a super-set of RFC7223 counters with some simplifications. Different “readers” of our statistics interface will have to process the data to a specific format such as RFC7223 or ifMIB.\r\n\r\n* the only data type is uint64_t\r\n* the name of an interface is stored only in the directory structure (e.g. `/var/run/snabb/<pid>/counters/<name>/<stat-counter>`)\r\n* time/date is represented as seconds elapsed since epoch\r\n* I probably don't want to expose unicast packet counts but instead total packets / multicast / broadcast\r\n* I only want to expose fields that are actually implemented in a meaningful way. Readers must provide sensible defaults for fields that are not present."}}
{"actor":{"display_login":"eugeneia"}}