{"action":"closed","number":1065,"pull_request":{"url":"https://api.github.com/repos/snabbco/snabb/pulls/1065","id":92929733,"html_url":"https://github.com/snabbco/snabb/pull/1065","diff_url":"https://github.com/snabbco/snabb/pull/1065.diff","patch_url":"https://github.com/snabbco/snabb/pull/1065.patch","issue_url":"https://api.github.com/repos/snabbco/snabb/issues/1065","number":1065,"state":"closed","locked":false,"title":"updated README.troubleshooting.md","user":{"login":"chrgraf","id":22005482,"avatar_url":"https://avatars.githubusercontent.com/u/22005482?v=3","gravatar_id":"","url":"https://api.github.com/users/chrgraf","html_url":"https://github.com/chrgraf","followers_url":"https://api.github.com/users/chrgraf/followers","following_url":"https://api.github.com/users/chrgraf/following{/other_user}","gists_url":"https://api.github.com/users/chrgraf/gists{/gist_id}","starred_url":"https://api.github.com/users/chrgraf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrgraf/subscriptions","organizations_url":"https://api.github.com/users/chrgraf/orgs","repos_url":"https://api.github.com/users/chrgraf/repos","events_url":"https://api.github.com/users/chrgraf/events{/privacy}","received_events_url":"https://api.github.com/users/chrgraf/received_events","type":"User","site_admin":false},"body":"Dear all,\r\n\r\nI have been updating the README.troubleshooting.md (https://github.com/Igalia/snabb/blob/lwaftr/src/program/snabbvmx/doc/README.troubleshooting.md)\r\n\r\nPlease have a review on this updated doc below:\r\n\r\n\r\n# Testing & troubleshooting\r\n\r\n## Troubleshooting\r\n\r\n### Troubleshooting lwAFTR\r\n\r\nIf traffic managed by the lwAFTR component is not responding as expected: packets\r\nthat should get decapsulate don't match a softwire, some packets are dropped,\r\ntraffic don't get in, etc, it's convenient to inspect lwAFTR counters to\r\ndiagnose the problem.  The Snabb's `lwaftr query` tool can be used to obtain\r\nthe counter values of a lwAFTR instance.\r\n\r\nThe Snabb's lwAFTR manual includes a chapter covering troubleshooting and counters,\r\nwith charts about the most common lwAFTR paths (encapsulation, decapsulation,\r\nhairpinning, etc).  Please refer to that guide for troubleshooting the\r\nlwAFTR logic (chapter 8 - Counters).\r\n\r\nThe manual also includes a section covering troubleshooting related with running\r\na lwAFTR instance.  This information can be useful in case of running the lwAFTR\r\nas a standalone application and not via SnabbVMX.\r\n\r\n### Troubleshooting SnabbVMX\r\n\r\nThis section covers the most common problems when running a SnabbVMX instance.\r\n\r\n#### Cannot start the SnabbVMX instance\r\n\r\n`Description`: When running a snabbvmx instance, an error reports `failed to lock\r\n<NIC>`.\r\n\r\n```bash\r\n$ sudo ./snabb snabbvmx lwaftr --conf snabbvmx-lwaftr.cfg --id xe1 \\\r\n   --pci 0000:81:00.0 --mac 02:AA:AA:AA:AA:AA\r\ncore/main.lua:26: failed to lock /sys/bus/pci/devices/0000:81:00.0/resource0\r\nstack traceback:\r\n        core/main.lua:137: in function <core/main.lua:135>\r\n        [C]: in function 'error'\r\n        core/main.lua:26: in function 'assert'\r\n        lib/hardware/pci.lua:143: in function 'map_pci_memory_locked'\r\n```\r\n\r\n`Solution`: This error happens when trying to run a Snabb program, in this case\r\nSnabbVMX, on a NIC which is already in use.  Please check that there's no other\r\nSnabb instance running on the same NIC (in this example, `0000:81:00.0`).\r\n\r\n#### SnabbVMX running but not receiving traffic\r\n\r\n`Description`: a `snabbvmx lwaftr` instance is running on a NIC, but `snabbvmx\r\ntop` reports that the SnabbVMX instance is receiving no traffic.\r\n\r\n`Solution`: This is a common problem which might be originated by various causes.\r\n\r\n1. If no traffic is received at all, the most likely cause is that the selected\r\nlwAFTR binding table contains no valid softwires for incoming traffic.\r\n\r\n2. Another possible cause is that SnabbVMX is running with dynamic nexthop\r\nresolution, but dynamic nexthop resolution is not working.\r\n\r\n#### Dynamic nexthop resolution is not working\r\n\r\n`Description`: SnabbVMX is running with dynamic nexthop resolution, but no traffic\r\nis leaving the lwAFTR.\r\n\r\n`Solution`: Check that SnabbVMX is indeed running with dynamic nexthop resolution.\r\nSnabbVMX's configuration file should have a `cache_refresh_interval` attribute\r\nset to a value higher than 0.\r\n\r\n```bash\r\nipv6_interface = {\r\n   cache_refresh_interval = 1,\r\n},\r\nipv4_interface = {\r\n   ipv4_address = \"10.0.1.1\",\r\n   cache_refresh_interval = 1,\r\n},\r\n```\r\n\r\nIf that's correct, the likely cause is that refreshment packets are not arriving\r\nto the VM.  A refreshment packet is a packet that is sent periodically to the VM\r\nto trigger nexthop cache resolution.\r\n\r\nPackets might not be arriving to the VM because there's no VM actually running.\r\nWas SnabbVMX started with a sock address?\r\n\r\n```bash\r\nsudo ./snabb snabbvmx lwaftr --id xe0 --conf snabbvmx-lwaftr-xe0.cfg\r\n   --pci 02:00.0 --mac 02:aa:aa:aa:aa:aa --sock /tmp/vh1a.sock\r\n```\r\n\r\nIf that's correct, another cause is that the selected MAC address (02:aa:aa:\r\naa:aa:aa in the example above) doesn't match the lwAFTR's configuration\r\nattributes `aftr_mac_inet_side` and `aftr_mac_b4_side`.  Check the bottom of\r\nthe next section for more details.\r\n\r\n#### SnabbVMX doesn't respond to IPv4 or IPV6 pings\r\n\r\n`Description`: Trying to ping SnabbVMX on one of its IPv4 or IPv6 interfaces gets\r\nno response.\r\n\r\n`Solution`: Check that the target address is indeed the lwAFTR IPV4 and IPv6\r\naddress. Steps:\r\n\r\n1. Open the selected SnabbVMX configuration file and go to its lwAFTR\r\nconfiguration.\r\n2. Check that the values of attributes `aftr_ipv6_ip` and `aftr_ipv4_ip` are\r\nequals to the target address.\r\n\r\nIn case the address is correct, check that the SnabbVMX MAC address is the same\r\nas `aftr_mac_inet_side` and `aftr_mac_b4_side`.  In SnabbVMX these values must\r\nbe the same, as there's a single NIC for both interfaces.\r\n\r\nSnabbVMX should have been initiated with this MAC address value.  Search a\r\nSnabbVMX process to see its command line parameters:\r\n\r\n```bash\r\n$ ps aux | grep snabbvmx\r\nroot     16115  0.0  0.0 133856  2936 pts/2    S+   22:39   0:00\r\n   sudo ./snabb snabbvmx lwaftr --conf snabbvmx-lwaftr.cfg --id xe1\r\n   --pci 0000:81:00.0 --mac 02:AA:AA:AA:AA:AA\r\n```\r\n\r\n#### SnabbVMX decapsulation path works but not encapsulation\r\n\r\n`Description`: `snabbvmx top` reports that the running SnabbVMX instance is able\r\nto decapsulate packets, but not to encapsulate them.\r\n\r\n```\r\nlwaftr (rx/tx/txdrop in Mpps)            rx      tx    rxGb    txGb    txdrop\r\nlwaftr_v6                              1.53       0    4.80       0    0.000022\r\nlwaftr_v4                                 0    1.53       0    4.31    0\r\n```\r\n\r\n`Solution`: If the decapsulation path works, then the lwAFTR is able to\r\ndecapsulate IPv6 packets coming from the B4: therefore the IPv4 source address\r\nand source port of the IPv6 encapsulated packet match a softwire in the\r\nbinding table. However, when a packet arrives to the lwAFTR from the Internet\r\nthe packet is dropped, thus RX is 0 in lwaftr_v4.\r\n\r\nThe destination address and port of an incoming IPv4 packet should match a\r\nsoftwire in the binding table.  What's the destination address and port of\r\nincoming packets? Is the lwAFTR using a VLAN tag for its IPv4 interface? Most\r\nlikely incoming packets are not VLAN tagged.  Check if SnabbVMX's config file\r\nhas a VLAN tag set.  Also check if the referred lwAFTR config file has a VLAN\r\ntag set.\r\n\r\nWhat's the MTU size of the IPv4 interface?  Check that value in SnabbVMX and\r\nlwAFTR configuration file.  If the MTU size is too small, packets will be\r\nfragmented.  A extremely small MTU size and disabled fragmentation will cause\r\nmost of the packets to be dropped.\r\n\r\n#### Packets don't get mirrored\r\n\r\n`Description`: You're using a tap interface for monitoring lwAFTR packets but\r\nnothing gets in.\r\n\r\n`Solution`:\r\n\r\n1. Check that you're running SnabbVMX with mirroring enabled.\r\n2. Check that you're running `lwaftr monitor` pointing to the IPv4 address you\r\nwould like to monitor.  For testing purposes, set `lwaftr monitor` to `all`.\r\n3. Check that your tap interface is up.\r\n\r\n---\r\n\r\n## Tests overview\r\n\r\nTests are useful to detect bugs in the `lwaftr` or `snabbvmx` and double-check that\r\neverything is working as expected.\r\n\r\n**preparation**\r\n- have tcpdump, snabb or vmxlwaftr installed.\r\n- should work on xeon or i7; will fail on Celeron chips \r\n- hugepages : Ensure hugepages is set in sysctl\r\n\"vm.nr_hugepages = 5\" in sysctl.conf\r\nIf using vmxlwaftr, then this is done by the scripts automatically\r\n\r\n**Test types**\r\nSnabbVMX features three types of tests:\r\n\r\n* Lua selftests: Unit tests for several modules\r\n(**apps/lwaftr/V4V6.lua** and **apps/lwaftr/nh_fwd.lua**).\r\n* Bash selftests: Complex setups to test certain functionality (**program/\r\nsnabbvmx/tests/selftest.sh** and **program/snabbvmx/tests/nexthop/\r\nselftest.sh**).\r\n* End-to-end tests: Specific end-to-end test cases for SnabbVMX\r\n(**program/snabbvmx/tests/end-to-end/selftest.sh**).\r\n\r\nUsually Lua selftests and Bash selftests won't fail, as these tests must\r\nsuccessfully pass on every Snabb deliverable.  End-to-end tests won't fail\r\neither, but it's interesting to learn how to add new SnabbVMX's end-to-end\r\ntests to diagnose potential bugs.\r\n\r\nAll these tests are run by Snabb's Continuous Integration subsystem (snabb-bot).\r\nThe tests can also be run when executing **make test** (only if the Snabb source\r\nis available).\r\n\r\n\r\n### make tests\r\n\r\n```bash\r\n$ sudo make test\r\nTEST      apps.lwaftr.V4V6\r\nTEST      apps.lwaftr.nh_fwd\r\n...\r\nTEST      program/snabbvmx/tests/selftest.sh\r\nSKIPPED   testlog/program.snabbvmx.tests.selftest.sh\r\nTEST      program/snabbvmx/tests/nexthop/selftest.sh\r\nSKIPPED   testlog/program.snabbvmx.tests.nexthop.selftest.sh\r\n...\r\nTEST      program/snabbvmx/tests/end-to-end/selftest.sh\r\n...\r\nTEST      program/lwaftr/tests/end-to-end/selftest.sh\r\nTEST      program/lwaftr/tests/soaktest/selftest.sh\r\n```\r\nExecution of a test can return 3 values:\r\n\r\n* **TEST**: The test run successfully (exit code 0)\r\n* **SKIPPED**: The test was skipped, usually because it needs access to\r\na physical NIC and its PCI address was not set up (exit code 43).\r\n* **ERROR**: The test finished unexpectedly (exit code > 0 and != 43).\r\n\r\n### Lua selftests\r\n\r\n* **apps/lwaftr/V4V6.lua**: Builds customized IPv4 and IPv4-in-IPv6 packets,\r\nand joins the packets to a single link (*test_join*) or splits the packets\r\nto two different links (*test_split*).\r\n* **apps/lwaftr/nh_fwd.lua**: Builds customized packets and test the 3 code\r\npaths of next-hop forwarder: *from-wire-to-{lwaftr, vm}*,\r\n*from-vm-to-{lwaftr, wire}*, *from-lwaftr-to-{vm, wire}*.\r\n\r\nTo run an individual Lua module (app, program or library) selftest:\r\n\r\n```\r\n$ sudo ./snabb snsh -t apps.lwaftr.V4V6\r\nV4V6: selftest\r\nOK\r\n```\r\n\r\n### Bash selftests\r\n\r\n* **program/snabbvmx/tests/selftest.sh**: Tests Ping, ARP and NDP resolution\r\nby the VM.\r\n* **program/snabbvmx/tests/nexthop/selfttest.sh**: Tests nexthop resolution\r\nby the VM.\r\n\r\nFor instance, the result of executing `snabbvmx/tests/selftest.sh` would be the\r\nfollowing:\r\n\r\n```bash\r\n$ sudo SNABB_PCI0=83:00.0 SNABB_PCI1=03:00.0 \\\r\n       program/snabbvmx/tests/selftest.sh\r\n\r\nLaunch Snabbvmx\r\nWaiting for VM listening on telnet port 5000 to get ready... [OK]\r\nPing to lwAFTR inet side: OK\r\nPing to lwAFTR inet side (Good VLAN): OK\r\nPing to lwAFTR inet side (Bad VLAN): OK\r\nPing to lwAFTR B4 side: OK\r\nPing to lwAFTR B4 side (Good VLAN): OK\r\nPing to lwAFTR B4 side (Bad VLAN): OK\r\nARP request to lwAFTR: OK\r\nARP request to lwAFTR (Good VLAN): OK\r\nARP request to lwAFTR (Bad VLAN): OK\r\nNDP request to lwAFTR: OK\r\nNDP request to lwAFTR (Good VLAN): OK\r\nNDP request to lwAFTR (Bad VLAN): OK\r\n```\r\n\r\nNOTE: To successfully run the test, the `SNABB_PCI0` and `SNABB_PCI1` cards must\r\nbe wired to each other.\r\n\r\nThe test goes through several steps:\r\n\r\n1. Run **SnabbVMX** on NIC **SNABB_PCI0**.\r\n2. Run **QEMU**.\r\n3. Configure VM eth0 interface with MAC, IPv4/IPv6 address, ARP & NDP cache table.\r\n4. Run **tcpreplay** on NIC SNABB_PCI1. Sample packets reach the VM.\r\n5. Outgoing packets from the VM are mirrored to a tap interface (tap0).\r\n6. Capture **responses on tap0** and compare them with the expected results.\r\n\r\nThe input data, as well as the expected output, is at `program/snabbvmx/tests/pcap`.\r\n\r\nThe test validates VLAN packets too.  However, there are no VLAN tagged versions\r\nof the expected output.  The reason is that it is the NIC which tags and untags\r\na packet.  Since the packet did not leave the NIC yet, they come out from the VM\r\nuntagged.\r\n\r\n```bash\r\n$ tcpdump -qns 0 -ter good/arp-request-to-lwAFTR.pcap\r\nreading from file arp-request-to-lwAFTR.pcap, link-type EN10MB (Ethernet)\r\n52:54:00:00:00:01 > ff:ff:ff:ff:ff:ff, 802.1Q, length 46: vlan 333, p 0,\r\nethertype ARP, Request who-has 10.0.1.1 tell 10.0.1.100, length 28\r\n```\r\n\r\nThe other bash selftest validates correct resolution of the nexthop by the VM.\r\n\r\n```bash\r\n$ sudo SNABB_PCI0=03:00.0 SNABB_PCI1=83:00.0 \\\r\n       program/snabbvmx/tests/nexthop/selftest.sh\r\nWaiting for VM listening on telnet port 5000 to get ready... [OK]\r\nResolved MAC inet side: 6a:34:99:99:99:99 [OK]\r\nResolved MAC inet side: 4f:12:99:99:99:99 [OK]\r\n```\r\n\r\nThe test goes through several steps:\r\n\r\n1. Run SnabbVMX on SNABB_PCI0.\r\n2. Run VM.\r\n3. Send packets in loop to SNABB_PCI1 for 10 seconds. Packets reach the VM.\r\n\r\n```bash\r\n$ packetblaster replay -D 10 $PCAP_INPUT/v4v6-256.pcap $SNABB_PCI1\r\n```\r\n\r\n4. Retrieve nexthop values (snabbvmx nexthop).\r\n5. Compare to expected values.\r\n6. Timeout if 10 seconds elapsed.\r\n\r\nNOTE: Currently the test is not working correctly: the returned MAC should be\r\n`02:99:99:99:99:99`.\r\n\r\n\r\n## End-to-end tests - details\r\n\r\n**lwaftr vs snabbvmx**\r\nAs both lwaftr and snabbvmx provide a different functionality and use different config-files, both the lwaftr and snabbvmx have their dedicated end-to-end tests. \r\n\r\nAlthough SnabbVMX works on a single interface, `snabbvmx check` requires that\r\nthe packet split (IPv4 / IPv6) is already done and provides a split output too.\r\n\r\nSnabb's lwAFTR includes an end-to-end test suite counterpart.  In most cases,\r\nthe lwAFTR's correctness will be tested via Snabb's lwAFTR end-to-end tests.\r\nHowever, since SnabbVMX uses a different configuration file, the network design\r\nthat it brings up might be slightly different than Snabb's lwAFTR. For instance,\r\nSnabb's lwAFTR fragmentation is always active, while in SnabbVMX it is an optional\r\nargument, either for IPV4 and IPv6 interfaces.\r\n\r\nModifications in the app chain might execute the lwAFTR's data plane in a different\r\nway, bringing up conditions that were not covered by the lwAFTR's data-plane.\r\nFor this reason a specific end-to-end test suite was added, covering specifics\r\nneeds of SnabbVMX.  If a bug is found, its resolution will most likely happen\r\nin Snabb's lwAFTR code, resulting in the addition of a new test to the lwAFTR's\r\ntest suite.\r\n\r\n\r\n**lwaftr**\r\n```\r\ncd src/program/lwaftr/tests/end-to-end\r\n```\r\n**snabbvmx**\r\n```\r\ncd src/program/snabbvmx/tests/end-to-end\r\n```\r\n\r\n**interactive and scripted tests**\r\nTo develop an end-to-end tests, it's recommended to first run it interactively. Once the config-files, pcaps and counters are derived, the test can be added to the scripted tests in test_env.sh.\r\n\r\n**interactive**\r\nTo run an interactive end-to-end test, either use the snabbvmx or lwaftr app - Have in mind that the test is running with the app specified (lwaftr or snabbvmx)\r\n\r\n- snabb snabbvmx check\r\n- snabb lwaftr check\r\n\r\n**end-to-end interactive usage**\r\n```\r\n$ sudo ./snabb snabbvmx check\r\nUsage: check [-r] CONF V4-IN.PCAP V6-IN.PCAP V4-OUT.PCAP V6-OUT.PCAP\r\n                  [COUNTERS.LUA]\r\n```\r\n\r\nParameters:\r\n\r\n- **CONF**: SnabbVMX (icmp_snabbvmx-lwaftr-xe.cfg) or lwaftr (icmp_snabbvmx-lwaftr-xe1.conf) configuration file.\r\n- **V4-IN.PCAP**: Incoming IPv4 packets (from Internet).\r\n- **V6-IN.PCAP**: Incoming IPv6 packets (from b4).\r\n- **V4-OUT.PCAP**: Outgoing IPv4 packets (to Internet, decapsulated).\r\n- **V6-OUT.PCAP**: Outgoing IPv6 packets (to b4, encapsulated)\r\n- **[COUNTERS.LUA]**: Lua file with counter values. Will be regenerated via [-r] parm\r\n\r\n\r\n## How to run SnabbVMX interactive end-to-end test\r\n\r\nIf you detected an error in the lwAFTR, the first step is to obtain the\r\nconfiguration file that SnabbVMX was using, as well as a copy of lwAFTR's\r\nconfiguration and binding table.  With that information and knowing the error\r\nreport (ping to lwAFTR but it doesn't reply, valid softwire packet doesn't get\r\ndecapsulated, etc), you craft a hand-made packet that meets the testing case.\r\n\r\n**obtaining the config-files**\r\nTo run a test, the following config-files are required:\r\n- the binding-table : binding_table.txt.s\r\n- lwaftr conf       : snabbvmx-lwaftr-xe[0-9].conf\r\n- snabbvmx cfg      : snabbvmx-lwaftr-xe[0-9].cfg\r\n\r\nIf you are running lwaftr check, then snabbvmx config-file (snabbvmx-lwaftr-xe[0-9].cfg) is not required\r\n\r\nIt is fine to copy or manually craft the config-files.\r\nA running snabbvmx can be used as well to copy the config-files from the running container \r\nTo gain the used config-files from the running container, either run the collect-support-infos.sh (https://github.com/mwiget/vmxlwaftr/blob/igalia/SUPPORT-INFO.md)\r\nor execute a shell within the dockers container and copy configs and binding-table from the /tmp directory.\r\n\r\nNote: the check application is just using a single interface. If the running container consists of two or more snabb-instances, then just take one of them for when running the check.\r\n \r\n**collect-support-infos.sh**\r\n```\r\nlab@ubuntu1:~/vmxlwaftr/tests$ ./collect-support-infos.sh lwaftr3-16.2R3\r\ncollecting data in container lwaftr3-16.2R3 ...\r\ntar: Removing leading `/' from member names\r\ntar: stats.xml: Cannot stat: No such file or directory\r\ntar: Removing leading `../' from member names\r\ntar: Exiting with failure status due to previous errors\r\ntransferring data from the container to host ...\r\n-rw-r--r-- 1 lab lab 22700552 Nov  8 13:35 support-info-20161108-1335.tgz\r\n\r\nlab@ubuntu1:~/vmxlwaftr/tests/t1$ tar -tvzf support-info-20161108-1335.tgz\r\n-rw-rw-r-- root/root        55 2016-10-31 14:07 VERSION\r\n-rw-r--r-- root/root      1166 2016-11-08 13:35 snabb_xe0.log\r\n-rw-r--r-- root/root      1167 2016-11-08 13:35 snabb_xe1.log\r\n-rw-r--r-- root/root        18 2016-11-04 15:36 mac_xe0\r\n-rw-r--r-- root/root        18 2016-11-04 15:36 mac_xe1\r\n-rw-r--r-- root/root        16 2016-11-04 15:36 pci_xe0\r\n-rw-r--r-- root/root        16 2016-11-04 15:36 pci_xe1\r\n-rw-r--r-- root/root  86560454 2016-11-08 13:35 binding_table.txt\r\n-rw-r--r-- root/root      7132 2016-11-08 13:35 sysinfo.txt\r\n-rw-r--r-- root/root 139505710 2016-11-04 15:41 binding_table.txt.s\r\n-rw-r--r-- root/root       297 2016-11-04 15:42 snabbvmx-lwaftr-xe0.cfg\r\n-rw-r--r-- root/root       297 2016-11-04 15:42 snabbvmx-lwaftr-xe1.cfg\r\n-rw-r--r-- root/root       377 2016-11-04 15:42 test-snabbvmx-lwaftr-xe0.cfg\r\n-rw-r--r-- root/root       377 2016-11-04 15:43 test-snabbvmx-lwaftr-xe1.cfg\r\n-rw-r--r-- root/root       452 2016-11-04 15:42 snabbvmx-lwaftr-xe0.conf\r\n-rw-r--r-- root/root       377 2016-11-04 15:42 snabbvmx-lwaftr-xe1.conf\r\n-rw-r--r-- root/root      1239 2016-11-08 13:35 config.new\r\n-rw-r--r-- root/root      1699 2016-11-08 13:35 config.new1\r\n-rw-r--r-- root/root      1239 2016-11-04 15:41 config.old\r\n-rwxr-xr-x root/root       167 2016-11-04 15:42 test_snabb_lwaftr_xe0.sh\r\n-rwxr-xr-x root/root       167 2016-11-04 15:43 test_snabb_lwaftr_xe1.sh\r\n-rwxr-xr-x root/root       204 2016-11-04 15:42 test_snabb_snabbvmx_xe0.sh\r\n-rwxr-xr-x root/root       204 2016-11-04 15:43 test_snabb_snabbvmx_xe1.sh\r\n-rw-r--r-- root/root     33499 2016-11-04 15:36 config_drive/vmm-config.tgz\r\n-rw-r--r-- root/root      3126 2016-11-04 15:36 root/.bashrc\r\n-rwxr-xr-x root/root   2707019 2016-10-31 14:06 usr/local/bin/snabb\r\n```\r\n\r\n**/tmp inside docker container**\r\nThe snabbvmx config-files can be derived from the container's shell as well directly\r\n```\r\nlab@ubuntu1:~/vmxlwaftr/tests/t1$ docker exec -ti lwaftr3-16.2R3 bash\r\npid 2654's current affinity mask: fffff\r\npid 2654's new affinity mask: ff3ff\r\nroot@8f5d057b8298:/# ls /tmp/\r\nbinding_table.txt        config.new                              mac_xe0  snabb_xe0.log             snabbvmx-lwaftr-xe1.cfg   test-snabbvmx-lwaftr-xe0.cfg  test_snabb_snabbvmx_xe0.sh  vhost_features_xe1.socket\r\nbinding_table.txt.s      config.new1                             mac_xe1  snabb_xe1.log             snabbvmx-lwaftr-xe1.conf  test-snabbvmx-lwaftr-xe1.cfg  test_snabb_snabbvmx_xe1.sh  vmxhdd.img\r\nbinding_table.txt.s.new  config.old                              pci_xe0  snabbvmx-lwaftr-xe0.cfg   support-info.tgz          test_snabb_lwaftr_xe0.sh      vFPC-20160922.img           xe0.socket\r\nbinding_table.txt.s.o    junos-vmx-x86-64-16.1-20160926.0.qcow2  pci_xe1  snabbvmx-lwaftr-xe0.conf  sysinfo.txt               test_snabb_lwaftr_xe1.sh      vhost_features_xe0.socket   xe1.socket\r\n```\r\nNote: press ctrl p ctrl q to exit the containers shell\r\n\r\n\r\n**some adoption of config-files is required**\r\nThe advantage of snabbvmx is the dynamic next-hop resolution via Junos. When running the the lwaftr or snabbvmx app standalone, then the next-hop resolution via Junos is missing and this must be corrected.\r\n**config as derived from a running vmxlwaftr container**\r\n```\r\nlab@ubuntu1:~/vmxlwaftr/tests/t1$ cat snabbvmx-lwaftr-xe1.cfg\r\nreturn {\r\n  lwaftr = \"snabbvmx-lwaftr-xe1.conf\",\r\n  settings = {\r\n  },\r\n  ipv6_interface = {\r\n    ipv6_address = \"\",\r\n    cache_refresh_interval = 1,\r\n    fragmentation = false,\r\n  },\r\n  ipv4_interface = {\r\n    ipv4_address = \"192.168.5.2\",\r\n    cache_refresh_interval = 1,\r\n    fragmentation = false,\r\n  },\r\n}\r\n```\r\n\r\n**Adopted config to use with vmxlwaftr**\r\nPlease make sure to change and add the below:\r\n- cache_refresh_interval = 0 (turns off next-hop learning via Junos)\r\n- mac_address (thats the own/self mac-address fo lwaftr)\r\n- next_hop_mac (next-hop mac to send the packets to)\r\n\r\nNote: as seen, the snabbvmx config icmp_snabbvmx-lwaftr-xe1.cfg references the snabb-configuration file icmp_snabbvmx-lwaftr-xe1.conf.\r\n\r\n```\r\nlab@ubuntu1:~/latest-snabb-binary/snabb/src/program/snabbvmx/tests/end-to-end/data$ cat icmp_snabbvmx-lwaftr-xe1.cfg\r\nreturn {\r\n  lwaftr = \"icmp_snabbvmx-lwaftr-xe1.conf\",\r\n  settings = {\r\n  },\r\n  ipv6_interface = {\r\n    ipv6_address = \"\",\r\n    cache_refresh_interval = 0,   <<< set this to 0\r\n    fragmentation = false,\r\n    mac_address = \"02:cf:69:15:81:01\",   <<< the lwaftr's own mac address. input pcap match this mac\r\n    next_hop_mac = \"90:e2:ba:94:2a:bc\",  <<< the next-hop mac to use for outgoing packets\r\n  },\r\n  ipv4_interface = {\r\n    ipv4_address = \"192.168.5.2\",\r\n    cache_refresh_interval = 0,   <<< set this to 0\r\n    fragmentation = false,\r\n    mac_address = \"02:cf:69:15:81:01\",   <<< the lwaftr's own mac address. input pcap match this mac\r\n    next_hop_mac = \"90:e2:ba:94:2a:bc\",  <<< the next-hop mac to use for outgoing packets\r\n  },\r\n}\r\n```  \r\n\r\n**The input pcaps**\r\nThe check app requires one or two input pcaps.\r\nIt is ok to:\r\n- only feed V4-IN.PCAP\r\n- only feed the V6-IN.PCAP\r\n- or feed both V4-IN.PCAP and V6-IN.PCAP\r\n\r\nNote for interactive tests: \r\nWhen only feeding one pcap, then the other empty pcap must be the empty.pcap in \r\nsrc/program/snabbvmx/tests/end-to-end/data/empty.pcap\r\n\r\n```\r\nlab@ubuntu1:~/latest-snabb-binary/snabb/src/program/snabbvmx/tests/end-to-end/data$ ls empty.pcap\r\nempty.pcap\r\nlab@ubuntu1:~/latest-snabb-binary/snabb/src/program/snabbvmx/tests/end-to-end/data$ file empty.pcap\r\nempty.pcap: tcpdump capture file (little-endian) - version 2.4 (Ethernet, capture length 65535)\r\nlab@ubuntu1:~/latest-snabb-binary/snabb/src/program/snabbvmx/tests/end-to-end/data$ tcpdump -r empty.pcap\r\nreading from file empty.pcap, link-type EN10MB (Ethernet)\r\n```\r\n\r\n**sample icmp-ipv4-in.pcap**\r\nFor below sample icmp-ipv4-in.pcap 6 * packets are seen. Only the 1st and 3rd frame is matching the binding-table and shall the encapsulated into lw4o6. The remaining 4 are not matching the binding-table. For those 4 packets the setting of  policy_icmpv4_outgoing will define if the  lwaftr will generate icmp dest unreachble or just silently discard the packets.\r\nsee lwaftr config-file: icmp_snabbvmx-lwaftr-xe1.conf.\r\n\r\n```\r\nlab@ubuntu1:~/latest-snabb-binary/snabb/src/program/snabbvmx/tests/end-to-end/data$ cat cg-binding_table.txt.s\r\npsid_map {\r\n  10.10.0.0  {psid_length=6, shift=10}\r\n  10.10.0.1  {psid_length=6, shift=10}\r\n  10.10.0.10 {psid_length=6, shift=10}\r\n}\r\nbr_addresses {\r\n  2a02:587:f700::100,\r\n}\r\nsoftwires {\r\n  { ipv4=10.10.0.0, psid=1, b4=2a02:587:f710::40 }\r\n  { ipv4=10.10.0.0, psid=2, b4=2a02:587:f710::41 }\r\n  { ipv4=10.10.0.0, psid=3, b4=2a02:587:f710::42 }\r\n  { ipv4=10.10.0.0, psid=4, b4=2a02:587:f710::43 }\r\n}\r\n\r\nlab@ubuntu1:~/latest-snabb-binary/snabb/src/program/snabbvmx/tests/end-to-end/data$ tcpdump -r icmp-ipv4-in.pcap -e\r\nreading from file icmp-ipv4-in.pcap, link-type EN10MB (Ethernet)\r\n11:27:10.976929 90:e2:ba:94:2a:bc (oui Unknown) > 02:cf:69:15:81:01 (oui Unknown), ethertype IPv4 (0x0800), length 42: 10.0.1.100.domain > 10.10.0.0.1024: [|domain]\r\n11:27:10.988846 90:e2:ba:94:2a:bc (oui Unknown) > 02:cf:69:15:81:01 (oui Unknown), ethertype IPv4 (0x0800), length 42: 10.0.1.100.domain > 10.10.0.0.domain: [|domain]\r\n11:27:11.001278 90:e2:ba:94:2a:bc (oui Unknown) > 02:cf:69:15:81:01 (oui Unknown), ethertype IPv4 (0x0800), length 242: 10.0.1.100 > 10.10.0.0: ICMP echo reply, id 1024, seq 0, length 208\r\n11:27:11.017448 90:e2:ba:94:2a:bc (oui Unknown) > 02:cf:69:15:81:01 (oui Unknown), ethertype IPv4 (0x0800), length 242: 10.0.1.100 > 10.10.0.0: ICMP echo reply, id 53, seq 0, length 208\r\n11:27:11.029226 90:e2:ba:94:2a:bc (oui Unknown) > 02:cf:69:15:81:01 (oui Unknown), ethertype IPv4 (0x0800), length 242: 192.168.0.0 > 192.168.1.100: ICMP echo reply, id 1024, seq 0, length 208\r\n11:27:11.040811 90:e2:ba:94:2a:bc (oui Unknown) > 02:cf:69:15:81:01 (oui Unknown), ethertype IPv4 (0x0800), length 42: 192.168.0.0.domain > 192.168.1.100.1024: [|domain]\r\n```\r\n\r\n\r\n\r\n**running the interactive check**\r\n\r\n```\r\nlab@ubuntu1:~/latest-snabb-binary/snabb/src/program/snabbvmx/tests/end-to-end/data$ sudo ~/latest-snabb-binary/snabb/src/snabb snabbvmx check -r  ./icmp_snabbvmx-lwaftr-xe1.cfg \"./icmp-ipv4-in.pcap\" \"./empty.pcap\" \"./outv4/outv4.pcap\" \"./outv6/icmp-ipv6-out.pcap\" icmp.lua\r\nloading compiled binding table from ./cg-binding_table.txt.s.o\r\ncompiled binding table ./cg-binding_table.txt.s.o is up to date.\r\nnh_fwd4: cache_refresh_interval set to 0 seconds\r\nnh_fwd4: static next_hop_mac 90:e2:ba:94:2a:bc\r\nnh_fwd6: cache_refresh_interval set to 0 seconds\r\nnh_fwd6: static next_hop_mac 90:e2:ba:94:2a:bc\r\ndone\r\n```\r\n\r\n**results**\r\n\r\nCheck that your output matches what you expect:\r\nFor given input, the output is matching what we expect:\r\n- 2 translated packets\r\n- an empty outv4.pcap. lwaftr did not generate icmp dst-unreachable, because of the \r\n```bash\r\nlab@ubuntu1:~/latest-snabb-binary/snabb/src/program/snabbvmx/tests/end-to-end/data$ tcpdump -n -r outv6/icmp-ipv6-out.pcap\r\nreading from file outv6/icmp-ipv6-out.pcap, link-type EN10MB (Ethernet)\r\n01:00:00.000000 IP6 2a02:587:f700::100 > 2a02:587:f710::40: IP 10.0.1.100.53 > 10.10.0.0.1024: [|domain]\r\n01:00:00.000000 IP6 2a02:587:f700::100 > 2a02:587:f710::40: IP 10.0.1.100 > 10.10.0.0: ICMP echo reply, id 1024, seq 0, length 208\r\n\r\nlab@ubuntu1:~/latest-snabb-binary/snabb/src/program/snabbvmx/tests/end-to-end/data$ tcpdump -n -r outv4/outv4.pcap\r\nreading from file outv4/outv4.pcap, link-type EN10MB (Ethernet)\r\n```\r\n\r\n**checking the counters**\r\nThe -r flag is set, as such the resulting counters file icmp.lua is generated freshly. The file does match our expectation:\r\n- 2 * IPv6 packets out >> [\"out-ipv6-packets\"] = 2\r\n- 4 packets dropped becasue of the policy, as such outv4.pcap is an empty pcap-file\r\n```\r\nlab@ubuntu1:~/latest-snabb-binary/snabb/src/program/snabbvmx/tests/end-to-end/data$ grep policy_icmpv4_outgoing icmp_snabbvmx-lwaftr-xe1.conf\r\npolicy_icmpv4_outgoing = drop,\r\n```\r\n\r\n**cat icmp.lua**\r\n```\r\nlab@ubuntu1:~/latest-snabb-binary/snabb/src/program/snabbvmx/tests/end-to-end/data$ cat icmp.lua\r\nreturn {\r\n   [\"drop-all-ipv4-iface-bytes\"] = 568,\r\n   [\"drop-all-ipv4-iface-packets\"] = 4,\r\n   [\"drop-no-dest-softwire-ipv4-bytes\"] = 568,\r\n   [\"drop-no-dest-softwire-ipv4-packets\"] = 4,\r\n   [\"drop-out-by-policy-icmpv4-packets\"] = 4,\r\n   [\"in-ipv4-bytes\"] = 852,\r\n   [\"in-ipv4-packets\"] = 6,\r\n   [\"out-ipv6-bytes\"] = 364,\r\n   [\"out-ipv6-packets\"] = 2,\r\n}\r\n```\r\n\r\n\r\n**summary interactive check**\r\n\r\nAt this stage the interactive test is finished.\r\nThe following is defined:\r\n- lwaftr and  snabbvmx configs\r\n- binding-table\r\n- input pcaps\r\n\r\nFurther more, the interactive tests produced results, which is:\r\n- expected out V4-OUT.PCAP and V6-OUT.PCAP (which can be empty)\r\n- when using -r switch, then a counters-file got generated as well\r\n\r\nWith known input and results, the test can now be added to the scripted end-to-end.sh - to be executed with all other tests to ensure snabbvmx behaves as it should.\r\n\r\n**Note:**\r\nFurther more, this procedure can be ideally used to report issues!\r\n\r\nThe input-pcaps are Checking what values are in the counters can give you a hint about whether\r\nthings are working correctly or not.\r\n\r\nTip: packets always arrive only in one interface, but the output might be\r\nempty for both interfaces, non-empty, and empty or non-empty for both cases.\r\n\r\n\r\n## adding the icmp-test towards the scripted end-to-end tests\r\n\r\n**structure end-to-end tests**\r\n**program/snabbvmx/tests/end-to-end/selftest.sh**: Runs end-to-end tests\r\n(normal and VLAN).\r\n\r\nThe end-to-end tests are a test suite that tests the correctness of the lwAFTR\r\nlogic.\r\n\r\nFiles in **program/snabbvmx/tests/end-to-end/**:\r\n\r\n* **test_env.sh**: Contains the test cases.\r\n* **core-end-to-end.sh**: Runs the test cases using **snabbvmx check**.\r\n* **data/**: Directory containing sample pcap input, expected pcap output,\r\nconfiguration files and binding tables.\r\n* **end-to-end.sh**: Runs **core-end-to-end.sh** on normal packets.\r\n* **end-to-end-vlan.sh**: Runs **core-end-to-end.sh** on VLAN packets.\r\n* **selftest.sh**: Runs both **end-to-end.sh** and **end-to-end-vlan.sh**\r\n\r\n**Note**: make sure all pcap and config-files are located in the data-drectory\r\n```\r\n/home/lab/latest-snabb-binary/snabb/src/program/snabbvmx/tests/end-to-end/data\r\n```\r\n**Note**: make sure all counter-files counters-drectory\r\n```\r\n/home/lab/latest-snabb-binary/snabb/src/program/snabbvmx/tests/end-to-end/data/counters\r\n```\r\n\r\n**adding the icmp-test**:\r\n\r\nAll tests are defined in the test_env.sh file.\r\nThis file has to be edited to include the icmp-test for example. \r\nThe check app has two criteria's to decide if the test is successful or not:\r\n- check the counters-file\r\n- check the resulting out.pcap file\r\n\r\nWhen running the interactive check with the -r option, then a \"good\" counters-file is created. This counters-file (here icmp.lua) can be referenced in the test_env.sh and if the actual test produced different values as defined in the icmp.lua file, then the test will fail\r\nAnother option is to rely on the outgoing pcaps. When running the interactive check, the resulting out.pcap files have been written. The test_env.sh can now run the same test and compare its pcap with the previous stored (./outv4/outv4.pcap , ./outv6/icmp-ipv6-out.pcap) files. If the files differ, then the test fails.\r\n\r\n**pass-criteria based on counters-file**\r\n```\r\nlab@ubuntu1:~/latest-snabb-binary/snabb/src/program/snabbvmx/tests/end-to-end$ vi test_env.sh\r\n...\r\nTEST_DATA=(\r\n    \"ICMP Test - pass-critria out.pcap\"\r\n    \"icmp_snabbvmx-lwaftr-xe1.cfg\" \"icmp-ipv4-in.pcap\" \"\" \"\" \"\"\r\n    \"icmp.lua\"\r\n...\r\n    \"IPv6 fragments and fragmentation is off\"\r\n    \"snabbvmx-lwaftr-xe1.cfg\" \"\" \"regressiontest-signedntohl-frags.pcap\" \"\" \"\"\r\n    \"drop-all-ipv6-fragments.lua\"\r\n)\r\n```\r\nRunning the end-to end.sh based on this configuration file shall not try to compare the out.pcap files. It shall only if the resulting counters are matching the previous defined icmp.lua counters.\r\nIf counters do match, then the test passed.\r\n\r\n \r\n**pass-criteria based on pcap-file**\r\n```\r\nlab@ubuntu1:~/latest-snabb-binary/snabb/src/program/snabbvmx/tests/end-to-end$ vi test_env.sh\r\n...\r\n# Contains an array of test cases.\r\n#\r\n# A test case is a group of 7 data fields, structured as 3 rows:\r\n#  - \"test_name\"\r\n#  - \"snabbvmx_conf\" \"v4_in.pcap\" \"v6_in.pcap\" \"v4_out.pcap\" \"v6_out.pcap\"\r\n#  - \"counters\"\r\n#\r\n# Notice spaces and new lines are not taken into account.\r\nTEST_DATA=(\r\n    \"ICMP Test - pass-critria out.pcap\"\r\n    \"icmp_snabbvmx-lwaftr-xe1.cfg\" \"icmp-ipv4-in.pcap\" \"empty.pcap\" \"\" \"icmp-ipv6-out.pcap\"\r\n    \"\"\r\n\r\n    \"IPv6 fragments and fragmentation is off\"\r\n    \"snabbvmx-lwaftr-xe1.cfg\" \"\" \"regressiontest-signedntohl-frags.pcap\" \"\" \"\"\r\n    \"drop-all-ipv6-fragments.lua\"\r\n)\r\n\r\n\r\nlab@ubuntu1:~/latest-snabb-binary/snabb/src/program/snabbvmx/tests/end-to-end$ sudo ./end-to-end.sh\r\nTesting: ICMP Test - pass-critria out.pcap\r\nloading compiled binding table from data/cg-binding_table.txt.s.o\r\ncompiled binding table data/cg-binding_table.txt.s.o is up to date.\r\nnh_fwd4: cache_refresh_interval set to 0 seconds\r\nnh_fwd4: static next_hop_mac 90:e2:ba:94:2a:bc\r\nnh_fwd6: cache_refresh_interval set to 0 seconds\r\nnh_fwd6: static next_hop_mac 90:e2:ba:94:2a:bc\r\ndone\r\nTest passed\r\nTesting: IPv6 fragments and fragmentation is off\r\nloading compiled binding table from data/binding_table.txt.s.o\r\ncompiled binding table data/binding_table.txt.s.o is up to date.\r\nnh_fwd4: cache_refresh_interval set to 0 seconds\r\nnh_fwd4: static next_hop_mac 90:e2:ba:94:2a:bc\r\nnh_fwd6: cache_refresh_interval set to 0 seconds\r\nnh_fwd6: static next_hop_mac 90:e2:ba:94:2a:bc\r\ndone\r\nTest passed\r\nAll end-to-end lwAFTR tests passed.\r\n```\r\n\r\n\r\ndiff README.troubleshooting.md cg-trouble.md > diff.txt\r\n[diff.txt.gz](https://github.com/snabbco/snabb/files/580367/diff.txt.gz)\r\n\r\nthe updated doc\r\n[cg-trouble.md.gz](https://github.com/snabbco/snabb/files/580368/cg-trouble.md.gz)\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n","created_at":"2016-11-09T09:34:22Z","updated_at":"2016-11-09T12:08:30Z","closed_at":"2016-11-09T12:08:30Z","merged_at":null,"merge_commit_sha":"dea966dd2b0a421dde4e5a58a85d38fc7df6d001","assignee":{"login":"kbara","id":1588250,"avatar_url":"https://avatars.githubusercontent.com/u/1588250?v=3","gravatar_id":"","url":"https://api.github.com/users/kbara","html_url":"https://github.com/kbara","followers_url":"https://api.github.com/users/kbara/followers","following_url":"https://api.github.com/users/kbara/following{/other_user}","gists_url":"https://api.github.com/users/kbara/gists{/gist_id}","starred_url":"https://api.github.com/users/kbara/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kbara/subscriptions","organizations_url":"https://api.github.com/users/kbara/orgs","repos_url":"https://api.github.com/users/kbara/repos","events_url":"https://api.github.com/users/kbara/events{/privacy}","received_events_url":"https://api.github.com/users/kbara/received_events","type":"User","site_admin":false},"assignees":[{"login":"kbara","id":1588250,"avatar_url":"https://avatars.githubusercontent.com/u/1588250?v=3","gravatar_id":"","url":"https://api.github.com/users/kbara","html_url":"https://github.com/kbara","followers_url":"https://api.github.com/users/kbara/followers","following_url":"https://api.github.com/users/kbara/following{/other_user}","gists_url":"https://api.github.com/users/kbara/gists{/gist_id}","starred_url":"https://api.github.com/users/kbara/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kbara/subscriptions","organizations_url":"https://api.github.com/users/kbara/orgs","repos_url":"https://api.github.com/users/kbara/repos","events_url":"https://api.github.com/users/kbara/events{/privacy}","received_events_url":"https://api.github.com/users/kbara/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/snabbco/snabb/pulls/1065/commits","review_comments_url":"https://api.github.com/repos/snabbco/snabb/pulls/1065/comments","review_comment_url":"https://api.github.com/repos/snabbco/snabb/pulls/comments{/number}","comments_url":"https://api.github.com/repos/snabbco/snabb/issues/1065/comments","statuses_url":"https://api.github.com/repos/snabbco/snabb/statuses/3f7f744c95d774531510cba93c8b3ec4bb0f310a","head":{"label":"Igalia:lwaftr","ref":"lwaftr","sha":"3f7f744c95d774531510cba93c8b3ec4bb0f310a","user":{"login":"Igalia","id":1576566,"avatar_url":"https://avatars.githubusercontent.com/u/1576566?v=3","gravatar_id":"","url":"https://api.github.com/users/Igalia","html_url":"https://github.com/Igalia","followers_url":"https://api.github.com/users/Igalia/followers","following_url":"https://api.github.com/users/Igalia/following{/other_user}","gists_url":"https://api.github.com/users/Igalia/gists{/gist_id}","starred_url":"https://api.github.com/users/Igalia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Igalia/subscriptions","organizations_url":"https://api.github.com/users/Igalia/orgs","repos_url":"https://api.github.com/users/Igalia/repos","events_url":"https://api.github.com/users/Igalia/events{/privacy}","received_events_url":"https://api.github.com/users/Igalia/received_events","type":"Organization","site_admin":false},"repo":{"id":34452469,"name":"snabb","full_name":"Igalia/snabb","owner":{"login":"Igalia","id":1576566,"avatar_url":"https://avatars.githubusercontent.com/u/1576566?v=3","gravatar_id":"","url":"https://api.github.com/users/Igalia","html_url":"https://github.com/Igalia","followers_url":"https://api.github.com/users/Igalia/followers","following_url":"https://api.github.com/users/Igalia/following{/other_user}","gists_url":"https://api.github.com/users/Igalia/gists{/gist_id}","starred_url":"https://api.github.com/users/Igalia/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Igalia/subscriptions","organizations_url":"https://api.github.com/users/Igalia/orgs","repos_url":"https://api.github.com/users/Igalia/repos","events_url":"https://api.github.com/users/Igalia/events{/privacy}","received_events_url":"https://api.github.com/users/Igalia/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Igalia/snabb","description":"Snabb Switch: Fast open source packet processing","fork":true,"url":"https://api.github.com/repos/Igalia/snabb","forks_url":"https://api.github.com/repos/Igalia/snabb/forks","keys_url":"https://api.github.com/repos/Igalia/snabb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Igalia/snabb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Igalia/snabb/teams","hooks_url":"https://api.github.com/repos/Igalia/snabb/hooks","issue_events_url":"https://api.github.com/repos/Igalia/snabb/issues/events{/number}","events_url":"https://api.github.com/repos/Igalia/snabb/events","assignees_url":"https://api.github.com/repos/Igalia/snabb/assignees{/user}","branches_url":"https://api.github.com/repos/Igalia/snabb/branches{/branch}","tags_url":"https://api.github.com/repos/Igalia/snabb/tags","blobs_url":"https://api.github.com/repos/Igalia/snabb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Igalia/snabb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Igalia/snabb/git/refs{/sha}","trees_url":"https://api.github.com/repos/Igalia/snabb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Igalia/snabb/statuses/{sha}","languages_url":"https://api.github.com/repos/Igalia/snabb/languages","stargazers_url":"https://api.github.com/repos/Igalia/snabb/stargazers","contributors_url":"https://api.github.com/repos/Igalia/snabb/contributors","subscribers_url":"https://api.github.com/repos/Igalia/snabb/subscribers","subscription_url":"https://api.github.com/repos/Igalia/snabb/subscription","commits_url":"https://api.github.com/repos/Igalia/snabb/commits{/sha}","git_commits_url":"https://api.github.com/repos/Igalia/snabb/git/commits{/sha}","comments_url":"https://api.github.com/repos/Igalia/snabb/comments{/number}","issue_comment_url":"https://api.github.com/repos/Igalia/snabb/issues/comments{/number}","contents_url":"https://api.github.com/repos/Igalia/snabb/contents/{+path}","compare_url":"https://api.github.com/repos/Igalia/snabb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Igalia/snabb/merges","archive_url":"https://api.github.com/repos/Igalia/snabb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Igalia/snabb/downloads","issues_url":"https://api.github.com/repos/Igalia/snabb/issues{/number}","pulls_url":"https://api.github.com/repos/Igalia/snabb/pulls{/number}","milestones_url":"https://api.github.com/repos/Igalia/snabb/milestones{/number}","notifications_url":"https://api.github.com/repos/Igalia/snabb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Igalia/snabb/labels{/name}","releases_url":"https://api.github.com/repos/Igalia/snabb/releases{/id}","deployments_url":"https://api.github.com/repos/Igalia/snabb/deployments","created_at":"2015-04-23T11:35:43Z","updated_at":"2016-10-24T11:13:48Z","pushed_at":"2016-11-09T11:51:49Z","git_url":"git://github.com/Igalia/snabb.git","ssh_url":"git@github.com:Igalia/snabb.git","clone_url":"https://github.com/Igalia/snabb.git","svn_url":"https://github.com/Igalia/snabb","homepage":"","size":16875,"stargazers_count":9,"watchers_count":9,"language":"Lua","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3,"mirror_url":null,"open_issues_count":38,"forks":3,"open_issues":38,"watchers":9,"default_branch":"lwaftr"}},"base":{"label":"snabbco:lwaftr","ref":"lwaftr","sha":"80c622dd945867ce44007a123d4244525ffd9b9e","user":{"login":"snabbco","id":1913310,"avatar_url":"https://avatars.githubusercontent.com/u/1913310?v=3","gravatar_id":"","url":"https://api.github.com/users/snabbco","html_url":"https://github.com/snabbco","followers_url":"https://api.github.com/users/snabbco/followers","following_url":"https://api.github.com/users/snabbco/following{/other_user}","gists_url":"https://api.github.com/users/snabbco/gists{/gist_id}","starred_url":"https://api.github.com/users/snabbco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/snabbco/subscriptions","organizations_url":"https://api.github.com/users/snabbco/orgs","repos_url":"https://api.github.com/users/snabbco/repos","events_url":"https://api.github.com/users/snabbco/events{/privacy}","received_events_url":"https://api.github.com/users/snabbco/received_events","type":"Organization","site_admin":false},"repo":{"id":4857740,"name":"snabb","full_name":"snabbco/snabb","owner":{"login":"snabbco","id":1913310,"avatar_url":"https://avatars.githubusercontent.com/u/1913310?v=3","gravatar_id":"","url":"https://api.github.com/users/snabbco","html_url":"https://github.com/snabbco","followers_url":"https://api.github.com/users/snabbco/followers","following_url":"https://api.github.com/users/snabbco/following{/other_user}","gists_url":"https://api.github.com/users/snabbco/gists{/gist_id}","starred_url":"https://api.github.com/users/snabbco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/snabbco/subscriptions","organizations_url":"https://api.github.com/users/snabbco/orgs","repos_url":"https://api.github.com/users/snabbco/repos","events_url":"https://api.github.com/users/snabbco/events{/privacy}","received_events_url":"https://api.github.com/users/snabbco/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/snabbco/snabb","description":"Snabb: Simple and fast packet networking","fork":false,"url":"https://api.github.com/repos/snabbco/snabb","forks_url":"https://api.github.com/repos/snabbco/snabb/forks","keys_url":"https://api.github.com/repos/snabbco/snabb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/snabbco/snabb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/snabbco/snabb/teams","hooks_url":"https://api.github.com/repos/snabbco/snabb/hooks","issue_events_url":"https://api.github.com/repos/snabbco/snabb/issues/events{/number}","events_url":"https://api.github.com/repos/snabbco/snabb/events","assignees_url":"https://api.github.com/repos/snabbco/snabb/assignees{/user}","branches_url":"https://api.github.com/repos/snabbco/snabb/branches{/branch}","tags_url":"https://api.github.com/repos/snabbco/snabb/tags","blobs_url":"https://api.github.com/repos/snabbco/snabb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/snabbco/snabb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/snabbco/snabb/git/refs{/sha}","trees_url":"https://api.github.com/repos/snabbco/snabb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/snabbco/snabb/statuses/{sha}","languages_url":"https://api.github.com/repos/snabbco/snabb/languages","stargazers_url":"https://api.github.com/repos/snabbco/snabb/stargazers","contributors_url":"https://api.github.com/repos/snabbco/snabb/contributors","subscribers_url":"https://api.github.com/repos/snabbco/snabb/subscribers","subscription_url":"https://api.github.com/repos/snabbco/snabb/subscription","commits_url":"https://api.github.com/repos/snabbco/snabb/commits{/sha}","git_commits_url":"https://api.github.com/repos/snabbco/snabb/git/commits{/sha}","comments_url":"https://api.github.com/repos/snabbco/snabb/comments{/number}","issue_comment_url":"https://api.github.com/repos/snabbco/snabb/issues/comments{/number}","contents_url":"https://api.github.com/repos/snabbco/snabb/contents/{+path}","compare_url":"https://api.github.com/repos/snabbco/snabb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/snabbco/snabb/merges","archive_url":"https://api.github.com/repos/snabbco/snabb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/snabbco/snabb/downloads","issues_url":"https://api.github.com/repos/snabbco/snabb/issues{/number}","pulls_url":"https://api.github.com/repos/snabbco/snabb/pulls{/number}","milestones_url":"https://api.github.com/repos/snabbco/snabb/milestones{/number}","notifications_url":"https://api.github.com/repos/snabbco/snabb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/snabbco/snabb/labels{/name}","releases_url":"https://api.github.com/repos/snabbco/snabb/releases{/id}","deployments_url":"https://api.github.com/repos/snabbco/snabb/deployments","created_at":"2012-07-02T09:06:38Z","updated_at":"2016-11-08T21:02:45Z","pushed_at":"2016-11-09T11:51:49Z","git_url":"git://github.com/snabbco/snabb.git","ssh_url":"git@github.com:snabbco/snabb.git","clone_url":"https://github.com/snabbco/snabb.git","svn_url":"https://github.com/snabbco/snabb","homepage":"","size":13192,"stargazers_count":1497,"watchers_count":1497,"language":"Lua","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":194,"mirror_url":null,"open_issues_count":121,"forks":194,"open_issues":121,"watchers":1497,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/snabbco/snabb/pulls/1065"},"html":{"href":"https://github.com/snabbco/snabb/pull/1065"},"issue":{"href":"https://api.github.com/repos/snabbco/snabb/issues/1065"},"comments":{"href":"https://api.github.com/repos/snabbco/snabb/issues/1065/comments"},"review_comments":{"href":"https://api.github.com/repos/snabbco/snabb/pulls/1065/comments"},"review_comment":{"href":"https://api.github.com/repos/snabbco/snabb/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/snabbco/snabb/pulls/1065/commits"},"statuses":{"href":"https://api.github.com/repos/snabbco/snabb/statuses/3f7f744c95d774531510cba93c8b3ec4bb0f310a"}},"merged":false,"mergeable":true,"mergeable_state":"clean","merged_by":null,"comments":2,"review_comments":0,"commits":1326,"additions":10160,"deletions":5513,"changed_files":639}}
{"actor":{"display_login":"wingo"}}