{"action":"created","issue":{"url":"https://api.github.com/repos/onfsdn/faucet/issues/10","repository_url":"https://api.github.com/repos/onfsdn/faucet","labels_url":"https://api.github.com/repos/onfsdn/faucet/issues/10/labels{/name}","comments_url":"https://api.github.com/repos/onfsdn/faucet/issues/10/comments","events_url":"https://api.github.com/repos/onfsdn/faucet/issues/10/events","html_url":"https://github.com/onfsdn/faucet/issues/10","id":171713145,"number":10,"title":"faucet_mininet_test inconsistently fails outside docker container","user":{"login":"leonexis","id":3529333,"avatar_url":"https://avatars.githubusercontent.com/u/3529333?v=3","gravatar_id":"","url":"https://api.github.com/users/leonexis","html_url":"https://github.com/leonexis","followers_url":"https://api.github.com/users/leonexis/followers","following_url":"https://api.github.com/users/leonexis/following{/other_user}","gists_url":"https://api.github.com/users/leonexis/gists{/gist_id}","starred_url":"https://api.github.com/users/leonexis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leonexis/subscriptions","organizations_url":"https://api.github.com/users/leonexis/orgs","repos_url":"https://api.github.com/users/leonexis/repos","events_url":"https://api.github.com/users/leonexis/events{/privacy}","received_events_url":"https://api.github.com/users/leonexis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-08-17T17:21:04Z","updated_at":"2016-08-31T23:18:41Z","closed_at":null,"body":"I've been trying to get the `faucet_mininet_test.py` tests running outside a docker container and have all the software requirements installed. The problem is that some tests will fail intermittently and almost seem like it is due to race conditions. This occurs on:\r\n\r\n - Ubuntu 14.04 host > VMware WS 12 > Ubuntu 16.04 desktop guest\r\n - Ubuntu 14.04 host > VMware WS 12 > Ubuntu 16.04 server guest (fresh install for this test)\r\n - Ubuntu 14.04 host > Virtualbox 5 > Ubuntu 16.04 server guest (fresh install for this test)\r\n\r\nHowever, the tests always seem to pass when run in the docker container (built locally from `v1_0-265-g20168ee` in the `onfsd/faucet fork`, as per `README.docker.md`). This works on:\r\n\r\n - Ubuntu 14.04 host > VMware WS 12 > Ubuntu 16.04 desktop guest > docker > reannz/faucet-test\r\n - Ubuntu 14.04 host > docker > reannz/faucet-test\r\n\r\nSince this is working in docker on my Ubuntu 16.04 desktop guest, it should work outside the docker as well. The software requirements come from `faucet_mininet_test.py` itself. \r\n\r\nHowever, when run outside the docker container, sometimes the tests pass, but most of the time it does not. I ran 10 tests in a row and recorded their results (only test and final exception provided in ticket description, attached full logs). Passes 3, 4, and 10 passed the tests completely, the rest failed in some manner. This would suggest that my checked out version of faucet can function properly on a VM.\r\n\r\nI did notice in the docker, however, that Mininet complains that it can't make all the sysctl calls it needs:\r\n\r\n```\r\n================= Starting OVS ==================\r\n * /etc/openvswitch/conf.db does not exist\r\n * Creating empty database /etc/openvswitch/conf.db\r\n * Starting ovsdb-server\r\n * Configuring Open vSwitch system IDs\r\n * Starting ovs-vswitchd\r\n * Enabling remote OVSDB managers\r\n=========== Running faucet unit tests ===========\r\ntest_tagged (faucet_mininet_test.FaucetMultipleDPTaggedTest) ... *** Error setting resource limits. Mininet's performance may be affected.\r\nt4ds1h1 t4ds1h1-eth0.100:s14d-eth1\r\n[...]\r\n```\r\n\r\nThis may point to some weirdness where the test works under Docker conditions with reduced resources, but not directly on a VM, even the VM running the Docker instance. I've tried modifying the tests to run with user-mode OVS only, but got similar results. Since the Docker was run in privileged mode, I would guess that it has access to the kernel DP anyway.\r\n\r\nI hope this information helps. I'm in the process of writing tutorials on setting up the latest dev version of Faucet after my last articles on covering the Ryu API and creating a Ryu controller application based in part on the table structure used in Faucet on Inside-OpenFlow.com. \r\n\r\nIn any case, I would rather write instructions to make this run without having to use Docker and I always try to have the reader run any tests supplied with manually installed software first to ensure software is installed properly before continuing to usage instructions. I plan on covering additional controller applications on top of Faucet, so requiring Docker would get in the way...\r\n\r\nPlease let me know if there is anything I can do to help. I'm willing to test any changes in any environment I can reproduce.\r\n\r\n## Test Runs Outside Docker\r\nAll tests run right after the other on the same guest (Ubuntu 16.04 on VMware, specs at end of this description).\r\n\r\n### Pass 1 - 12 Errors, 1 Failed\r\n[faucet-1.log.txt](https://github.com/onfsdn/faucet/files/423083/faucet-1.log.txt)\r\n```\r\nERROR: __main__.FaucetUntaggedACLTest.test_port5001_blocked (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n\r\nERROR: __main__.FaucetTaggedControlPlaneTest.test_ping_controller (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n\r\nERROR: __main__.FaucetUntaggedSameVlanIPv6RouteTest.test_untagged (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n\r\nERROR: __main__.FaucetUntaggedACLTest.test_port5002_unblocked (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n\r\nERROR: __main__.FaucetUntaggedHostMoveTest.test_untagged (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n\r\nERROR: __main__.FaucetUntaggedACLMirrorTest.test_untagged (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n\r\nERROR: __main__.FaucetUntaggedControlPlaneTest.test_ping_controller (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n\r\nERROR: __main__.FaucetTaggedControlPlaneTest.test_tagged (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n\r\nERROR: __main__.FaucetUntaggedMirrorTest.test_untagged (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n\r\nERROR: __main__.FaucetUntaggedOutputTest.test_untagged (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n\r\nERROR: __main__.FaucetUntaggedTest.test_untagged (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n\r\nERROR: __main__.FaucetTaggedAndUntaggedVlanTest.test_untagged (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n\r\nFAIL: __main__.FaucetUntaggedMaxHostsTest.test_untagged (subunit.RemotedTestCase)\r\nAssertionError: 2 != 3\r\n```\r\n\r\n### Pass 2 - 1 Failed\r\n[faucet-2.log.txt](https://github.com/onfsdn/faucet/files/423084/faucet-2.log.txt)\r\n```\r\nFAIL: __main__.FaucetUntaggedHUPTest.test_untagged (subunit.RemotedTestCase)\r\nAssertionError: 0 != 8.333333333333334\r\n```\r\n\r\n### Pass 3 - All Passed\r\n[faucet-3.log.txt](https://github.com/onfsdn/faucet/files/423085/faucet-3.log.txt)\r\n\r\n### Pass 4 - All Passed\r\n[faucet-4.log.txt](https://github.com/onfsdn/faucet/files/423086/faucet-4.log.txt)\r\n\r\n### Pass 5 - 1 Failed\r\n[faucet-5.log.txt](https://github.com/onfsdn/faucet/files/423090/faucet-5.log.txt)\r\n```\r\nFAIL: test_untagged (__main__.FaucetSingleUntaggedIPv4RouteTest)\r\nAssertionError: None is not true\r\n```\r\n\r\n### Pass 6 - 1 Error\r\n[faucet-6.log.txt](https://github.com/onfsdn/faucet/files/423091/faucet-6.log.txt)\r\n```\r\nERROR: __main__.FaucetUntaggedNoVLanUnicastFloodTest.test_untagged (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n```\r\n\r\n### Pass 7 - 4 Errors\r\n[faucet-7.log.txt](https://github.com/onfsdn/faucet/files/423092/faucet-7.log.txt)\r\n```\r\nERROR: __main__.FaucetTaggedTest.test_tagged (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n\r\nERROR: __main__.FaucetTaggedAndUntaggedTest.test_seperate_untagged_tagged (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n\r\nERROR: __main__.FaucetUntaggedACLTest.test_untagged (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n\r\nERROR: __main__.FaucetTaggedAndUntaggedVlanTest.test_untagged (subunit.RemotedTestCase)\r\nValueError: No JSON object could be decoded\r\n```\r\n\r\n### Pass 8 - 1 Error (Mininet exited in `checkStartup`, see log)\r\n[faucet-8.log.txt](https://github.com/onfsdn/faucet/files/423094/faucet-8.log.txt)\r\n```\r\nERROR: __main__.FaucetMultipleDPTaggedTest.test_tagged (subunit.RemotedTestCase)\r\nSystemExit: 1\r\n```\r\n\r\n### Pass 9 - 2 Failed\r\n[faucet-9.log.txt](https://github.com/onfsdn/faucet/files/423095/faucet-9.log.txt)\r\n```\r\nFAIL: __main__.FaucetUntaggedHostPermanentLearnTest.test_untagged (subunit.RemotedTestCase)\r\nAssertionError: 100.0 != 50.0\r\n\r\nFAIL: __main__.FaucetUntaggedMaxHostsTest.test_untagged (subunit.RemotedTestCase)\r\nAssertionError: 2 != 3\r\n```\r\n\r\n### Pass 10 - Passed\r\n[faucet-10.log.txt](https://github.com/onfsdn/faucet/files/423096/faucet-10.log.txt)\r\n\r\n## System Information\r\n```\r\nHost: Ubuntu 14.04 64bit, AMD FX 8320 8-core with 32GB RAM\r\nVMware: VMware Workstation 12 Pro v12.1.1 build-3770994\r\nDesktop Guest (VMware): Ubuntu 16.04 64bit, 8-cores, 8GB RAM\r\nServer Guest(VMware): Ubuntu 16.04 64bit, 8-cores, 4GB RAM\r\nVirtualBox: VirtualBox v5.0.10 r104061\r\nServer Guest (VirtualBox): Ubuntu 16.04 64bit, 1-core, 4GB RAM\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/onfsdn/faucet/issues/comments/243931521","html_url":"https://github.com/onfsdn/faucet/issues/10#issuecomment-243931521","issue_url":"https://api.github.com/repos/onfsdn/faucet/issues/10","id":243931521,"user":{"login":"shivarammysore","id":1679347,"avatar_url":"https://avatars.githubusercontent.com/u/1679347?v=3","gravatar_id":"","url":"https://api.github.com/users/shivarammysore","html_url":"https://github.com/shivarammysore","followers_url":"https://api.github.com/users/shivarammysore/followers","following_url":"https://api.github.com/users/shivarammysore/following{/other_user}","gists_url":"https://api.github.com/users/shivarammysore/gists{/gist_id}","starred_url":"https://api.github.com/users/shivarammysore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shivarammysore/subscriptions","organizations_url":"https://api.github.com/users/shivarammysore/orgs","repos_url":"https://api.github.com/users/shivarammysore/repos","events_url":"https://api.github.com/users/shivarammysore/events{/privacy}","received_events_url":"https://api.github.com/users/shivarammysore/received_events","type":"User","site_admin":false},"created_at":"2016-08-31T23:18:40Z","updated_at":"2016-08-31T23:18:40Z","body":"Thanks! I'll take a look for differences. On my box here the provided\ninstructions work (always the way isn't it :-).\n\nI'm on for reference:\n\nDescription: Ubuntu 16.04.1 LTS\n\nfaucet@faucet:~/faucet$ uname -a\n\nLinux  faucet 4.4.0-34-generic #53-Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016\nx86_64 x86_64 x86_64 GNU/Linux\n\nOn Thu, Sep 1, 2016 at 10:51 AM, Leo Scott <leonexis@gmail.com> wrote:\n\n> Had just finished my draft to Josh before you sent this. Here it is for\n> the mailing list (short answer: yes, on two platforms).\n>\n> Tests fail when following the README.docker.md instructions. Docker image\n> in privilege mode, with tcpdump apparmor profile removed and openvswitch\n> module loaded before running. Image built with --no-cache to ensure prior\n> builds did not affect test.\n>\n> You can see my complete build and run console log (including commands) in\n> https://github.com/onfsdn/faucet/issues/10, see my last comment. I am\n> performing the exact directions in the README.docker.md file (at least as\n> of this morning). The last comment demonstrates the same issue running the\n> docker image directly on my workstation (I usually use separate VM\n> workspaces for different projects, even when running Docker containers).\n>\n> Otherwise, I use my automated tests at https://github.com/inside-\n> openflow/faucet-vagrant-tests using vagrant to test different\n> installation instructions that I reference, or plan on referencing, in my\n> articles. I don't expect those to pass all the time. I just use them to\n> give me a hint when something might be wrong and I can run the tests\n> manually.\n>\n> Sorry for short response, on US central time and already left work and not\n> at my workstation. Hope the information helps.\n> -Leo\n>\n>\n> On Wed, Aug 31, 2016 at 5:49 PM Shivaram Mysore <shivaram.mysore@gmail.com>\n> wrote:\n>\n>> Hi Leo,\n>>\n>> (included faucet-dev@openflowsdn.org mailing list as others may also be\n>> interested)\n>>\n>> One of our colleagues had a similar problem, when he did not follow\n>> Brad's excellent instructions.\n>>\n>> Can you please confirm if you  did:\n>>\n>> docker build -t reannz/faucet-tests -f Dockerfile.tests .\n>> apparmor_parser -R /etc/apparmor.d/usr.sbin.tcpdump\n>> modprobe openvswitch\n>> sudo docker run --privileged -ti reannz/faucet-tests\n>>\n>> thanks\n>>\n>>\n>> On Wed, Aug 31, 2016 at 11:14 AM, Leo Scott <notifications@github.com>\n>> wrote:\n>>\n>>> I ran the Docker instance directly on my workstation (rather than in a\n>>> VM) and I get the same freeze as before. I pressed Ctrl+C after half an\n>>> hour. This seems to be a new issue. Should I write a new ticket then?\n>>>\n>>> full logs:\n>>>\n>>> docker-build-20160831.txt\n>>> <https://github.com/onfsdn/faucet/files/447971/docker-build-20160831.txt>\n>>> docker-run-20160831.txt\n>>> <https://github.com/onfsdn/faucet/files/447972/docker-run-20160831.txt>\n>>>\n>>> excerpt:\n>>>\n>>> *** Ping: testing ping reachability\n>>> u8c1 -> u8c2 u8c3 u8c4\n>>> u8c2 -> u8c1 u8c3 u8c4\n>>> u8c3 -> u8c1 u8c2 u8c4\n>>> u8c4 -> u8c1 u8c2 u8c3\n>>> *** Results: 0% dropped (12/12 received)\n>>> ..^CTraceback (most recent call last):\n>>>   File \"/usr/local/lib/python2.7/dist-packages/concurrencytest.py\", line 82, in do_fork\n>>>     process_suite.run(subunit_result)\n>>>   File \"/usr/lib/python2.7/unittest/suite.py\", line 108, in run\n>>>     test(result)\n>>>   File \"/usr/lib/python2.7/unittest/case.py\", line 393, in __call__\n>>>     return self.run(*args, **kwds)\n>>>   File \"/usr/lib/python2.7/unittest/case.py\", line 320, in run\n>>>     self.setUp()\n>>>   File \"./faucet_mininet_test.py\", line 1197, in setUp\n>>>     self.start_net()\n>>>   File \"./faucet_mininet_test.py\", line 305, in start_net\n>>>     self.net.waitConnected()\n>>>   File \"build/bdist.linux-x86_64/egg/mininet/net.py\", line 185, in waitConnected\n>>>     if switch.connected():\n>>>   File \"build/bdist.linux-x86_64/egg/mininet/node.py\", line 1118, in connected\n>>>     uuid, 'is_connected' ):\n>>>   File \"build/bdist.linux-x86_64/egg/mininet/node.py\", line 1081, in vsctl\n>>>     return self.cmd( 'ovs-vsctl', *args, **kwargs )\n>>>   File \"build/bdist.linux-x86_64/egg/mininet/node.py\", line 351, in cmd\n>>>     return self.waitOutput( verbose )\n>>>   File \"build/bdist.linux-x86_64/egg/mininet/node.py\", line 338, in waitOutput\n>>>     data = self.monitor( findPid=findPid )\n>>>   File \"build/bdist.linux-x86_64/egg/mininet/node.py\", line 305, in monitor\n>>>     data = self.read( 1024 )\n>>>   File \"build/bdist.linux-x86_64/egg/mininet/node.py\", line 217, in read\n>>>     data = os.read( self.stdout.fileno(), maxbytes - count )\n>>> KeyboardInterrupt\n>>> ETraceback (most recent call last):\n>>>   File \"./faucet_mininet_test.py\", line 1882, in <module>\n>>>     run_tests()\n>>>   File \"./faucet_mininet_test.py\", line 1858, in run_tests\n>>>     results.append(parallel_runner.run(parallel_suite))\n>>>   File \"/usr/lib/python2.7/unittest/runner.py\", line 151, in run\n>>>     test(result)\n>>>   File \"/usr/local/lib/python2.7/dist-packages/unittest2/suite.py\", line 87, in __call__\n>>>     return self.run(*args, **kwds)\n>>>   File \"/usr/local/lib/python2.7/dist-packages/testtools/testsuite.py\", line 95, in run\n>>>     finished_test = queue.get()\n>>>   File \"/usr/lib/python2.7/Queue.py\", line 168, in get\n>>>     self.not_empty.wait()\n>>>   File \"/usr/lib/python2.7/threading.py\", line 340, in wait\n>>>     waiter.acquire()\n>>> KeyboardInterrupt\n>>>\n>>> real    31m2.707s\n>>> user    6m20.525s\n>>> sys     0m58.517s\n>>>\n>>> Container built with --no-cache to ensure the most up-to-date build.\n>>> Workstation has 32GB RAM and 8 (real) cores, so I doubt resources were an\n>>> issue.\n>>>\n>>> —\n>>> You are receiving this because you commented.\n>>>\n>>\n>>> Reply to this email directly, view it on GitHub\n>>> <https://github.com/onfsdn/faucet/issues/10#issuecomment-243852272>, or mute\n>>> the thread\n>>> <https://github.com/notifications/unsubscribe-auth/ABmf8609c-gZ4izyBcBz0mSbb2fIhc6cks5qlcR8gaJpZM4Jmr2l>\n>>> .\n>>>\n>> --\n>> Latest info about Ryu Faucet is available @ http://onfsdn.github.io/\n>> faucet/\n>> ---\n>> You received this message because you are subscribed to the Google Groups\n>> \"Faucet Dev\" group.\n>> To unsubscribe from this group and stop receiving emails from it, send an\n>> email to faucet-dev+unsubscribe@openflowsdn.org.\n>> To post to this group, send email to faucet-dev@openflowsdn.org.\n>> To view this discussion on the web visit https://groups.google.com/a/\n>> openflowsdn.org/d/msgid/faucet-dev/CAJRY%2BjqE7a-J%\n>> 3D4ixG39qNNB8tZWFuGO6gk6gzMEgsiMjKRcHJQ%40mail.gmail.com\n>> <https://groups.google.com/a/openflowsdn.org/d/msgid/faucet-dev/CAJRY%2BjqE7a-J%3D4ixG39qNNB8tZWFuGO6gk6gzMEgsiMjKRcHJQ%40mail.gmail.com?utm_medium=email&utm_source=footer>\n>> .\n>>\n> --\n> Latest info about Ryu Faucet is available @ http://onfsdn.github.io/\n> faucet/\n> ---\n> You received this message because you are subscribed to the Google Groups\n> \"Faucet Dev\" group.\n> To unsubscribe from this group and stop receiving emails from it, send an\n> email to faucet-dev+unsubscribe@openflowsdn.org.\n> To post to this group, send email to faucet-dev@openflowsdn.org.\n> To view this discussion on the web visit https://groups.google.com/a/\n> openflowsdn.org/d/msgid/faucet-dev/CAPVxS6GbBRYGR6KDF8EMVMfsV25im\n> Ud%2Bohhs5_Qpa2KG6WKFng%40mail.gmail.com\n> <https://groups.google.com/a/openflowsdn.org/d/msgid/faucet-dev/CAPVxS6GbBRYGR6KDF8EMVMfsV25imUd%2Bohhs5_Qpa2KG6WKFng%40mail.gmail.com?utm_medium=email&utm_source=footer>\n> .\n>\n"}}
{"actor":{"display_login":"shivarammysore"}}