{"action":"created","issue":{"url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/897","repository_url":"https://api.github.com/repos/rcbops/rpc-openstack","labels_url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/897/labels{/name}","comments_url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/897/comments","events_url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/897/events","html_url":"https://github.com/rcbops/rpc-openstack/issues/897","id":140811882,"number":897,"title":"[r12.0.0rc2] — Neutron failing to bind ports when creating instances","user":{"login":"jpmontez","id":5495,"avatar_url":"https://avatars.githubusercontent.com/u/5495?v=3","gravatar_id":"","url":"https://api.github.com/users/jpmontez","html_url":"https://github.com/jpmontez","followers_url":"https://api.github.com/users/jpmontez/followers","following_url":"https://api.github.com/users/jpmontez/following{/other_user}","gists_url":"https://api.github.com/users/jpmontez/gists{/gist_id}","starred_url":"https://api.github.com/users/jpmontez/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jpmontez/subscriptions","organizations_url":"https://api.github.com/users/jpmontez/orgs","repos_url":"https://api.github.com/users/jpmontez/repos","events_url":"https://api.github.com/users/jpmontez/events{/privacy}","received_events_url":"https://api.github.com/users/jpmontez/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rcbops/rpc-openstack/labels/prio-1","name":"prio-1","color":"eb6420"},{"url":"https://api.github.com/repos/rcbops/rpc-openstack/labels/status-blocked","name":"status-blocked","color":"e11d21"},{"url":"https://api.github.com/repos/rcbops/rpc-openstack/labels/status-doing","name":"status-doing","color":"cccccc"}],"state":"open","locked":false,"assignee":{"login":"d34dh0r53","id":248299,"avatar_url":"https://avatars.githubusercontent.com/u/248299?v=3","gravatar_id":"","url":"https://api.github.com/users/d34dh0r53","html_url":"https://github.com/d34dh0r53","followers_url":"https://api.github.com/users/d34dh0r53/followers","following_url":"https://api.github.com/users/d34dh0r53/following{/other_user}","gists_url":"https://api.github.com/users/d34dh0r53/gists{/gist_id}","starred_url":"https://api.github.com/users/d34dh0r53/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/d34dh0r53/subscriptions","organizations_url":"https://api.github.com/users/d34dh0r53/orgs","repos_url":"https://api.github.com/users/d34dh0r53/repos","events_url":"https://api.github.com/users/d34dh0r53/events{/privacy}","received_events_url":"https://api.github.com/users/d34dh0r53/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/rcbops/rpc-openstack/milestones/39","html_url":"https://github.com/rcbops/rpc-openstack/milestones/r12.0.0rc4","labels_url":"https://api.github.com/repos/rcbops/rpc-openstack/milestones/39/labels","id":1669361,"number":39,"title":"r12.0.0rc4","description":null,"creator":{"login":"d34dh0r53","id":248299,"avatar_url":"https://avatars.githubusercontent.com/u/248299?v=3","gravatar_id":"","url":"https://api.github.com/users/d34dh0r53","html_url":"https://github.com/d34dh0r53","followers_url":"https://api.github.com/users/d34dh0r53/followers","following_url":"https://api.github.com/users/d34dh0r53/following{/other_user}","gists_url":"https://api.github.com/users/d34dh0r53/gists{/gist_id}","starred_url":"https://api.github.com/users/d34dh0r53/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/d34dh0r53/subscriptions","organizations_url":"https://api.github.com/users/d34dh0r53/orgs","repos_url":"https://api.github.com/users/d34dh0r53/repos","events_url":"https://api.github.com/users/d34dh0r53/events{/privacy}","received_events_url":"https://api.github.com/users/d34dh0r53/received_events","type":"User","site_admin":false},"open_issues":4,"closed_issues":0,"state":"open","created_at":"2016-03-28T15:51:34Z","updated_at":"2016-03-28T19:23:17Z","due_on":null,"closed_at":null},"comments":18,"created_at":"2016-03-14T22:13:35Z","updated_at":"2016-03-28T21:03:05Z","closed_at":null,"body":"##### What version (commit) do you encounter this issue on?\r\nr12.0.0rc2\r\n\r\n##### What type of issue is this (bug, feature, documentation, regression)?\r\nBug\r\n\r\n##### What is the expected behavior?\r\nTo create a successful VM with the `openstack server create` command\r\n\r\n##### What is actually happening?\r\nThe instance fails to build. An exception is raised in the Nova and Neutron logs.\r\n\r\nThe logs show that the internal endpoints aren't being used by the Neutron service, but rather the public ones. Because the firewall doesn't allow public API access, connections fail and exceptions are raised.\r\n\r\n##### How can we reproduce this issue?\r\nWithin a customer lab behind a firewall w/ no public access to the APIs, try and create a server.\r\n\r\n##### How can we verify your issue is addressed?\r\nBy logging into the QE team's Sydney lab and using the utility container to try and create a basic server.\r\n\r\n`openstack server create --flavor 2 --image cirros --nic net-id=XXX test`\r\n\r\n##### If this is a large issue, what can be done immediately to alleviate the issue while working on a longer-term solution?\r\nPossibly by force overriding the endpoint used by the Keystone adaptor\r\n\r\n##### What is your evaluation on the importance of this issue, and why?\r\nIf the internal URL isn't respected, many services involving Neutron and Keystone will break.\r\n\r\n##### What branches / milestones do you believe this issue should target?\r\nThe list below is what is currently supported for backports, remove branches you do not wish to target.\r\n- [ ] Master\r\n- [x] liberty-12.0\r\n\r\n```\r\nroot@627551-infra01_utility_container-2484aa62:~# openstack server create --flavor 2 --image cirros --nic net-id=5fadf69c-136f-499d-9f3d-9e79f9bea358 --wait test\r\nError creating server: test\r\n```\r\n\r\nRelevant Neutron log:\r\n```\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'tag': u'12c3241e-e54e-457c-976c-ef0e4be6b3b9', 'name': 'network-vif-deleted', 'server_uuid': u'f5176db2-4604-4917-9987-303616088744'}]\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova Traceback (most recent call last):\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova   File \"/openstack/venvs/neutron-12.0.7/lib/python2.7/site-packages/neutron/notifiers/nova.py\", line 247, in send_events\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova     batched_events)\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova   File \"/openstack/venvs/neutron-12.0.7/lib/python2.7/site-packages/novaclient/v2/contrib/server_external_events.py\", line 39, in create\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova     return_raw=True)\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova   File \"/openstack/venvs/neutron-12.0.7/lib/python2.7/site-packages/novaclient/base.py\", line 172, in _create\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova     _resp, body = self.api.client.post(url, body=body)\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova   File \"/openstack/venvs/neutron-12.0.7/lib/python2.7/site-packages/keystoneclient/adapter.py\", line 176, in post\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova     return self.request(url, 'POST', **kwargs)\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova   File \"/openstack/venvs/neutron-12.0.7/lib/python2.7/site-packages/novaclient/client.py\", line 91, in request\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova     **kwargs)\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova   File \"/openstack/venvs/neutron-12.0.7/lib/python2.7/site-packages/keystoneclient/adapter.py\", line 206, in request\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova     resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova   File \"/openstack/venvs/neutron-12.0.7/lib/python2.7/site-packages/keystoneclient/adapter.py\", line 95, in request\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova     return self.session.request(url, method, **kwargs)\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova   File \"/openstack/venvs/neutron-12.0.7/lib/python2.7/site-packages/keystoneclient/utils.py\", line 337, in inner\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova     return func(*args, **kwargs)\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova   File \"/openstack/venvs/neutron-12.0.7/lib/python2.7/site-packages/keystoneclient/session.py\", line 390, in request\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova     resp = send(**kwargs)\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova   File \"/openstack/venvs/neutron-12.0.7/lib/python2.7/site-packages/keystoneclient/session.py\", line 434, in _send_request\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova     raise exceptions.ConnectionRefused(msg)\r\n2016-03-15 09:09:11.149 3114 ERROR neutron.notifiers.nova ConnectFailure: Unable to establish connection to http://119.XXX:8774/v2.1/f193144181954dfc973717808068b3ee/os-server-external-events\r\n```\r\n\r\nNote that 119.XX is the public IP address for the environment. The correct, internal IP address begins with 172.20.XX."},"comment":{"url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/comments/202579956","html_url":"https://github.com/rcbops/rpc-openstack/issues/897#issuecomment-202579956","issue_url":"https://api.github.com/repos/rcbops/rpc-openstack/issues/897","id":202579956,"user":{"login":"d34dh0r53","id":248299,"avatar_url":"https://avatars.githubusercontent.com/u/248299?v=3","gravatar_id":"","url":"https://api.github.com/users/d34dh0r53","html_url":"https://github.com/d34dh0r53","followers_url":"https://api.github.com/users/d34dh0r53/followers","following_url":"https://api.github.com/users/d34dh0r53/following{/other_user}","gists_url":"https://api.github.com/users/d34dh0r53/gists{/gist_id}","starred_url":"https://api.github.com/users/d34dh0r53/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/d34dh0r53/subscriptions","organizations_url":"https://api.github.com/users/d34dh0r53/orgs","repos_url":"https://api.github.com/users/d34dh0r53/repos","events_url":"https://api.github.com/users/d34dh0r53/events{/privacy}","received_events_url":"https://api.github.com/users/d34dh0r53/received_events","type":"User","site_admin":false},"created_at":"2016-03-28T21:03:05Z","updated_at":"2016-03-28T21:03:05Z","body":"@jacobwagner thanks, let me know"}}