{"action":"created","issue":{"url":"https://api.github.com/repos/apache/cloudstack/issues/1413","repository_url":"https://api.github.com/repos/apache/cloudstack","labels_url":"https://api.github.com/repos/apache/cloudstack/issues/1413/labels{/name}","comments_url":"https://api.github.com/repos/apache/cloudstack/issues/1413/comments","events_url":"https://api.github.com/repos/apache/cloudstack/issues/1413/events","html_url":"https://github.com/apache/cloudstack/pull/1413","id":133438996,"number":1413,"title":"Fix unique mac address per rVPC router","user":{"login":"wilderrodrigues","id":5129209,"avatar_url":"https://avatars.githubusercontent.com/u/5129209?v=3","gravatar_id":"","url":"https://api.github.com/users/wilderrodrigues","html_url":"https://github.com/wilderrodrigues","followers_url":"https://api.github.com/users/wilderrodrigues/followers","following_url":"https://api.github.com/users/wilderrodrigues/following{/other_user}","gists_url":"https://api.github.com/users/wilderrodrigues/gists{/gist_id}","starred_url":"https://api.github.com/users/wilderrodrigues/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wilderrodrigues/subscriptions","organizations_url":"https://api.github.com/users/wilderrodrigues/orgs","repos_url":"https://api.github.com/users/wilderrodrigues/repos","events_url":"https://api.github.com/users/wilderrodrigues/events{/privacy}","received_events_url":"https://api.github.com/users/wilderrodrigues/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-02-13T14:55:28Z","updated_at":"2016-02-14T20:08:42Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/cloudstack/pulls/1413","html_url":"https://github.com/apache/cloudstack/pull/1413","diff_url":"https://github.com/apache/cloudstack/pull/1413.diff","patch_url":"https://github.com/apache/cloudstack/pull/1413.patch"},"body":"This PR fixes an issue we faced yesterday in production with Redundant VPCs and Private Gateways.\r\n\r\nIn that case, we ended up with 2 interfaces - 1 per VPC router - having the same mac address. Also, both interfaces were up, which caused traffic issues."},"comment":{"url":"https://api.github.com/repos/apache/cloudstack/issues/comments/183966121","html_url":"https://github.com/apache/cloudstack/pull/1413#issuecomment-183966121","issue_url":"https://api.github.com/repos/apache/cloudstack/issues/1413","id":183966121,"user":{"login":"wilderrodrigues","id":5129209,"avatar_url":"https://avatars.githubusercontent.com/u/5129209?v=3","gravatar_id":"","url":"https://api.github.com/users/wilderrodrigues","html_url":"https://github.com/wilderrodrigues","followers_url":"https://api.github.com/users/wilderrodrigues/followers","following_url":"https://api.github.com/users/wilderrodrigues/following{/other_user}","gists_url":"https://api.github.com/users/wilderrodrigues/gists{/gist_id}","starred_url":"https://api.github.com/users/wilderrodrigues/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wilderrodrigues/subscriptions","organizations_url":"https://api.github.com/users/wilderrodrigues/orgs","repos_url":"https://api.github.com/users/wilderrodrigues/repos","events_url":"https://api.github.com/users/wilderrodrigues/events{/privacy}","received_events_url":"https://api.github.com/users/wilderrodrigues/received_events","type":"User","site_admin":false},"created_at":"2016-02-14T20:08:42Z","updated_at":"2016-02-14T20:08:42Z","body":"During the tests we found a couple of issues:\r\n\r\n1. Default route was being set to private gateway interface after the master router went through a crash (or stop/start)\r\n2. Once restarting the Old master router, it was going to backup mode (good) but the private gateway interface was set to UP (bad)\r\n\r\nManual Tests:\r\n\r\n1. VPC created\r\n2. Tier added\r\n3. VM created\r\n4. Pub IP acquired\r\n5. ACL + rule created\r\n6. PF created\r\n7. SSH into machine and pigged google\r\n\r\n* Interface and Route on Master\r\n\r\n```\r\n5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000\r\n    link/ether 06:fb:e8:00:00:66 brd ff:ff:ff:ff:ff:ff\r\n    inet 10.0.3.100/24 brd 10.0.3.255 scope global eth3\r\nroot@r-3-VM:~#\r\n\r\nroot@r-3-VM:~# ip route show\r\ndefault via 192.168.23.1 dev eth1 \r\n10.0.1.0/26 dev eth2  proto kernel  scope link  src 10.0.1.16 \r\n10.0.3.0/24 dev eth3  proto kernel  scope link  src 10.0.3.100 \r\n169.254.0.0/16 dev eth0  proto kernel  scope link  src 169.254.1.55 \r\n192.168.23.0/24 dev eth1  proto kernel  scope link  src 192.168.23.53 \r\nroot@r-3-VM:~#\r\n```\r\n\r\nInterface and Route on Backup\r\n\r\n```\r\n5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000\r\n    link/ether 06:7d:6a:00:00:66 brd ff:ff:ff:ff:ff:ff\r\n    inet 10.0.3.100/24 brd 10.0.3.255 scope global eth3\r\nroot@r-4-VM:~#\r\n\r\nroot@r-4-VM:~# ip route show\r\n10.0.1.0/26 dev eth2  proto kernel  scope link  src 10.0.1.34 \r\n169.254.0.0/16 dev eth0  proto kernel  scope link  src 169.254.1.198 \r\nroot@r-4-VM:~#\r\n```\r\n\r\n* Stop Master Router\r\n* Interface and Route on (new) Master\r\n\r\n```\r\n5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000\r\n    link/ether 06:7d:6a:00:00:66 brd ff:ff:ff:ff:ff:ff\r\n    inet 10.0.3.100/24 brd 10.0.3.255 scope global eth3\r\nroot@r-4-VM:~#\r\n\r\nroot@r-4-VM:~# ip route show\r\ndefault via 192.168.23.1 dev eth1 \r\n10.0.1.0/26 dev eth2  proto kernel  scope link  src 10.0.1.34 \r\n10.0.3.0/24 dev eth3  proto kernel  scope link  src 10.0.3.100 \r\n169.254.0.0/16 dev eth0  proto kernel  scope link  src 169.254.1.198 \r\n192.168.23.0/24 dev eth1  proto kernel  scope link  src 192.168.23.53 \r\nroot@r-4-VM:~#\r\n```\r\n\r\n* Restart the (old) Master router\r\n* SSH into the VM and ping google again\r\n\r\n ```\r\nsbpltk1zffh04:sbp_dev wrodrigues$ ssh root@192.168.23.54\r\nroot@192.168.23.54's password: \r\n# ping 8.8.8.8\r\nPING 8.8.8.8 (8.8.8.8): 56 data bytes\r\n64 bytes from 8.8.8.8: seq=0 ttl=47 time=13.045 ms\r\n64 bytes from 8.8.8.8: seq=1 ttl=47 time=12.019 ms\r\n64 bytes from 8.8.8.8: seq=2 ttl=47 time=11.299 ms\r\n^C\r\n--- 8.8.8.8 ping statistics ---\r\n3 packets transmitted, 3 packets received, 0% packet loss\r\nround-trip min/avg/max = 11.299/12.121/13.045 ms\r\n# \r\n```\r\n\r\n* Interface is DOWN on Backup router\r\n\r\n```\r\n5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000\r\n    link/ether 06:fb:e8:00:00:66 brd ff:ff:ff:ff:ff:ff\r\n    inet 10.0.3.100/24 brd 10.0.3.255 scope global eth3\r\nroot@r-3-VM:~# \r\n```\r\n\r\n* Route on the backup router\r\n\r\n```\r\nroot@r-3-VM:~# ip route show\r\n10.0.1.0/26 dev eth2  proto kernel  scope link  src 10.0.1.16 \r\n169.254.0.0/16 dev eth0  proto kernel  scope link  src 169.254.3.56 \r\nroot@r-3-VM:~# \r\n```\r\n\r\n* Routes working fine! SSH into the VM and pinged google again.\r\n\r\n```sbpltk1zffh04:sbp_dev wrodrigues$ ssh root@192.168.23.54\r\nroot@192.168.23.54's password: \r\n# ping 8.8.8.8\r\nPING 8.8.8.8 (8.8.8.8): 56 data bytes\r\n64 bytes from 8.8.8.8: seq=0 ttl=47 time=19.080 ms\r\n64 bytes from 8.8.8.8: seq=1 ttl=47 time=10.590 ms\r\n64 bytes from 8.8.8.8: seq=2 ttl=47 time=13.668 ms\r\n^C\r\n--- 8.8.8.8 ping statistics ---\r\n3 packets transmitted, 3 packets received, 0% packet loss\r\nround-trip min/avg/max = 10.590/14.446/19.080 ms\r\n# exitConnection to 192.168.23.54 closed.\r\nsbpltk1zffh04:sbp_dev wrodrigues$ \r\n```\r\n\r\nI will update the integration test in order to cover the scenario above.\r\n\r\nCheers,\r\nWilder"}}