{"action":"created","issue":{"url":"https://api.github.com/repos/apache/cloudstack/issues/2633","repository_url":"https://api.github.com/repos/apache/cloudstack","labels_url":"https://api.github.com/repos/apache/cloudstack/issues/2633/labels{/name}","comments_url":"https://api.github.com/repos/apache/cloudstack/issues/2633/comments","events_url":"https://api.github.com/repos/apache/cloudstack/issues/2633/events","html_url":"https://github.com/apache/cloudstack/issues/2633","id":321695921,"number":2633,"title":"SSVM cannot reconnect after connection disruption if there is an active event.","user":{"login":"PaulAngus","id":4810220,"avatar_url":"https://avatars0.githubusercontent.com/u/4810220?v=4","gravatar_id":"","url":"https://api.github.com/users/PaulAngus","html_url":"https://github.com/PaulAngus","followers_url":"https://api.github.com/users/PaulAngus/followers","following_url":"https://api.github.com/users/PaulAngus/following{/other_user}","gists_url":"https://api.github.com/users/PaulAngus/gists{/gist_id}","starred_url":"https://api.github.com/users/PaulAngus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PaulAngus/subscriptions","organizations_url":"https://api.github.com/users/PaulAngus/orgs","repos_url":"https://api.github.com/users/PaulAngus/repos","events_url":"https://api.github.com/users/PaulAngus/events{/privacy}","received_events_url":"https://api.github.com/users/PaulAngus/received_events","type":"User","site_admin":false},"labels":[{"id":928084492,"url":"https://api.github.com/repos/apache/cloudstack/labels/Critical","name":"Critical","color":"db8172","default":false},{"id":584073449,"url":"https://api.github.com/repos/apache/cloudstack/labels/ssvm","name":"ssvm","color":"c5def5","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/apache/cloudstack/milestones/5","html_url":"https://github.com/apache/cloudstack/milestone/5","labels_url":"https://api.github.com/repos/apache/cloudstack/milestones/5/labels","id":3033298,"number":5,"title":"4.11.1.0","description":"Milestone covering PRs for the next 4.11.1.0 release. All PRs using this milestone should send PR against the 4.11 branch.","creator":{"login":"DaanHoogland","id":2486961,"avatar_url":"https://avatars1.githubusercontent.com/u/2486961?v=4","gravatar_id":"","url":"https://api.github.com/users/DaanHoogland","html_url":"https://github.com/DaanHoogland","followers_url":"https://api.github.com/users/DaanHoogland/followers","following_url":"https://api.github.com/users/DaanHoogland/following{/other_user}","gists_url":"https://api.github.com/users/DaanHoogland/gists{/gist_id}","starred_url":"https://api.github.com/users/DaanHoogland/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DaanHoogland/subscriptions","organizations_url":"https://api.github.com/users/DaanHoogland/orgs","repos_url":"https://api.github.com/users/DaanHoogland/repos","events_url":"https://api.github.com/users/DaanHoogland/events{/privacy}","received_events_url":"https://api.github.com/users/DaanHoogland/received_events","type":"User","site_admin":false},"open_issues":11,"closed_issues":92,"state":"open","created_at":"2018-01-13T20:06:17Z","updated_at":"2018-05-09T19:27:00Z","due_on":null,"closed_at":null},"comments":0,"created_at":"2018-05-09T19:26:39Z","updated_at":"2018-05-10T07:47:42Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"<!--\r\nVerify first that your issue/request is not already reported on GitHub.\r\nAlso test if the latest release and master branch are affected too.\r\nAlways add information AFTER of these HTML comments, but no need to delete the comments.\r\n-->\r\n\r\n##### ISSUE TYPE\r\n<!-- Pick one below and delete the rest -->\r\n * Bug Report\r\n\r\n##### COMPONENT NAME\r\n<!--\r\nCategorize the issue, e.g. API, VR, VPN, UI, etc.\r\n-->\r\n~~~\r\nSystem VMs\r\n(Maybe also KVM hosts)\r\n~~~\r\n\r\n\r\n##### CLOUDSTACK VERSION\r\n<!--\r\nNew line separated list of affected versions, commit ID for issues on master branch.\r\n-->\r\n~~~\r\n4.11.0\r\n~~~\r\n\r\n##### CONFIGURATION\r\n<!--\r\nInformation about the configuration if relevant, e.g. basic network, advanced networking, etc.  N/A otherwise\r\n-->\r\n\r\n\r\n##### OS / ENVIRONMENT\r\n<!--\r\nInformation about the environment if relevant, N/A otherwise\r\n-->\r\n4.11.0 environment with VMware \r\n\r\n##### SUMMARY\r\n<!-- Explain the problem/feature briefly -->\r\n\r\nIf there is an interruption to mgmt server <-> agent communications while an action is taking place (such as the mgmt server restarting when the  ssvm is performing a snapshot) the SSVM will not be able to reconnect due to following error:\r\n2018-05-09 11:37:09,403 INFO  [cloud.agent.Agent] (Agent-Handler-9:null) Lost connection to host: 10.220.136.127. Dealing with the remaining commands...\r\n2018-05-09 11:37:09,404 INFO  [cloud.agent.Agent] (Agent-Handler-9:null) Cannot connect because we still have 1 commands in progress.\r\n\r\n\r\n##### STEPS TO REPRODUCE\r\n<!--\r\nFor bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate.\r\n\r\nFor new features, show how the feature would be used.\r\n-->\r\n\r\n<!-- Paste example playbooks or commands between quotes below -->\r\n~~~\r\nDuring a volume snapshot exporting the ovf restart the management server.  \r\n~~~\r\n\r\n<!-- You can also paste gist.github.com links for larger files -->\r\n\r\n##### EXPECTED RESULTS\r\n<!-- What did you expect to happen when running the steps above? -->\r\n\r\n~~~\r\nSSVM reconnects.\r\n~~~\r\n\r\n##### ACTUAL RESULTS\r\n<!-- What actually happened? -->\r\n\r\n<!-- Paste verbatim command output between quotes below -->\r\n~~~\r\nThe storage VM does not reconnect to the management server and has an error such as: \r\nINFO  [cloud.agent.Agent] (Agent-Handler-9:null) Lost connection to host: 10.220.136.127. Dealing with the remaining commands...\r\nINFO  [cloud.agent.Agent] (Agent-Handler-9:null) Cannot connect because we still have 1 commands in progress.\r\n\r\nOnce the job had finished it will reconnect but until this point all other jobs failed unless there is another secondary storage vm up and running.\r\nThe backup job even though it is forced to complete from secondary storage is still left in the db as state backing up forever so it does not make that it even waiting for it to finish.\r\n~~~\r\n"},"comment":{"url":"https://api.github.com/repos/apache/cloudstack/issues/comments/387979511","html_url":"https://github.com/apache/cloudstack/issues/2633#issuecomment-387979511","issue_url":"https://api.github.com/repos/apache/cloudstack/issues/2633","id":387979511,"user":{"login":"rhtyd","id":95203,"avatar_url":"https://avatars0.githubusercontent.com/u/95203?v=4","gravatar_id":"","url":"https://api.github.com/users/rhtyd","html_url":"https://github.com/rhtyd","followers_url":"https://api.github.com/users/rhtyd/followers","following_url":"https://api.github.com/users/rhtyd/following{/other_user}","gists_url":"https://api.github.com/users/rhtyd/gists{/gist_id}","starred_url":"https://api.github.com/users/rhtyd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhtyd/subscriptions","organizations_url":"https://api.github.com/users/rhtyd/orgs","repos_url":"https://api.github.com/users/rhtyd/repos","events_url":"https://api.github.com/users/rhtyd/events{/privacy}","received_events_url":"https://api.github.com/users/rhtyd/received_events","type":"User","site_admin":false},"created_at":"2018-05-10T07:47:42Z","updated_at":"2018-05-10T07:47:42Z","author_association":"MEMBER","body":"@PaulAngus This is by design of agent (not specific to any cloudstack version), any pending job in Agent's internal queue will block it from reconnecting until the job finishes. /cc @nvazquez @DaanHoogland - any comments on how to deal with this?"}}
{"actor":{"display_login":"rhtyd"}}