{"action":"created","issue":{"url":"https://api.github.com/repos/vmware/vic/issues/7410","repository_url":"https://api.github.com/repos/vmware/vic","labels_url":"https://api.github.com/repos/vmware/vic/issues/7410/labels{/name}","comments_url":"https://api.github.com/repos/vmware/vic/issues/7410/comments","events_url":"https://api.github.com/repos/vmware/vic/issues/7410/events","html_url":"https://github.com/vmware/vic/issues/7410","id":300789238,"node_id":"MDU6SXNzdWUzMDA3ODkyMzg=","number":7410,"title":"Concurrent Exec tests intermittently fail with various errors","user":{"login":"matthewavery","id":8620942,"node_id":"MDQ6VXNlcjg2MjA5NDI=","avatar_url":"https://avatars2.githubusercontent.com/u/8620942?v=4","gravatar_id":"","url":"https://api.github.com/users/matthewavery","html_url":"https://github.com/matthewavery","followers_url":"https://api.github.com/users/matthewavery/followers","following_url":"https://api.github.com/users/matthewavery/following{/other_user}","gists_url":"https://api.github.com/users/matthewavery/gists{/gist_id}","starred_url":"https://api.github.com/users/matthewavery/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthewavery/subscriptions","organizations_url":"https://api.github.com/users/matthewavery/orgs","repos_url":"https://api.github.com/users/matthewavery/repos","events_url":"https://api.github.com/users/matthewavery/events{/privacy}","received_events_url":"https://api.github.com/users/matthewavery/received_events","type":"User","site_admin":false},"labels":[{"id":327810897,"node_id":"MDU6TGFiZWwzMjc4MTA4OTc=","url":"https://api.github.com/repos/vmware/vic/labels/component/portlayer/execution","name":"component/portlayer/execution","color":"B2EBF2","default":false},{"id":327811117,"node_id":"MDU6TGFiZWwzMjc4MTExMTc=","url":"https://api.github.com/repos/vmware/vic/labels/component/portlayer/interaction","name":"component/portlayer/interaction","color":"B2EBF2","default":false},{"id":310959982,"node_id":"MDU6TGFiZWwzMTA5NTk5ODI=","url":"https://api.github.com/repos/vmware/vic/labels/kind/defect","name":"kind/defect","color":"C5CAE9","default":false},{"id":825364638,"node_id":"MDU6TGFiZWw4MjUzNjQ2Mzg=","url":"https://api.github.com/repos/vmware/vic/labels/priority/p1","name":"priority/p1","color":"d96605","default":false},{"id":465799667,"node_id":"MDU6TGFiZWw0NjU3OTk2Njc=","url":"https://api.github.com/repos/vmware/vic/labels/source/customer","name":"source/customer","color":"C8E6C9","default":false},{"id":680163399,"node_id":"MDU6TGFiZWw2ODAxNjMzOTk=","url":"https://api.github.com/repos/vmware/vic/labels/team/foundation","name":"team/foundation","color":"FFE0B2","default":false}],"state":"open","locked":false,"assignee":{"login":"lcastellano","id":7454168,"node_id":"MDQ6VXNlcjc0NTQxNjg=","avatar_url":"https://avatars3.githubusercontent.com/u/7454168?v=4","gravatar_id":"","url":"https://api.github.com/users/lcastellano","html_url":"https://github.com/lcastellano","followers_url":"https://api.github.com/users/lcastellano/followers","following_url":"https://api.github.com/users/lcastellano/following{/other_user}","gists_url":"https://api.github.com/users/lcastellano/gists{/gist_id}","starred_url":"https://api.github.com/users/lcastellano/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lcastellano/subscriptions","organizations_url":"https://api.github.com/users/lcastellano/orgs","repos_url":"https://api.github.com/users/lcastellano/repos","events_url":"https://api.github.com/users/lcastellano/events{/privacy}","received_events_url":"https://api.github.com/users/lcastellano/received_events","type":"User","site_admin":false},"assignees":[{"login":"lcastellano","id":7454168,"node_id":"MDQ6VXNlcjc0NTQxNjg=","avatar_url":"https://avatars3.githubusercontent.com/u/7454168?v=4","gravatar_id":"","url":"https://api.github.com/users/lcastellano","html_url":"https://github.com/lcastellano","followers_url":"https://api.github.com/users/lcastellano/followers","following_url":"https://api.github.com/users/lcastellano/following{/other_user}","gists_url":"https://api.github.com/users/lcastellano/gists{/gist_id}","starred_url":"https://api.github.com/users/lcastellano/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lcastellano/subscriptions","organizations_url":"https://api.github.com/users/lcastellano/orgs","repos_url":"https://api.github.com/users/lcastellano/repos","events_url":"https://api.github.com/users/lcastellano/events{/privacy}","received_events_url":"https://api.github.com/users/lcastellano/received_events","type":"User","site_admin":false}],"milestone":null,"comments":12,"created_at":"2018-02-27T20:55:31Z","updated_at":"2018-09-05T19:48:30Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"After PR #6787 exec has become more stable in the concurrent and container shutdown paths. However, it still suffers from some race condition style failures. This has mainly been seen in the CI environment and has passed locally(consecutively) against a high resource complex VSAN deployment. below is a short catalog of the failures that have been seen that will cause *intermittent* failures: \r\n\r\n`Simple Concurrent Exec`:\r\n* Failed with a `Feb 26 2018 16:27:06.142Z ERROR op=301.310: CommitHandler error on handle(c13c27abff7908e6146498325fa944c3) for 46bec94ca67b332d0f924df4f4d7c84168693b876a8aa15658050b2e3b2bf46e: The operation is not allowed in the current state.`\r\n\r\n`Exec During Poweroff Of A Container Performing A Long Running Task`: \r\n* Failed because all execs actually succeeded(might need to improve test expectations)\r\n* Container reported being in an invalid state. \r\n\r\n`Exec During Poweroff Of A Container Performing A Short Running Task`:\r\n  * Long running exec returned an rc of 1 even though the output looked present.(looks successful in the portlayer of Exec-Failure-1) \r\n\r\nREFERENCE LOGS:\r\n\r\n[Exec-CI-Failure-1-16518.zip](https://github.com/vmware/vic/files/1764632/Exec-CI-Failure-1-16518.zip)\r\n[Exec-CI-Failure-2-16515.zip](https://github.com/vmware/vic/files/1764634/Exec-CI-Failure-2-16515.zip)\r\n[Exec-Failure-3-FROM-FULL-CI.html.zip](https://github.com/vmware/vic/files/1764649/Exec-Failure-3-FROM-FULL-CI.html.zip)\r\n[Exec-Failure-3-FROM-FULL-CI-16510.zip](https://github.com/vmware/vic/files/1764644/Exec-Failure-3-FROM-FULL-CI-16510.zip)\r\n\r\nCurrently using the following for basic concurrency testing:\r\n```bash\r\nc=1;date;id=`docker ps -q | awk '{print$1}'`; for i in `seq 1 $c`; do docker ^Cec $id /bin/echo /tmp/$i &done;for i in `seq 1 $c`;do wait %$i 2>/dev/null;done;date\r\n```\r\n\r\nTODO:\r\n- [ ] investigate why vCenter reconfigure slows down over time with number of execs. Could be number of extraconfig keys in the VM config (we should not be sending the entire set to VC, but VC may be relaying the entire set to ESX - compare/contrast with ESX directy). Could also be that reconfigures result in accrual of state, irrespective of the keys. "},"comment":{"url":"https://api.github.com/repos/vmware/vic/issues/comments/418858555","html_url":"https://github.com/vmware/vic/issues/7410#issuecomment-418858555","issue_url":"https://api.github.com/repos/vmware/vic/issues/7410","id":418858555,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODg1ODU1NQ==","user":{"login":"hickeng","id":3923729,"node_id":"MDQ6VXNlcjM5MjM3Mjk=","avatar_url":"https://avatars1.githubusercontent.com/u/3923729?v=4","gravatar_id":"","url":"https://api.github.com/users/hickeng","html_url":"https://github.com/hickeng","followers_url":"https://api.github.com/users/hickeng/followers","following_url":"https://api.github.com/users/hickeng/following{/other_user}","gists_url":"https://api.github.com/users/hickeng/gists{/gist_id}","starred_url":"https://api.github.com/users/hickeng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hickeng/subscriptions","organizations_url":"https://api.github.com/users/hickeng/orgs","repos_url":"https://api.github.com/users/hickeng/repos","events_url":"https://api.github.com/users/hickeng/events{/privacy}","received_events_url":"https://api.github.com/users/hickeng/received_events","type":"User","site_admin":false},"created_at":"2018-09-05T19:48:30Z","updated_at":"2018-09-05T19:48:30Z","author_association":"MEMBER","body":"@zjs it's just better mitigated. No additional work planned at this time for 1.4.3. Moving to backlog.\r\n\r\n@renmaosheng I'm going to defer any investigation for the prior to comments as we were seeing odd network level issues over this timeframe. If it occurs again we will open a dedicated issue for the output as it's not part of the same symptom set covered by this issue.\r\n"}}
{"actor":{"display_login":"hickeng"}}