{"action":"opened","issue":{"url":"https://api.github.com/repos/openebs/openebs/issues/2312","repository_url":"https://api.github.com/repos/openebs/openebs","labels_url":"https://api.github.com/repos/openebs/openebs/issues/2312/labels{/name}","comments_url":"https://api.github.com/repos/openebs/openebs/issues/2312/comments","events_url":"https://api.github.com/repos/openebs/openebs/issues/2312/events","html_url":"https://github.com/openebs/openebs/issues/2312","id":386780234,"node_id":"MDU6SXNzdWUzODY3ODAyMzQ=","number":2312,"title":"(DU): Rescheduled application is taking > 300 sec to come to running state.","user":{"login":"dargasudarshan","id":30148069,"node_id":"MDQ6VXNlcjMwMTQ4MDY5","avatar_url":"https://avatars3.githubusercontent.com/u/30148069?v=4","gravatar_id":"","url":"https://api.github.com/users/dargasudarshan","html_url":"https://github.com/dargasudarshan","followers_url":"https://api.github.com/users/dargasudarshan/followers","following_url":"https://api.github.com/users/dargasudarshan/following{/other_user}","gists_url":"https://api.github.com/users/dargasudarshan/gists{/gist_id}","starred_url":"https://api.github.com/users/dargasudarshan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dargasudarshan/subscriptions","organizations_url":"https://api.github.com/users/dargasudarshan/orgs","repos_url":"https://api.github.com/users/dargasudarshan/repos","events_url":"https://api.github.com/users/dargasudarshan/events{/privacy}","received_events_url":"https://api.github.com/users/dargasudarshan/received_events","type":"User","site_admin":false},"labels":[{"id":1148599806,"node_id":"MDU6TGFiZWwxMTQ4NTk5ODA2","url":"https://api.github.com/repos/openebs/openebs/labels/Data%20Unavailability","name":"Data Unavailability","color":"8def93","default":false},{"id":538374935,"node_id":"MDU6TGFiZWw1MzgzNzQ5MzU=","url":"https://api.github.com/repos/openebs/openebs/labels/kind/bug","name":"kind/bug","color":"ee2211","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-12-03T11:43:41Z","updated_at":"2018-12-03T11:43:41Z","closed_at":null,"author_association":"MEMBER","body":"<!-- Make sure to include as much information as possible so we can fix it as quickly as possible. -->\r\n<!-- If you know how to fix this bug, please open a pull request on https://github.com/openebs/openebs/compare/?template=bugs.md -->\r\n<!-- If you can't answer some sections, please delete them -->\r\n\r\n## Description\r\n<!-- Provide a description of this bug -->\r\n- Rescheduled application is taking > 300 sec to running and start taking IO's.\r\n\r\n## Expected Behavior\r\n<!-- Tell us what should happen -->\r\n-  Rescheduled application should be up and running in less than 120 seconds. \r\n\r\n<!-- Tell us what happens instead -->\r\n-  More time it takes causes data unavailability to the end user.\r\n\r\n## Possible Solution\r\n<!-- Suggest a fix or reason for this bug -->\r\n\r\n## Steps to Reproduce\r\n<!-- Provide a link to a live example or steps to reproduce this bug -->\r\n- Deploy percona application on cStor engine.\r\n- Taint node on which application is running.\r\n>> Observed that application is rescheduled on the other node.\r\nBut it took more than 5 minutes to coming to running state. Mean while the liveness probe failed after 120 seconds.\r\n\r\nRescheduled application pod events:\r\n\r\n```\r\nEvents:\r\n  Type     Reason                  Age               From                                              Message\r\n  ----     ------                  ----              ----                                              -------\r\n  Normal   Scheduled               20m               default-scheduler                                 Successfully assigned app-percona-ns/percona-6966c6db86-fzp5m to ip-10-0-1-87.us-west-1.compute.internal\r\n  Warning  FailedAttachVolume      20m               attachdetach-controller                           Multi-Attach error for volume \"pvc-60d4beaa-f6e0-11e8-b62f-064308797e24\" Volume is already used by pod(s) percona-6966c6db86-fknmm\r\n  Normal   SuccessfulAttachVolume  17m               attachdetach-controller                           AttachVolume.Attach succeeded for volume \"pvc-60d4beaa-f6e0-11e8-b62f-064308797e24\"\r\n  Warning  FailedMount             8m (x2 over 12m)  kubelet, ip-10-0-1-87.us-west-1.compute.internal  MountVolume.WaitForAttach failed for volume \"pvc-60d4beaa-f6e0-11e8-b62f-064308797e24\" : failed to get any path for iscsi disk, last err seen:\r\niscsi: failed to attach disk: Error: iscsiadm: Could not login to [iface: default, target: iqn.2016-09.com.openebs.cstor:pvc-60d4beaa-f6e0-11e8-b62f-064308797e24, portal: 100.71.52.4,3260].\r\niscsiadm: initiator reported error (5 - encountered iSCSI login failure)\r\niscsiadm: Could not log into all portals\r\nLogging in to [iface: default, target: iqn.2016-09.com.openebs.cstor:pvc-60d4beaa-f6e0-11e8-b62f-064308797e24, portal: 100.71.52.4,3260] (multiple)\r\n (exit status 5)\r\n  Warning  FailedMount  5m (x3 over 14m)  kubelet, ip-10-0-1-87.us-west-1.compute.internal  MountVolume.WaitForAttach failed for volume \"pvc-60d4beaa-f6e0-11e8-b62f-064308797e24\" : failed to get any path for iscsi disk, last err seen:\r\niscsi: failed to attach disk: Error: iscsiadm: Could not login to [iface: default, target: iqn.2016-09.com.openebs.cstor:pvc-60d4beaa-f6e0-11e8-b62f-064308797e24, portal: 100.71.52.4,3260].\r\niscsiadm: initiator reported error (8 - connection timed out)\r\niscsiadm: Could not log into all portals\r\nLogging in to [iface: default, target: iqn.2016-09.com.openebs.cstor:pvc-60d4beaa-f6e0-11e8-b62f-064308797e24, portal: 100.71.52.4,3260] (multiple)\r\n (exit status 8)\r\n  Warning  FailedMount  4m (x7 over 18m)  kubelet, ip-10-0-1-87.us-west-1.compute.internal  Unable to mount volumes for pod \"percona-6966c6db86-fzp5m_app-percona-ns(afa77eb2-f6eb-11e8-b62f-064308797e24)\": timeout expired waiting for volumes to attach or mount for pod \"app-percona-ns\"/\"percona-6966c6db86-fzp5m\". list of unmounted volumes=[data-vol]. list of unattached volumes=[data-vol default-token-vf9pq]\r\n  Normal   Pulled       4m                kubelet, ip-10-0-1-87.us-west-1.compute.internal  Container image \"openebs/tests-custom-percona:latest\" already present on machine\r\n  Normal   Created      4m                kubelet, ip-10-0-1-87.us-west-1.compute.internal  Created container\r\n  Normal   Started      4m                kubelet, ip-10-0-1-87.us-west-1.compute.internal  Started container\r\n```\r\nLiveness pod logs:\r\n\r\n```\r\nkubectl logs -f percona-liveness-q968g-sll9v -n litmus percona-liveness\r\nDec 03 11:06:29 Waiting for mysql server to start accepting connections\r\nDec 03 11:06:29 MySQL server is ready to accept connections, starting Liveness check\r\nDec 03 11:06:29 MySQL db server is alive\r\nDec 03 11:06:39 MySQL db server is alive\r\nDec 03 11:06:49 MySQL db server is alive\r\nDec 03 11:06:59 MySQL db server is alive\r\nDec 03 11:07:09 MySQL db server is alive\r\nDec 03 11:07:19 MySQL db server is alive\r\nDec 03 11:07:29 MySQL db server is alive\r\nDec 03 11:07:39 MySQL db server is alive\r\nDec 03 11:07:49 MySQL db server is alive\r\nDec 03 11:08:01 Lost connection to db server, retrying after 10 sec\r\nDec 03 11:08:11 Lost connection to db server, retrying after 10 sec\r\nDec 03 11:08:21 Lost connection to db server, retrying after 10 sec\r\nDec 03 11:08:31 Lost connection to db server, retrying after 10 sec\r\nDec 03 11:08:41 Lost connection to db server, retrying after 10 sec\r\nDec 03 11:08:51 Lost connection to db server, retrying after 10 sec\r\nDec 03 11:09:01 Lost connection to db server, retrying after 10 sec\r\nDec 03 11:09:11 Lost connection to db server, retrying after 10 sec\r\nDec 03 11:09:21 Lost connection to db server, retrying after 10 sec\r\nDec 03 11:09:31 Lost connection to db server, retrying after 10 sec\r\nDec 03 11:09:41 Lost connection to db server, retrying after 10 sec\r\nDec 03 11:09:51 Lost connection to db server, retrying after 10 sec\r\nDec 03 11:10:00 Liveness check failed after 120s, exiting\r\n```\r\n## Screenshots\r\n<!-- Add screenshots of this bug -->\r\n\r\n## Your Environment\r\n<!-- Include as many relevant details about the environment you experienced the bug in -->\r\n* `kubectl get nodes`:\r\n```\r\nkubectl get nodes\r\nNAME                                       STATUS    ROLES     AGE       VERSION\r\nip-10-0-1-156.us-west-1.compute.internal   Ready     master    5h        v1.11.0\r\nip-10-0-1-212.us-west-1.compute.internal   Ready     node      5h        v1.11.0\r\nip-10-0-1-217.us-west-1.compute.internal   Ready     node      5h        v1.11.0\r\nip-10-0-1-87.us-west-1.compute.internal    Ready     node      5h        v1.11.0\r\n```\r\n* `kubectl get pods --all-namespaces`:\r\n* `kubectl get services`:\r\n* `kubectl get sc`:\r\n* `kubectl get pv`:\r\n* `kubectl get pvc`:\r\n* OS (from `/etc/os-release`):\r\n```\r\nubuntu@ip-10-0-1-156:~$ cat /etc/os-release\r\nNAME=\"Ubuntu\"\r\nVERSION=\"16.04.5 LTS (Xenial Xerus)\"\r\nID=ubuntu\r\nID_LIKE=debian\r\nPRETTY_NAME=\"Ubuntu 16.04.5 LTS\"\r\nVERSION_ID=\"16.04\"\r\nHOME_URL=\"http://www.ubuntu.com/\"\r\nSUPPORT_URL=\"http://help.ubuntu.com/\"\r\nBUG_REPORT_URL=\"http://bugs.launchpad.net/ubuntu/\"\r\nVERSION_CODENAME=xenial\r\nUBUNTU_CODENAME=xenial\r\n```\r\n\r\n* Kernel (from `uname -a`):\r\n* Install tools:\r\n* Others:\r\nOpenEBS version: ci"}}
{"actor":{"display_login":"dargasudarshan"}}