{"action":"created","issue":{"url":"https://api.github.com/repos/openebs/litmus/issues/435","repository_url":"https://api.github.com/repos/openebs/litmus","labels_url":"https://api.github.com/repos/openebs/litmus/issues/435/labels{/name}","comments_url":"https://api.github.com/repos/openebs/litmus/issues/435/comments","events_url":"https://api.github.com/repos/openebs/litmus/issues/435/events","html_url":"https://github.com/openebs/litmus/pull/435","id":388958258,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM3MDkwNjg5","number":435,"title":"Fast litmus","user":{"login":"vishnuitta","id":25415415,"node_id":"MDQ6VXNlcjI1NDE1NDE1","avatar_url":"https://avatars2.githubusercontent.com/u/25415415?v=4","gravatar_id":"","url":"https://api.github.com/users/vishnuitta","html_url":"https://github.com/vishnuitta","followers_url":"https://api.github.com/users/vishnuitta/followers","following_url":"https://api.github.com/users/vishnuitta/following{/other_user}","gists_url":"https://api.github.com/users/vishnuitta/gists{/gist_id}","starred_url":"https://api.github.com/users/vishnuitta/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishnuitta/subscriptions","organizations_url":"https://api.github.com/users/vishnuitta/orgs","repos_url":"https://api.github.com/users/vishnuitta/repos","events_url":"https://api.github.com/users/vishnuitta/events{/privacy}","received_events_url":"https://api.github.com/users/vishnuitta/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-12-08T20:51:55Z","updated_at":"2018-12-08T20:55:07Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/openebs/litmus/pulls/435","html_url":"https://github.com/openebs/litmus/pull/435","diff_url":"https://github.com/openebs/litmus/pull/435.diff","patch_url":"https://github.com/openebs/litmus/pull/435.patch"},"body":"This PR makes the pool kill/pool delete litmus jobs to introduce chaos faster.\r\n\r\nAs the pool kill/delete can happen only for cstor, below are the changes done:\r\n- find the chaoslib file by looking at chaos_type\r\n- do creation/updation of litmus CR with run_id environment variable\r\n- Removed the verification of pool provisioning\r\n- modified checks to find whether chaos induced or not\r\n- Avoided pumba pods deletion to save time during creation\r\n- Having faster checks to verify container restarts\r\n\r\nThis PR also have changes to log elapsed time for a task and total time at a task.\r\n\r\nSigned-off-by: Vishnu Itta <vitta@mayadata.io>"},"comment":{"url":"https://api.github.com/repos/openebs/litmus/issues/comments/445489201","html_url":"https://github.com/openebs/litmus/pull/435#issuecomment-445489201","issue_url":"https://api.github.com/repos/openebs/litmus/issues/435","id":445489201,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NTQ4OTIwMQ==","user":{"login":"vishnuitta","id":25415415,"node_id":"MDQ6VXNlcjI1NDE1NDE1","avatar_url":"https://avatars2.githubusercontent.com/u/25415415?v=4","gravatar_id":"","url":"https://api.github.com/users/vishnuitta","html_url":"https://github.com/vishnuitta","followers_url":"https://api.github.com/users/vishnuitta/followers","following_url":"https://api.github.com/users/vishnuitta/following{/other_user}","gists_url":"https://api.github.com/users/vishnuitta/gists{/gist_id}","starred_url":"https://api.github.com/users/vishnuitta/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vishnuitta/subscriptions","organizations_url":"https://api.github.com/users/vishnuitta/orgs","repos_url":"https://api.github.com/users/vishnuitta/repos","events_url":"https://api.github.com/users/vishnuitta/events{/privacy}","received_events_url":"https://api.github.com/users/vishnuitta/received_events","type":"User","site_admin":false},"created_at":"2018-12-08T20:55:07Z","updated_at":"2018-12-08T20:55:07Z","author_association":"MEMBER","body":"Below is ansible output:\r\n```# ansible-playbook ./percona/chaos/openebs_pool_failure/test.yml -i /etc/ansible/hosts -vv\r\nansible-playbook 2.7.4\r\n  config file = None\r\n  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']\r\n  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible\r\n  executable location = /usr/local/bin/ansible-playbook\r\n  python version = 2.7.15rc1 (default, Nov 12 2018, 14:31:15) [GCC 7.3.0]\r\nNo config file found; using defaults\r\n/etc/ansible/hosts did not meet host_list requirements, check plugin documentation if this is unexpected\r\n/etc/ansible/hosts did not meet script requirements, check plugin documentation if this is unexpected\r\n[DEPRECATION WARNING]: Specifying include variables at the top-level of the task is deprecated. Please see: \r\nhttps://docs.ansible.com/ansible/playbooks_roles.html#task-include-files-and-encouraging-reuse   for currently supported syntax regarding \r\nincluded files and variables. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting \r\ndeprecation_warnings=False in ansible.cfg.\r\n\r\nPLAYBOOK: test.yml ******************************************************************************************************************************\r\n1 plays in ./percona/chaos/openebs_pool_failure/test.yml\r\n\r\nPLAY [localhost] ********************************************************************************************************************************\r\n2018-12-08T20:30:51.012495 (delta: 0.213799)         elapsed: 0.213799 ******** \r\n=============================================================================== \r\n\r\nTASK [Gathering Facts] **************************************************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:2\r\n2018-12-08T20:30:51.097123 (delta: 0.084229)         elapsed: 0.298427 ******** \r\nok: [127.0.0.1]\r\nMETA: ran handlers\r\n\r\nTASK [Checking status of liveness pod] **********************************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:14\r\n2018-12-08T20:31:00.898062 (delta: 9.80056)         elapsed: 10.099366 ******** \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Get application pod name] *****************************************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:22\r\n2018-12-08T20:31:00.967198 (delta: 0.068461)         elapsed: 10.168502 ******* \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl get pods -n app-percona-ns -l name=percona --no-headers -o=custom-columns=NAME:\\\".metadata.name\\\"\", \"delta\": \"0:00:02.165499\", \"end\": \"2018-12-08 20:31:04.354379\", \"rc\": 0, \"start\": \"2018-12-08 20:31:02.188880\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"percona-6966c6db86-t7d5f\", \"stdout_lines\": [\"percona-6966c6db86-t7d5f\"]}\r\n\r\nTASK [Generate unique string for use in dbname] *************************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:30\r\n2018-12-08T20:31:04.404664 (delta: 3.437086)         elapsed: 13.605968 ******* \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"echo $(mktemp) | cut -d '.' -f 2\", \"delta\": \"0:00:01.470021\", \"end\": \"2018-12-08 20:31:06.053319\", \"rc\": 0, \"start\": \"2018-12-08 20:31:04.583298\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"FJHnkffCam\", \"stdout_lines\": [\"FJHnkffCam\"]}\r\n\r\nTASK [Create some test data in the mysql database] **********************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:36\r\n2018-12-08T20:31:06.101755 (delta: 1.696701)         elapsed: 15.303059 ******* \r\nincluded: /common/utils/mysql_data_persistence.yml for 127.0.0.1\r\n\r\nTASK [Create some test data in the mysql database] **********************************************************************************************\r\ntask path: /common/utils/mysql_data_persistence.yml:4\r\n2018-12-08T20:31:06.201388 (delta: 0.09925)         elapsed: 15.402692 ******** \r\nchanged: [127.0.0.1] => (item=mysql -uroot -pk8sDem0 -e 'create database tdbFJHnkffCam;') => {\"changed\": true, \"cmd\": \"kubectl exec percona-6966c6db86-t7d5f -n app-percona-ns -- mysql -uroot -pk8sDem0 -e 'create database tdbFJHnkffCam;'\", \"delta\": \"0:00:01.679177\", \"end\": \"2018-12-08 20:31:08.091337\", \"failed_when_result\": false, \"item\": \"mysql -uroot -pk8sDem0 -e 'create database tdbFJHnkffCam;'\", \"rc\": 0, \"start\": \"2018-12-08 20:31:06.412160\", \"stderr\": \"mysql: [Warning] Using a password on the command line interface can be insecure.\", \"stderr_lines\": [\"mysql: [Warning] Using a password on the command line interface can be insecure.\"], \"stdout\": \"\", \"stdout_lines\": []}\r\nchanged: [127.0.0.1] => (item=mysql -uroot -pk8sDem0 -e 'create table ttbl (Data VARCHAR(20));' tdbFJHnkffCam) => {\"changed\": true, \"cmd\": \"kubectl exec percona-6966c6db86-t7d5f -n app-percona-ns -- mysql -uroot -pk8sDem0 -e 'create table ttbl (Data VARCHAR(20));' tdbFJHnkffCam\", \"delta\": \"0:00:02.406508\", \"end\": \"2018-12-08 20:31:10.895916\", \"failed_when_result\": false, \"item\": \"mysql -uroot -pk8sDem0 -e 'create table ttbl (Data VARCHAR(20));' tdbFJHnkffCam\", \"rc\": 0, \"start\": \"2018-12-08 20:31:08.489408\", \"stderr\": \"mysql: [Warning] Using a password on the command line interface can be insecure.\", \"stderr_lines\": [\"mysql: [Warning] Using a password on the command line interface can be insecure.\"], \"stdout\": \"\", \"stdout_lines\": []}\r\nchanged: [127.0.0.1] => (item=mysql -uroot -pk8sDem0 -e 'insert into ttbl (Data) VALUES (\"tdata\");' tdbFJHnkffCam) => {\"changed\": true, \"cmd\": \"kubectl exec percona-6966c6db86-t7d5f -n app-percona-ns -- mysql -uroot -pk8sDem0 -e 'insert into ttbl (Data) VALUES (\\\"tdata\\\");' tdbFJHnkffCam\", \"delta\": \"0:00:02.939917\", \"end\": \"2018-12-08 20:31:14.452731\", \"failed_when_result\": false, \"item\": \"mysql -uroot -pk8sDem0 -e 'insert into ttbl (Data) VALUES (\\\"tdata\\\");' tdbFJHnkffCam\", \"rc\": 0, \"start\": \"2018-12-08 20:31:11.512814\", \"stderr\": \"mysql: [Warning] Using a password on the command line interface can be insecure.\", \"stderr_lines\": [\"mysql: [Warning] Using a password on the command line interface can be insecure.\"], \"stdout\": \"\", \"stdout_lines\": []}\r\n\r\nTASK [Checking for the Corrupted tables] ********************************************************************************************************\r\ntask path: /common/utils/mysql_data_persistence.yml:21\r\n2018-12-08T20:31:14.588200 (delta: 8.386384)         elapsed: 23.789504 ******* \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Verify mysql data persistence] ************************************************************************************************************\r\ntask path: /common/utils/mysql_data_persistence.yml:30\r\n2018-12-08T20:31:14.693697 (delta: 0.10456)         elapsed: 23.895001 ******** \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Delete/drop MySQL database] ***************************************************************************************************************\r\ntask path: /common/utils/mysql_data_persistence.yml:43\r\n2018-12-08T20:31:14.915643 (delta: 0.221417)         elapsed: 24.116947 ******* \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Verify successful db delete] **************************************************************************************************************\r\ntask path: /common/utils/mysql_data_persistence.yml:51\r\n2018-12-08T20:31:15.116649 (delta: 0.200936)         elapsed: 24.317953 ******* \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Record the chaos util path] ***************************************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:47\r\n2018-12-08T20:31:15.218926 (delta: 0.10164)         elapsed: 24.42023 ********* \r\nok: [127.0.0.1] => {\"ansible_facts\": {\"chaos_util_path\": \"/chaoslib/openebs/cstor_pool_kill.yml\"}, \"changed\": false}\r\n\r\nTASK [Record the chaos util path] ***************************************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:52\r\n2018-12-08T20:31:15.398235 (delta: 0.178916)         elapsed: 24.599539 ******* \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Record test instance/run ID] **************************************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:61\r\n2018-12-08T20:31:15.499873 (delta: 0.101263)         elapsed: 24.701177 ******* \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Construct testname appended with runID] ***************************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:65\r\n2018-12-08T20:31:15.607439 (delta: 0.107171)         elapsed: 24.808743 ******* \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [include_tasks] ****************************************************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:69\r\n2018-12-08T20:31:15.708488 (delta: 0.100648)         elapsed: 24.909792 ******* \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Display the app information passed via the test job] **************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:77\r\n2018-12-08T20:31:15.811089 (delta: 0.10221)         elapsed: 25.012393 ******** \r\nok: [127.0.0.1] => {\r\n    \"msg\": [\r\n        \"The application info is as follows:\", \r\n        \"Namespace    : app-percona-ns\", \r\n        \"Label        : name=percona\", \r\n        \"PVC          : percona-mysql-claim\"\r\n    ]\r\n}\r\n\r\nTASK [Verify that the AUT (Application Under Test) is running] **********************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:87\r\n2018-12-08T20:31:16.002554 (delta: 0.191082)         elapsed: 25.203858 ******* \r\nchanged: [127.0.0.1] => {\"attempts\": 1, \"changed\": true, \"cmd\": \"kubectl get pods -n app-percona-ns -l name=percona --no-headers -o custom-columns=:status.phase\", \"delta\": \"0:00:01.869813\", \"end\": \"2018-12-08 20:31:18.485269\", \"rc\": 0, \"start\": \"2018-12-08 20:31:16.615456\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"Running\", \"stdout_lines\": [\"Running\"]}\r\n\r\nTASK [include] **********************************************************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:100\r\n2018-12-08T20:31:18.794835 (delta: 2.791881)         elapsed: 27.996139 ******* \r\nincluded: /chaoslib/openebs/cstor_pool_kill.yml for 127.0.0.1\r\n\r\nTASK [include] **********************************************************************************************************************************\r\ntask path: /chaoslib/openebs/cstor_pool_kill.yml:1\r\n2018-12-08T20:31:19.104232 (delta: 0.309019)         elapsed: 28.305536 ******* \r\nincluded: /chaoslib/openebs/cstor_kill_and_verify_pool.yml for 127.0.0.1 => (item=0)\r\nincluded: /chaoslib/openebs/cstor_kill_and_verify_pool.yml for 127.0.0.1 => (item=1)\r\n\r\nTASK [Derive PV from application PVC] ***********************************************************************************************************\r\ntask path: /chaoslib/openebs/cstor_kill_and_verify_pool.yml:1\r\n2018-12-08T20:31:19.491145 (delta: 0.386443)         elapsed: 28.692449 ******* \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl get pvc percona-mysql-claim -o custom-columns=:spec.volumeName -n app-percona-ns --no-headers\", \"delta\": \"0:00:01.167161\", \"end\": \"2018-12-08 20:31:21.034799\", \"rc\": 0, \"start\": \"2018-12-08 20:31:19.867638\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"pvc-9a98ee27-f943-11e8-bdc1-42010a8001ea\", \"stdout_lines\": [\"pvc-9a98ee27-f943-11e8-bdc1-42010a8001ea\"]}\r\n\r\nTASK [Randomly select the pool deployment from cvr] *********************************************************************************************\r\ntask path: /chaoslib/openebs/cstor_kill_and_verify_pool.yml:10\r\n2018-12-08T20:31:21.103377 (delta: 1.61187)         elapsed: 30.304681 ******** \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl get cvr -n openebs -l openebs.io/persistent-volume=pvc-9a98ee27-f943-11e8-bdc1-42010a8001ea --no-headers -o=jsonpath='{range .items[*]}{.metadata.labels.cstorpool\\\\.openebs\\\\.io\\\\/name}{\\\"\\\\n\\\"}{end}' | shuf -n1 | awk '{print $1}'\", \"delta\": \"0:00:01.339138\", \"end\": \"2018-12-08 20:31:22.618589\", \"rc\": 0, \"start\": \"2018-12-08 20:31:21.279451\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"cstor-sparse-pool-huk4\", \"stdout_lines\": [\"cstor-sparse-pool-huk4\"]}\r\n\r\nTASK [Get the pod of pool deployment] ***********************************************************************************************************\r\ntask path: /chaoslib/openebs/cstor_kill_and_verify_pool.yml:20\r\n2018-12-08T20:31:22.778598 (delta: 1.674866)         elapsed: 31.979902 ******* \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl get pods -n openebs | grep cstor-sparse-pool-huk4 | grep -w \\\"Running\\\" | awk '{print $1}'\", \"delta\": \"0:00:01.294477\", \"end\": \"2018-12-08 20:31:24.695353\", \"rc\": 0, \"start\": \"2018-12-08 20:31:23.400876\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"cstor-sparse-pool-huk4-7878b6759b-55kvn\", \"stdout_lines\": [\"cstor-sparse-pool-huk4-7878b6759b-55kvn\"]}\r\n\r\nTASK [Get the runningStatus of pool pod] ********************************************************************************************************\r\ntask path: /chaoslib/openebs/cstor_kill_and_verify_pool.yml:28\r\n2018-12-08T20:31:24.797843 (delta: 2.018866)         elapsed: 33.999147 ******* \r\nchanged: [127.0.0.1] => {\"attempts\": 1, \"changed\": true, \"cmd\": \"kubectl get pod cstor-sparse-pool-huk4-7878b6759b-55kvn -n openebs -o=jsonpath='{range .status.containerStatuses[*]}{.state}{\\\"\\\\n\\\"}{end}' | grep -w running | wc -l\", \"delta\": \"0:00:02.367093\", \"end\": \"2018-12-08 20:31:27.785972\", \"rc\": 0, \"start\": \"2018-12-08 20:31:25.418879\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"2\", \"stdout_lines\": [\"2\"]}\r\n\r\nTASK [include_tasks] ****************************************************************************************************************************\r\ntask path: /chaoslib/openebs/cstor_kill_and_verify_pool.yml:40\r\n2018-12-08T20:31:27.896532 (delta: 3.098299)         elapsed: 37.097836 ******* \r\nincluded: /chaoslib/pumba/pod_failure_by_sigkill.yaml for 127.0.0.1\r\n\r\nTASK [Setup pumba chaos infrastructure] *********************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:2\r\n2018-12-08T20:31:28.239885 (delta: 0.34299)         elapsed: 37.441189 ******** \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl apply -f /chaoslib/pumba/pumba_kube.yaml -n openebs\", \"delta\": \"0:00:01.231852\", \"end\": \"2018-12-08 20:31:30.440302\", \"rc\": 0, \"start\": \"2018-12-08 20:31:29.208450\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"daemonset.extensions/pumba unchanged\", \"stdout_lines\": [\"daemonset.extensions/pumba unchanged\"]}\r\n\r\nTASK [Confirm that the pumba ds is running on all nodes] ****************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:10\r\n2018-12-08T20:31:30.556986 (delta: 2.316569)         elapsed: 39.75829 ******** \r\nchanged: [127.0.0.1] => {\"attempts\": 1, \"changed\": true, \"cmd\": \"kubectl get pod -l app=pumba --no-headers -o custom-columns=:status.phase -n openebs | sort | uniq\", \"delta\": \"0:00:01.345039\", \"end\": \"2018-12-08 20:31:32.331580\", \"rc\": 0, \"start\": \"2018-12-08 20:31:30.986541\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"Running\", \"stdout_lines\": [\"Running\"]}\r\n\r\nTASK [Identify the application node] ************************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:22\r\n2018-12-08T20:31:32.409860 (delta: 1.852433)         elapsed: 41.611164 ******* \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl get pod cstor-sparse-pool-huk4-7878b6759b-55kvn -n openebs --no-headers -o custom-columns=:spec.nodeName\", \"delta\": \"0:00:01.212520\", \"end\": \"2018-12-08 20:31:33.812205\", \"rc\": 0, \"start\": \"2018-12-08 20:31:32.599685\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"gke-vitta-default-pool-80367796-bmqr\", \"stdout_lines\": [\"gke-vitta-default-pool-80367796-bmqr\"]}\r\n\r\nTASK [Record the application node name] *********************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:30\r\n2018-12-08T20:31:34.013277 (delta: 1.603)         elapsed: 43.214581 ********** \r\nok: [127.0.0.1] => {\"ansible_facts\": {\"app_node\": \"gke-vitta-default-pool-80367796-bmqr\"}, \"changed\": false}\r\n\r\nTASK [Record the application container] *********************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:36\r\n2018-12-08T20:31:34.386116 (delta: 0.372415)         elapsed: 43.58742 ******** \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Record the app_container] *****************************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:43\r\n2018-12-08T20:31:34.489352 (delta: 0.102842)         elapsed: 43.690656 ******* \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Record the pumba pod on app node] *********************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:49\r\n2018-12-08T20:31:34.595685 (delta: 0.106279)         elapsed: 43.796989 ******* \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl get pod -l app=pumba -o wide -n openebs | grep gke-vitta-default-pool-80367796-bmqr | awk '{print $1}'\", \"delta\": \"0:00:01.047809\", \"end\": \"2018-12-08 20:31:35.922464\", \"rc\": 0, \"start\": \"2018-12-08 20:31:34.874655\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"pumba-j995b\", \"stdout_lines\": [\"pumba-j995b\"]}\r\n\r\nTASK [Record restartCount] **********************************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:58\r\n2018-12-08T20:31:36.093220 (delta: 1.497096)         elapsed: 45.294524 ******* \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl get pod cstor-sparse-pool-huk4-7878b6759b-55kvn -n openebs -o=jsonpath='{.status.containerStatuses[?(@.name==\\\"cstor-pool\\\")].restartCount}'\", \"delta\": \"0:00:01.541428\", \"end\": \"2018-12-08 20:31:37.941337\", \"rc\": 0, \"start\": \"2018-12-08 20:31:36.399909\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"5\", \"stdout_lines\": [\"5\"]}\r\n\r\nTASK [Force kill the application pod using pumba] ***********************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:66\r\n2018-12-08T20:31:37.992039 (delta: 1.898199)         elapsed: 47.193343 ******* \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl exec pumba-j995b -n openebs -- pumba kill --signal SIGKILL re2:k8s_cstor-pool_;\", \"delta\": \"0:00:02.042718\", \"end\": \"2018-12-08 20:31:40.279110\", \"rc\": 0, \"start\": \"2018-12-08 20:31:38.236392\", \"stderr\": \"time=\\\"2018-12-08T20:31:39Z\\\" level=info msg=\\\"Kill containers\\\" \\ntime=\\\"2018-12-08T20:31:40Z\\\" level=info msg=\\\"Killing /k8s_cstor-pool_cstor-sparse-pool-huk4-7878b6759b-55kvn_openebs_74b30b38-f942-11e8-bdc1-42010a8001ea_5 (9325a0e890618c661afc67ad2df12b49a573c31301d1fc46796f427cbd72b405) with signal SIGKILL\\\" \", \"stderr_lines\": [\"time=\\\"2018-12-08T20:31:39Z\\\" level=info msg=\\\"Kill containers\\\" \", \"time=\\\"2018-12-08T20:31:40Z\\\" level=info msg=\\\"Killing /k8s_cstor-pool_cstor-sparse-pool-huk4-7878b6759b-55kvn_openebs_74b30b38-f942-11e8-bdc1-42010a8001ea_5 (9325a0e890618c661afc67ad2df12b49a573c31301d1fc46796f427cbd72b405) with signal SIGKILL\\\" \"], \"stdout\": \"\", \"stdout_lines\": []}\r\n\r\nTASK [Verify restartCount] **********************************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:75\r\n2018-12-08T20:31:40.387620 (delta: 2.395204)         elapsed: 49.588924 ******* \r\nFAILED - RETRYING: Verify restartCount (30 retries left).\r\nchanged: [127.0.0.1] => {\"attempts\": 2, \"changed\": true, \"cmd\": \"kubectl get pod cstor-sparse-pool-huk4-7878b6759b-55kvn -n openebs -o=jsonpath='{.status.containerStatuses[?(@.name==\\\"cstor-pool\\\")].restartCount}'\", \"delta\": \"0:00:00.727692\", \"end\": \"2018-12-08 20:31:45.380554\", \"rc\": 0, \"start\": \"2018-12-08 20:31:44.652862\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"6\", \"stdout_lines\": [\"6\"]}\r\n\r\nTASK [Delete the pumba daemonset] ***************************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:89\r\n2018-12-08T20:31:45.501392 (delta: 5.113226)         elapsed: 54.702696 ******* \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Confirm that the pumba ds is deleted successfully] ****************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:96\r\n2018-12-08T20:31:45.602838 (delta: 0.100969)         elapsed: 54.804142 ******* \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Check for pool pod in running state] ******************************************************************************************************\r\ntask path: /chaoslib/openebs/cstor_kill_and_verify_pool.yml:47\r\n2018-12-08T20:31:45.777163 (delta: 0.173912)         elapsed: 54.978467 ******* \r\nchanged: [127.0.0.1] => {\"attempts\": 1, \"changed\": true, \"cmd\": \"kubectl get pod cstor-sparse-pool-huk4-7878b6759b-55kvn -n openebs | grep -w \\\"Running\\\" | wc -l\", \"delta\": \"0:00:01.048041\", \"end\": \"2018-12-08 20:31:47.564891\", \"rc\": 0, \"start\": \"2018-12-08 20:31:46.516850\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"1\", \"stdout_lines\": [\"1\"]}\r\n\r\nTASK [Get the runningStatus of pool pod] ********************************************************************************************************\r\ntask path: /chaoslib/openebs/cstor_kill_and_verify_pool.yml:58\r\n2018-12-08T20:31:47.635217 (delta: 1.857659)         elapsed: 56.836521 ******* \r\nchanged: [127.0.0.1] => {\"attempts\": 1, \"changed\": true, \"cmd\": \"kubectl get pod cstor-sparse-pool-huk4-7878b6759b-55kvn -n openebs -o=jsonpath='{range .status.containerStatuses[*]}{.state}{\\\"\\\\n\\\"}{end}' | grep -w running | wc -l\", \"delta\": \"0:00:01.402595\", \"end\": \"2018-12-08 20:31:49.700004\", \"rc\": 0, \"start\": \"2018-12-08 20:31:48.297409\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"2\", \"stdout_lines\": [\"2\"]}\r\n\r\nTASK [Derive PV from application PVC] ***********************************************************************************************************\r\ntask path: /chaoslib/openebs/cstor_kill_and_verify_pool.yml:1\r\n2018-12-08T20:31:49.909136 (delta: 2.273436)         elapsed: 59.11044 ******** \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl get pvc percona-mysql-claim -o custom-columns=:spec.volumeName -n app-percona-ns --no-headers\", \"delta\": \"0:00:01.471880\", \"end\": \"2018-12-08 20:31:51.869911\", \"rc\": 0, \"start\": \"2018-12-08 20:31:50.398031\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"pvc-9a98ee27-f943-11e8-bdc1-42010a8001ea\", \"stdout_lines\": [\"pvc-9a98ee27-f943-11e8-bdc1-42010a8001ea\"]}\r\n\r\nTASK [Randomly select the pool deployment from cvr] *********************************************************************************************\r\ntask path: /chaoslib/openebs/cstor_kill_and_verify_pool.yml:10\r\n2018-12-08T20:31:51.944224 (delta: 2.034693)         elapsed: 61.145528 ******* \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl get cvr -n openebs -l openebs.io/persistent-volume=pvc-9a98ee27-f943-11e8-bdc1-42010a8001ea --no-headers -o=jsonpath='{range .items[*]}{.metadata.labels.cstorpool\\\\.openebs\\\\.io\\\\/name}{\\\"\\\\n\\\"}{end}' | shuf -n1 | awk '{print $1}'\", \"delta\": \"0:00:01.580353\", \"end\": \"2018-12-08 20:31:53.899989\", \"rc\": 0, \"start\": \"2018-12-08 20:31:52.319636\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"cstor-sparse-pool-huk4\", \"stdout_lines\": [\"cstor-sparse-pool-huk4\"]}\r\n\r\nTASK [Get the pod of pool deployment] ***********************************************************************************************************\r\ntask path: /chaoslib/openebs/cstor_kill_and_verify_pool.yml:20\r\n2018-12-08T20:31:53.964322 (delta: 2.019624)         elapsed: 63.165626 ******* \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl get pods -n openebs | grep cstor-sparse-pool-huk4 | grep -w \\\"Running\\\" | awk '{print $1}'\", \"delta\": \"0:00:01.424088\", \"end\": \"2018-12-08 20:31:55.822270\", \"rc\": 0, \"start\": \"2018-12-08 20:31:54.398182\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"cstor-sparse-pool-huk4-7878b6759b-55kvn\", \"stdout_lines\": [\"cstor-sparse-pool-huk4-7878b6759b-55kvn\"]}\r\n\r\nTASK [Get the runningStatus of pool pod] ********************************************************************************************************\r\ntask path: /chaoslib/openebs/cstor_kill_and_verify_pool.yml:28\r\n2018-12-08T20:31:55.994519 (delta: 2.029877)         elapsed: 65.195823 ******* \r\nchanged: [127.0.0.1] => {\"attempts\": 1, \"changed\": true, \"cmd\": \"kubectl get pod cstor-sparse-pool-huk4-7878b6759b-55kvn -n openebs -o=jsonpath='{range .status.containerStatuses[*]}{.state}{\\\"\\\\n\\\"}{end}' | grep -w running | wc -l\", \"delta\": \"0:00:01.542819\", \"end\": \"2018-12-08 20:31:57.785212\", \"rc\": 0, \"start\": \"2018-12-08 20:31:56.242393\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"2\", \"stdout_lines\": [\"2\"]}\r\n\r\nTASK [include_tasks] ****************************************************************************************************************************\r\ntask path: /chaoslib/openebs/cstor_kill_and_verify_pool.yml:40\r\n2018-12-08T20:31:58.002140 (delta: 2.007234)         elapsed: 67.203444 ******* \r\nincluded: /chaoslib/pumba/pod_failure_by_sigkill.yaml for 127.0.0.1\r\n\r\nTASK [Setup pumba chaos infrastructure] *********************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:2\r\n2018-12-08T20:31:58.152009 (delta: 0.149498)         elapsed: 67.353313 ******* \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl apply -f /chaoslib/pumba/pumba_kube.yaml -n openebs\", \"delta\": \"0:00:01.633309\", \"end\": \"2018-12-08 20:32:00.072656\", \"rc\": 0, \"start\": \"2018-12-08 20:31:58.439347\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"daemonset.extensions/pumba unchanged\", \"stdout_lines\": [\"daemonset.extensions/pumba unchanged\"]}\r\n\r\nTASK [Confirm that the pumba ds is running on all nodes] ****************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:10\r\n2018-12-08T20:32:00.122358 (delta: 1.969751)         elapsed: 69.323662 ******* \r\nchanged: [127.0.0.1] => {\"attempts\": 1, \"changed\": true, \"cmd\": \"kubectl get pod -l app=pumba --no-headers -o custom-columns=:status.phase -n openebs | sort | uniq\", \"delta\": \"0:00:01.370883\", \"end\": \"2018-12-08 20:32:02.079509\", \"rc\": 0, \"start\": \"2018-12-08 20:32:00.708626\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"Running\", \"stdout_lines\": [\"Running\"]}\r\n\r\nTASK [Identify the application node] ************************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:22\r\n2018-12-08T20:32:02.186332 (delta: 2.063612)         elapsed: 71.387636 ******* \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl get pod cstor-sparse-pool-huk4-7878b6759b-55kvn -n openebs --no-headers -o custom-columns=:spec.nodeName\", \"delta\": \"0:00:01.393785\", \"end\": \"2018-12-08 20:32:03.866482\", \"rc\": 0, \"start\": \"2018-12-08 20:32:02.472697\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"gke-vitta-default-pool-80367796-bmqr\", \"stdout_lines\": [\"gke-vitta-default-pool-80367796-bmqr\"]}\r\n\r\nTASK [Record the application node name] *********************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:30\r\n2018-12-08T20:32:03.920453 (delta: 1.733746)         elapsed: 73.121757 ******* \r\nok: [127.0.0.1] => {\"ansible_facts\": {\"app_node\": \"gke-vitta-default-pool-80367796-bmqr\"}, \"changed\": false}\r\n\r\nTASK [Record the application container] *********************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:36\r\n2018-12-08T20:32:04.002573 (delta: 0.08129)         elapsed: 73.203877 ******** \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Record the app_container] *****************************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:43\r\n2018-12-08T20:32:04.106901 (delta: 0.103881)         elapsed: 73.308205 ******* \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Record the pumba pod on app node] *********************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:49\r\n2018-12-08T20:32:04.218235 (delta: 0.110905)         elapsed: 73.419539 ******* \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl get pod -l app=pumba -o wide -n openebs | grep gke-vitta-default-pool-80367796-bmqr | awk '{print $1}'\", \"delta\": \"0:00:01.519857\", \"end\": \"2018-12-08 20:32:06.217338\", \"rc\": 0, \"start\": \"2018-12-08 20:32:04.697481\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"pumba-j995b\", \"stdout_lines\": [\"pumba-j995b\"]}\r\n\r\nTASK [Record restartCount] **********************************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:58\r\n2018-12-08T20:32:06.381473 (delta: 2.162856)         elapsed: 75.582777 ******* \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl get pod cstor-sparse-pool-huk4-7878b6759b-55kvn -n openebs -o=jsonpath='{.status.containerStatuses[?(@.name==\\\"cstor-pool\\\")].restartCount}'\", \"delta\": \"0:00:01.476814\", \"end\": \"2018-12-08 20:32:08.579721\", \"rc\": 0, \"start\": \"2018-12-08 20:32:07.102907\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"6\", \"stdout_lines\": [\"6\"]}\r\n\r\nTASK [Force kill the application pod using pumba] ***********************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:66\r\n2018-12-08T20:32:08.810364 (delta: 2.428363)         elapsed: 78.011668 ******* \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl exec pumba-j995b -n openebs -- pumba kill --signal SIGKILL re2:k8s_cstor-pool_;\", \"delta\": \"0:00:02.160029\", \"end\": \"2018-12-08 20:32:11.682371\", \"rc\": 0, \"start\": \"2018-12-08 20:32:09.522342\", \"stderr\": \"time=\\\"2018-12-08T20:32:11Z\\\" level=info msg=\\\"Kill containers\\\" \\ntime=\\\"2018-12-08T20:32:11Z\\\" level=info msg=\\\"Killing /k8s_cstor-pool_cstor-sparse-pool-huk4-7878b6759b-55kvn_openebs_74b30b38-f942-11e8-bdc1-42010a8001ea_6 (b91458fd667a0b513b3b8785769a9b62da1445303aa947197826e235dc773e3d) with signal SIGKILL\\\" \", \"stderr_lines\": [\"time=\\\"2018-12-08T20:32:11Z\\\" level=info msg=\\\"Kill containers\\\" \", \"time=\\\"2018-12-08T20:32:11Z\\\" level=info msg=\\\"Killing /k8s_cstor-pool_cstor-sparse-pool-huk4-7878b6759b-55kvn_openebs_74b30b38-f942-11e8-bdc1-42010a8001ea_6 (b91458fd667a0b513b3b8785769a9b62da1445303aa947197826e235dc773e3d) with signal SIGKILL\\\" \"], \"stdout\": \"\", \"stdout_lines\": []}\r\n\r\nTASK [Verify restartCount] **********************************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:75\r\n2018-12-08T20:32:11.789455 (delta: 2.978284)         elapsed: 80.990759 ******* \r\nFAILED - RETRYING: Verify restartCount (30 retries left).\r\nFAILED - RETRYING: Verify restartCount (29 retries left).\r\nFAILED - RETRYING: Verify restartCount (28 retries left).\r\nFAILED - RETRYING: Verify restartCount (27 retries left).\r\nFAILED - RETRYING: Verify restartCount (26 retries left).\r\nchanged: [127.0.0.1] => {\"attempts\": 6, \"changed\": true, \"cmd\": \"kubectl get pod cstor-sparse-pool-huk4-7878b6759b-55kvn -n openebs -o=jsonpath='{.status.containerStatuses[?(@.name==\\\"cstor-pool\\\")].restartCount}'\", \"delta\": \"0:00:00.875563\", \"end\": \"2018-12-08 20:32:32.697062\", \"rc\": 0, \"start\": \"2018-12-08 20:32:31.821499\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"7\", \"stdout_lines\": [\"7\"]}\r\n\r\nTASK [Delete the pumba daemonset] ***************************************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:89\r\n2018-12-08T20:32:32.803641 (delta: 21.014129)         elapsed: 102.004945 ***** \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Confirm that the pumba ds is deleted successfully] ****************************************************************************************\r\ntask path: /chaoslib/pumba/pod_failure_by_sigkill.yaml:96\r\n2018-12-08T20:32:32.857530 (delta: 0.053513)         elapsed: 102.058834 ****** \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Check for pool pod in running state] ******************************************************************************************************\r\ntask path: /chaoslib/openebs/cstor_kill_and_verify_pool.yml:47\r\n2018-12-08T20:32:32.907599 (delta: 0.049703)         elapsed: 102.108903 ****** \r\nchanged: [127.0.0.1] => {\"attempts\": 1, \"changed\": true, \"cmd\": \"kubectl get pod cstor-sparse-pool-huk4-7878b6759b-55kvn -n openebs | grep -w \\\"Running\\\" | wc -l\", \"delta\": \"0:00:01.504150\", \"end\": \"2018-12-08 20:32:34.619336\", \"rc\": 0, \"start\": \"2018-12-08 20:32:33.115186\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"1\", \"stdout_lines\": [\"1\"]}\r\n\r\nTASK [Get the runningStatus of pool pod] ********************************************************************************************************\r\ntask path: /chaoslib/openebs/cstor_kill_and_verify_pool.yml:58\r\n2018-12-08T20:32:34.782550 (delta: 1.874578)         elapsed: 103.983854 ****** \r\nchanged: [127.0.0.1] => {\"attempts\": 1, \"changed\": true, \"cmd\": \"kubectl get pod cstor-sparse-pool-huk4-7878b6759b-55kvn -n openebs -o=jsonpath='{range .status.containerStatuses[*]}{.state}{\\\"\\\\n\\\"}{end}' | grep -w running | wc -l\", \"delta\": \"0:00:01.026499\", \"end\": \"2018-12-08 20:32:36.179171\", \"rc\": 0, \"start\": \"2018-12-08 20:32:35.152672\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"2\", \"stdout_lines\": [\"2\"]}\r\n\r\nTASK [Verify AUT liveness post fault-injection] *************************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:109\r\n2018-12-08T20:32:36.291225 (delta: 1.50826)         elapsed: 105.492529 ******* \r\nchanged: [127.0.0.1] => {\"attempts\": 1, \"changed\": true, \"cmd\": \"kubectl get pods -n app-percona-ns -l name=percona --no-headers -o custom-columns=:status.phase\", \"delta\": \"0:00:01.292884\", \"end\": \"2018-12-08 20:32:37.974554\", \"rc\": 0, \"start\": \"2018-12-08 20:32:36.681670\", \"stderr\": \"\", \"stderr_lines\": [], \"stdout\": \"Running\", \"stdout_lines\": [\"Running\"]}\r\n\r\nTASK [Verify mysql data persistence] ************************************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:120\r\n2018-12-08T20:32:38.033713 (delta: 1.742117)         elapsed: 107.235017 ****** \r\nincluded: /common/utils/mysql_data_persistence.yml for 127.0.0.1\r\n\r\nTASK [Create some test data in the mysql database] **********************************************************************************************\r\ntask path: /common/utils/mysql_data_persistence.yml:4\r\n2018-12-08T20:32:38.130186 (delta: 0.096125)         elapsed: 107.33149 ******* \r\nskipping: [127.0.0.1] => (item=mysql -uroot -pk8sDem0 -e 'create database tdbFJHnkffCam;')  => {\"changed\": false, \"item\": \"mysql -uroot -pk8sDem0 -e 'create database tdbFJHnkffCam;'\", \"skip_reason\": \"Conditional result was False\"}\r\nskipping: [127.0.0.1] => (item=mysql -uroot -pk8sDem0 -e 'create table ttbl (Data VARCHAR(20));' tdbFJHnkffCam)  => {\"changed\": false, \"item\": \"mysql -uroot -pk8sDem0 -e 'create table ttbl (Data VARCHAR(20));' tdbFJHnkffCam\", \"skip_reason\": \"Conditional result was False\"}\r\nskipping: [127.0.0.1] => (item=mysql -uroot -pk8sDem0 -e 'insert into ttbl (Data) VALUES (\"tdata\");' tdbFJHnkffCam)  => {\"changed\": false, \"item\": \"mysql -uroot -pk8sDem0 -e 'insert into ttbl (Data) VALUES (\\\"tdata\\\");' tdbFJHnkffCam\", \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Checking for the Corrupted tables] ********************************************************************************************************\r\ntask path: /common/utils/mysql_data_persistence.yml:21\r\n2018-12-08T20:32:38.210848 (delta: 0.080163)         elapsed: 107.412152 ****** \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl exec percona-6966c6db86-t7d5f -n app-percona-ns -- mysqlcheck -c tdbFJHnkffCam -uroot -pk8sDem0\", \"delta\": \"0:00:01.774993\", \"end\": \"2018-12-08 20:32:40.353707\", \"failed_when_result\": false, \"rc\": 0, \"start\": \"2018-12-08 20:32:38.578714\", \"stderr\": \"mysqlcheck: [Warning] Using a password on the command line interface can be insecure.\", \"stderr_lines\": [\"mysqlcheck: [Warning] Using a password on the command line interface can be insecure.\"], \"stdout\": \"tdbFJHnkffCam.ttbl                                 OK\", \"stdout_lines\": [\"tdbFJHnkffCam.ttbl                                 OK\"]}\r\n\r\nTASK [Verify mysql data persistence] ************************************************************************************************************\r\ntask path: /common/utils/mysql_data_persistence.yml:30\r\n2018-12-08T20:32:40.415431 (delta: 2.204175)         elapsed: 109.616735 ****** \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl exec percona-6966c6db86-t7d5f -n app-percona-ns -- mysql -uroot -pk8sDem0 -e 'select * from ttbl' tdbFJHnkffCam;\", \"delta\": \"0:00:01.748645\", \"end\": \"2018-12-08 20:32:42.373077\", \"failed_when_result\": false, \"rc\": 0, \"start\": \"2018-12-08 20:32:40.624432\", \"stderr\": \"mysql: [Warning] Using a password on the command line interface can be insecure.\", \"stderr_lines\": [\"mysql: [Warning] Using a password on the command line interface can be insecure.\"], \"stdout\": \"Data\\ntdata\", \"stdout_lines\": [\"Data\", \"tdata\"]}\r\n\r\nTASK [Delete/drop MySQL database] ***************************************************************************************************************\r\ntask path: /common/utils/mysql_data_persistence.yml:43\r\n2018-12-08T20:32:42.437060 (delta: 2.020918)         elapsed: 111.638364 ****** \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Verify successful db delete] **************************************************************************************************************\r\ntask path: /common/utils/mysql_data_persistence.yml:51\r\n2018-12-08T20:32:42.491320 (delta: 0.053813)         elapsed: 111.692624 ****** \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Verify successful database delete] ********************************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:131\r\n2018-12-08T20:32:42.541711 (delta: 0.049912)         elapsed: 111.743015 ****** \r\nincluded: /common/utils/mysql_data_persistence.yml for 127.0.0.1\r\n\r\nTASK [Create some test data in the mysql database] **********************************************************************************************\r\ntask path: /common/utils/mysql_data_persistence.yml:4\r\n2018-12-08T20:32:42.658706 (delta: 0.116632)         elapsed: 111.86001 ******* \r\nskipping: [127.0.0.1] => (item=mysql -uroot -pk8sDem0 -e 'create database tdbFJHnkffCam;')  => {\"changed\": false, \"item\": \"mysql -uroot -pk8sDem0 -e 'create database tdbFJHnkffCam;'\", \"skip_reason\": \"Conditional result was False\"}\r\nskipping: [127.0.0.1] => (item=mysql -uroot -pk8sDem0 -e 'create table ttbl (Data VARCHAR(20));' tdbFJHnkffCam)  => {\"changed\": false, \"item\": \"mysql -uroot -pk8sDem0 -e 'create table ttbl (Data VARCHAR(20));' tdbFJHnkffCam\", \"skip_reason\": \"Conditional result was False\"}\r\nskipping: [127.0.0.1] => (item=mysql -uroot -pk8sDem0 -e 'insert into ttbl (Data) VALUES (\"tdata\");' tdbFJHnkffCam)  => {\"changed\": false, \"item\": \"mysql -uroot -pk8sDem0 -e 'insert into ttbl (Data) VALUES (\\\"tdata\\\");' tdbFJHnkffCam\", \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Checking for the Corrupted tables] ********************************************************************************************************\r\ntask path: /common/utils/mysql_data_persistence.yml:21\r\n2018-12-08T20:32:42.782583 (delta: 0.123387)         elapsed: 111.983887 ****** \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Verify mysql data persistence] ************************************************************************************************************\r\ntask path: /common/utils/mysql_data_persistence.yml:30\r\n2018-12-08T20:32:42.992870 (delta: 0.209829)         elapsed: 112.194174 ****** \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [Delete/drop MySQL database] ***************************************************************************************************************\r\ntask path: /common/utils/mysql_data_persistence.yml:43\r\n2018-12-08T20:32:43.220022 (delta: 0.226637)         elapsed: 112.421326 ****** \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl exec percona-6966c6db86-t7d5f -n app-percona-ns -- mysql -uroot -pk8sDem0 -e 'drop database tdbFJHnkffCam';\", \"delta\": \"0:00:02.665554\", \"end\": \"2018-12-08 20:32:46.987457\", \"rc\": 0, \"start\": \"2018-12-08 20:32:44.321903\", \"stderr\": \"mysql: [Warning] Using a password on the command line interface can be insecure.\", \"stderr_lines\": [\"mysql: [Warning] Using a password on the command line interface can be insecure.\"], \"stdout\": \"\", \"stdout_lines\": []}\r\n\r\nTASK [Verify successful db delete] **************************************************************************************************************\r\ntask path: /common/utils/mysql_data_persistence.yml:51\r\n2018-12-08T20:32:47.289195 (delta: 4.068704)         elapsed: 116.490499 ****** \r\nchanged: [127.0.0.1] => {\"changed\": true, \"cmd\": \"kubectl exec percona-6966c6db86-t7d5f -n app-percona-ns -- mysql -uroot -pk8sDem0 -e 'show databases';\", \"delta\": \"0:00:01.594190\", \"end\": \"2018-12-08 20:32:49.507608\", \"failed_when_result\": false, \"rc\": 0, \"start\": \"2018-12-08 20:32:47.913418\", \"stderr\": \"mysql: [Warning] Using a password on the command line interface can be insecure.\", \"stderr_lines\": [\"mysql: [Warning] Using a password on the command line interface can be insecure.\"], \"stdout\": \"Database\\ninformation_schema\\nInventory_bZCGK1Mn8g\\nmysql\\nperformance_schema\\nsys\\ntdb4BIyK8bsou\\ntdbFuodfkKvs7\\ntdbHWlDKoQNQC\\ntdbPxKWLA8KmE\\ntdbUZFJsTP966\\ntdbef5ScGhgO4\\ntdbeyrxPXw42l\\ntdbfRlNuqdzjY\\ntdbx9QWGjGjd0\\ntdbxOTfZJhFlS\", \"stdout_lines\": [\"Database\", \"information_schema\", \"Inventory_bZCGK1Mn8g\", \"mysql\", \"performance_schema\", \"sys\", \"tdb4BIyK8bsou\", \"tdbFuodfkKvs7\", \"tdbHWlDKoQNQC\", \"tdbPxKWLA8KmE\", \"tdbUZFJsTP966\", \"tdbef5ScGhgO4\", \"tdbeyrxPXw42l\", \"tdbfRlNuqdzjY\", \"tdbx9QWGjGjd0\", \"tdbxOTfZJhFlS\"]}\r\n\r\nTASK [Checking status of liveness pod] **********************************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:144\r\n2018-12-08T20:32:49.690043 (delta: 2.400397)         elapsed: 118.891347 ****** \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\n\r\nTASK [set_fact] *********************************************************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:153\r\n2018-12-08T20:32:49.803024 (delta: 0.112505)         elapsed: 119.004328 ****** \r\nok: [127.0.0.1] => {\"ansible_facts\": {\"flag\": \"Pass\"}, \"changed\": false}\r\n\r\nTASK [include_tasks] ****************************************************************************************************************************\r\ntask path: /litmus/apps/percona/chaos/openebs_pool_failure/test.yml:164\r\n2018-12-08T20:32:49.984712 (delta: 0.181195)         elapsed: 119.186016 ****** \r\nskipping: [127.0.0.1] => {\"changed\": false, \"skip_reason\": \"Conditional result was False\"}\r\nMETA: ran handlers\r\nMETA: ran handlers\r\n\r\nPLAY RECAP **************************************************************************************************************************************\r\n127.0.0.1                  : ok=49   changed=35   unreachable=0    failed=0   \r\n\r\n2018-12-08T20:32:50.088176 (delta: 0.102947)         elapsed: 119.28948 ******* \r\n=============================================================================== \r\n```"}}
{"actor":{"display_login":"vishnuitta"}}