{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/20543","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/20543/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/20543/comments","events_url":"https://api.github.com/repos/docker/docker/issues/20543/events","html_url":"https://github.com/docker/docker/issues/20543","id":135068309,"number":20543,"title":"Terminated/Killed integration-cli tests","user":{"login":"runcom","id":849915,"avatar_url":"https://avatars.githubusercontent.com/u/849915?v=3","gravatar_id":"","url":"https://api.github.com/users/runcom","html_url":"https://github.com/runcom","followers_url":"https://api.github.com/users/runcom/followers","following_url":"https://api.github.com/users/runcom/following{/other_user}","gists_url":"https://api.github.com/users/runcom/gists{/gist_id}","starred_url":"https://api.github.com/users/runcom/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/runcom/subscriptions","organizations_url":"https://api.github.com/users/runcom/orgs","repos_url":"https://api.github.com/users/runcom/repos","events_url":"https://api.github.com/users/runcom/events{/privacy}","received_events_url":"https://api.github.com/users/runcom/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/docker/docker/labels/kind/flakytest","name":"kind/flakytest","color":"EF7409"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-02-20T12:45:14Z","updated_at":"2016-02-26T20:22:51Z","closed_at":null,"body":"On latest master code, as already outlined here https://github.com/docker/docker/pull/20509#issuecomment-186560841, integration-cli tests are Killed/Terminated. I cannot see anything weird in test.log nor any other log files. I'm experiencing this on my own laptop but PRs are also affected (see linked PR before). The tests usually fails at `TestDaemonLoggingDriverNone` but sometimes also on `Volumes*` tests.\r\n\r\n```\r\nPASS: docker_cli_daemon_test.go:575: DockerDaemonSuite.TestDaemonKeyGeneration\t0.528s\r\nPASS: docker_cli_daemon_test.go:594: DockerDaemonSuite.TestDaemonKeyMigration\t0.524s\r\nPASS: docker_cli_daemon_test.go:1045: DockerDaemonSuite.TestDaemonLinksIpTablesRulesWhenLinkAndUnlink\t2.156s\r\nPASS: docker_cli_daemon_test.go:498: DockerDaemonSuite.TestDaemonLogLevelDebug\t0.518s\r\nPASS: docker_cli_daemon_test.go:508: DockerDaemonSuite.TestDaemonLogLevelFatal\t0.516s\r\nPASS: docker_cli_daemon_test.go:431: DockerDaemonSuite.TestDaemonLogLevelWrong\t5.501s\r\nPASS: docker_cli_daemon_test.go:1154: DockerDaemonSuite.TestDaemonLoggingDriverDefault\t1.066s\r\nPASS: docker_cli_daemon_test.go:1196: DockerDaemonSuite.TestDaemonLoggingDriverDefaultOverride\t1.063s\r\nPASS: docker_cli_daemon_test.go:1217: DockerDaemonSuite.TestDaemonLoggingDriverNone\t1.042s\r\n/go/src/github.com/docker/docker/hack/make/test-integration-cli: line 18:  2829 Terminated              ( bundle .integration-daemon-start; bundle .integration-daemon-setup; bundle_test_integration_cli; bundle .integration-daemon-stop ) 2>&1\r\n      2830                       | tee -a \"$DEST/test.log\"\r\n```"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/189467480","html_url":"https://github.com/docker/docker/issues/20543#issuecomment-189467480","issue_url":"https://api.github.com/repos/docker/docker/issues/20543","id":189467480,"user":{"login":"tophj-ibm","id":15270681,"avatar_url":"https://avatars.githubusercontent.com/u/15270681?v=3","gravatar_id":"","url":"https://api.github.com/users/tophj-ibm","html_url":"https://github.com/tophj-ibm","followers_url":"https://api.github.com/users/tophj-ibm/followers","following_url":"https://api.github.com/users/tophj-ibm/following{/other_user}","gists_url":"https://api.github.com/users/tophj-ibm/gists{/gist_id}","starred_url":"https://api.github.com/users/tophj-ibm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tophj-ibm/subscriptions","organizations_url":"https://api.github.com/users/tophj-ibm/orgs","repos_url":"https://api.github.com/users/tophj-ibm/repos","events_url":"https://api.github.com/users/tophj-ibm/events{/privacy}","received_events_url":"https://api.github.com/users/tophj-ibm/received_events","type":"User","site_admin":false},"created_at":"2016-02-26T20:22:51Z","updated_at":"2016-02-26T20:22:51Z","body":"@tiborvass @icecrime @estesp now that other people are looking at this, I'll update what I've found so far\r\n\r\nIt looks like 0.8 libcontainer vendor was the breaking commit (929f62e64d8760033ff6db4607c21a6431c46e4d)\r\n\r\nI've seen it happen on three different tests, they happen pretty frequently on power probably due to it being *slightly* slower:\r\n1.) DockerSuite.TestVolumesApiRemove\r\n2.) DockerDaemonSuite.TestDaemonLoggingDriverNoneLogs\r\n3.) DockerSuite.TestAttachClosedOnContainerStop\r\n\r\nHere are some gists of logs I've been collecting\r\n\r\ndocker.log with goroutine stacktraces for when TestVolumesApiRemove fails\r\nhttps://gist.github.com/tophj-ibm/354cf01dd6181dc9a25c#file-failing-test-log\r\n\r\ndocker.log with goroutine stacktraces for when TestVolumesApiRemove passes\r\nhttps://gist.github.com/tophj-ibm/4633e3ae3e8affbccdf7#file-gistfile1-txt\r\n\r\nIt looks like there is a libcontainer goroutine that is waiting on a process that is still running when it passes, and isn't there when it fails. Not sure if it's related but that's what I'm going to look into."}}