{"action":"created","issue":{"url":"https://api.github.com/repos/Terracotta-OSS/galvan/issues/46","repository_url":"https://api.github.com/repos/Terracotta-OSS/galvan","labels_url":"https://api.github.com/repos/Terracotta-OSS/galvan/issues/46/labels{/name}","comments_url":"https://api.github.com/repos/Terracotta-OSS/galvan/issues/46/comments","events_url":"https://api.github.com/repos/Terracotta-OSS/galvan/issues/46/events","html_url":"https://github.com/Terracotta-OSS/galvan/issues/46","id":162815955,"number":46,"title":"BasicExternalServer.after hangs waiting for ps output","user":{"login":"cljohnso","id":8737388,"avatar_url":"https://avatars.githubusercontent.com/u/8737388?v=3","gravatar_id":"","url":"https://api.github.com/users/cljohnso","html_url":"https://github.com/cljohnso","followers_url":"https://api.github.com/users/cljohnso/followers","following_url":"https://api.github.com/users/cljohnso/following{/other_user}","gists_url":"https://api.github.com/users/cljohnso/gists{/gist_id}","starred_url":"https://api.github.com/users/cljohnso/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cljohnso/subscriptions","organizations_url":"https://api.github.com/users/cljohnso/orgs","repos_url":"https://api.github.com/users/cljohnso/repos","events_url":"https://api.github.com/users/cljohnso/events{/privacy}","received_events_url":"https://api.github.com/users/cljohnso/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Terracotta-OSS/galvan/labels/bug","name":"bug","color":"ee0701"}],"state":"open","locked":false,"assignee":{"login":"jd0-sag","id":16048428,"avatar_url":"https://avatars.githubusercontent.com/u/16048428?v=3","gravatar_id":"","url":"https://api.github.com/users/jd0-sag","html_url":"https://github.com/jd0-sag","followers_url":"https://api.github.com/users/jd0-sag/followers","following_url":"https://api.github.com/users/jd0-sag/following{/other_user}","gists_url":"https://api.github.com/users/jd0-sag/gists{/gist_id}","starred_url":"https://api.github.com/users/jd0-sag/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jd0-sag/subscriptions","organizations_url":"https://api.github.com/users/jd0-sag/orgs","repos_url":"https://api.github.com/users/jd0-sag/repos","events_url":"https://api.github.com/users/jd0-sag/events{/privacy}","received_events_url":"https://api.github.com/users/jd0-sag/received_events","type":"User","site_admin":false},"assignees":[{"login":"jd0-sag","id":16048428,"avatar_url":"https://avatars.githubusercontent.com/u/16048428?v=3","gravatar_id":"","url":"https://api.github.com/users/jd0-sag","html_url":"https://github.com/jd0-sag","followers_url":"https://api.github.com/users/jd0-sag/followers","following_url":"https://api.github.com/users/jd0-sag/following{/other_user}","gists_url":"https://api.github.com/users/jd0-sag/gists{/gist_id}","starred_url":"https://api.github.com/users/jd0-sag/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jd0-sag/subscriptions","organizations_url":"https://api.github.com/users/jd0-sag/orgs","repos_url":"https://api.github.com/users/jd0-sag/repos","events_url":"https://api.github.com/users/jd0-sag/events{/privacy}","received_events_url":"https://api.github.com/users/jd0-sag/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2016-06-29T00:21:45Z","updated_at":"2016-06-29T02:58:19Z","closed_at":null,"body":"The `BasicExternalCluster.after` stalls waiting for completion of the \"Shepherding Thread\".  The \"Shepherding Thread\" is stalled waiting on completion of the `ps -eww -o pid,command` in `killProcessUnix`.  Completion of the `ps` process is waiting on consumption of it's output.  The stall is dependent on the amount of output produced from the `ps` command.\r\n\r\nFor a `Process` to complete, enough of its output (STDERR or STDOUT) must be consumed to allow the underlying UNIX process to complete -- otherwise it is blocked on writing output to its \"full\" buffers.\r\n\r\nThe `killProcessUnix' method needs to consume effectively all of the `ps` output **before** calling `Process.waitFor` to prevent a hang.  (In this case STDERR is not the culprit but very well could have been ... consider using `ProcessBuilder` and `redirectErrorStream(boolean)` to avoid a STDERR-cause stall.)\r\n\r\nThis fault is presently preventing Ehcache3 builds from completing.  \r\n\r\nThe attached [patch](https://github.com/Terracotta-OSS/galvan/files/338295/ps_stall.txt) is a hack that works locally.\r\n\r\n----\r\n\r\n```text\r\n\"Test worker\" #11 prio=5 os_prio=31 tid=0x00007ffcdbcc9000 nid=0x5503 in Object.wait() [0x000000011f95e000]\r\n   java.lang.Thread.State: WAITING (on object monitor)\r\n\tat java.lang.Object.wait(Native Method)\r\n\tat java.lang.Thread.join(Thread.java:1245)\r\n\t- locked <0x00000007a002ec58> (a org.terracotta.testing.rules.BasicExternalCluster$2)\r\n\tat java.lang.Thread.join(Thread.java:1319)\r\n\tat org.terracotta.testing.rules.BasicExternalCluster.after(BasicExternalCluster.java:155)\r\n\tat org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)\r\n\tat org.junit.rules.RunRules.evaluate(RunRules.java:20)\r\n\tat org.junit.runners.ParentRunner.run(ParentRunner.java:363)\r\n\tat org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)\r\n\tat org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)\r\n\tat org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)\r\n\tat org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:497)\r\n\tat org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)\r\n\tat org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)\r\n\tat org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)\r\n\tat org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)\r\n\tat com.sun.proxy.$Proxy2.processTestClass(Unknown Source)\r\n\tat org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r\n\tat java.lang.reflect.Method.invoke(Method.java:497)\r\n\tat org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)\r\n\tat org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)\r\n\tat org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:377)\r\n\tat org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)\r\n\tat org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:745)\r\n   Locked ownable synchronizers:\t- <0x00000007a0032f20> (a java.util.concurrent.ThreadPoolExecutor$Worker)\r\n```\r\n\r\n```text\r\n\"Shepherding Thread\" #21 prio=5 os_prio=31 tid=0x00007ffcdcdba800 nid=0x6503 in Object.wait() [0x000000011f737000]\r\n   java.lang.Thread.State: WAITING (on object monitor)\r\n\tat java.lang.Object.wait(Native Method)\r\n\tat java.lang.Object.wait(Object.java:502)\r\n\tat java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)\r\n\t- locked <0x00000007be4fa9c0> (a java.lang.UNIXProcess)\r\n\tat org.terracotta.testing.master.ServerProcess$ExitWaiter.killProcessUnix(ServerProcess.java:278)\r\n\tat org.terracotta.testing.master.ServerProcess$ExitWaiter.bringDownServer(ServerProcess.java:250)\r\n\t- locked <0x00000007a002f1c8> (a org.terracotta.testing.master.ServerProcess$ExitWaiter)\r\n\tat org.terracotta.testing.master.ServerProcess.stop(ServerProcess.java:160)\r\n\tat org.terracotta.testing.master.SynchronousProcessControl.shutdownServer(SynchronousProcessControl.java:209)\r\n\tat org.terracotta.testing.master.SynchronousProcessControl.shutDown(SynchronousProcessControl.java:154)\r\n\t- locked <0x00000007a003d270> (a org.terracotta.testing.master.SynchronousProcessControl)\r\n\tat org.terracotta.testing.rules.BasicExternalCluster$1.forceTerminateComponent(BasicExternalCluster.java:111)\r\n\tat org.terracotta.testing.master.TestStateManager.waitForFinish(TestStateManager.java:45)\r\n\t- locked <0x00000007a002edd8> (a org.terracotta.testing.master.TestStateManager)\r\n\tat org.terracotta.testing.rules.BasicExternalCluster$2.run(BasicExternalCluster.java:130)\r\n   Locked ownable synchronizers:\t- None\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/Terracotta-OSS/galvan/issues/comments/229243716","html_url":"https://github.com/Terracotta-OSS/galvan/issues/46#issuecomment-229243716","issue_url":"https://api.github.com/repos/Terracotta-OSS/galvan/issues/46","id":229243716,"user":{"login":"jd0-sag","id":16048428,"avatar_url":"https://avatars.githubusercontent.com/u/16048428?v=3","gravatar_id":"","url":"https://api.github.com/users/jd0-sag","html_url":"https://github.com/jd0-sag","followers_url":"https://api.github.com/users/jd0-sag/followers","following_url":"https://api.github.com/users/jd0-sag/following{/other_user}","gists_url":"https://api.github.com/users/jd0-sag/gists{/gist_id}","starred_url":"https://api.github.com/users/jd0-sag/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jd0-sag/subscriptions","organizations_url":"https://api.github.com/users/jd0-sag/orgs","repos_url":"https://api.github.com/users/jd0-sag/repos","events_url":"https://api.github.com/users/jd0-sag/events{/privacy}","received_events_url":"https://api.github.com/users/jd0-sag/received_events","type":"User","site_admin":false},"created_at":"2016-06-29T02:58:19Z","updated_at":"2016-06-29T02:58:19Z","body":"What is it actually waiting for, in this case?  Is it specific to some machine or test since the galvan tests passed this.\r\n\r\nIt looks like the problem (and why this might not be detected in some testing environments) may be related to the amount of data buffered by the reader:\r\n\r\n      BufferedReader outputReader = new BufferedReader(new InputStreamReader(ps.getInputStream()));\r\n\r\nTo resolve that problem, we probably just need to move the waitFor to be after the loop.\r\n\r\nThe ProcessBuilder suggestion to handle the STDERR case is a good one, though, so I will also give that a shot.  I will prototype this and hopefully it can be resolved soon."}}
{"actor":{"display_login":"jd0-sag"}}