{"action":"created","issue":{"url":"https://api.github.com/repos/Netflix/Hystrix/issues/1051","labels_url":"https://api.github.com/repos/Netflix/Hystrix/issues/1051/labels{/name}","comments_url":"https://api.github.com/repos/Netflix/Hystrix/issues/1051/comments","events_url":"https://api.github.com/repos/Netflix/Hystrix/issues/1051/events","html_url":"https://github.com/Netflix/Hystrix/issues/1051","id":126737596,"number":1051,"title":"Builds hang due to Hystrix locks","user":{"login":"marcingrzejszczak","id":3297437,"avatar_url":"https://avatars.githubusercontent.com/u/3297437?v=3","gravatar_id":"","url":"https://api.github.com/users/marcingrzejszczak","html_url":"https://github.com/marcingrzejszczak","followers_url":"https://api.github.com/users/marcingrzejszczak/followers","following_url":"https://api.github.com/users/marcingrzejszczak/following{/other_user}","gists_url":"https://api.github.com/users/marcingrzejszczak/gists{/gist_id}","starred_url":"https://api.github.com/users/marcingrzejszczak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marcingrzejszczak/subscriptions","organizations_url":"https://api.github.com/users/marcingrzejszczak/orgs","repos_url":"https://api.github.com/users/marcingrzejszczak/repos","events_url":"https://api.github.com/users/marcingrzejszczak/events{/privacy}","received_events_url":"https://api.github.com/users/marcingrzejszczak/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Netflix/Hystrix/labels/hystrix-core","name":"hystrix-core","color":"009800"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-01-14T20:05:46Z","updated_at":"2016-01-25T04:37:07Z","closed_at":null,"body":"Some of our builds are hanging on Hystrix related tests on our EC2 instance (C3 Compute Optimised Double Extra Large). This is the link of the last one: https://build.spring.io/download/SCE2E-SCSE2E-JOB1/build_logs/SCE2E-SCSE2E-JOB1-324.log\r\n\r\nI did a thread dump of that build when it was stalling and this is its part\r\n\r\n```\r\nThread 7521: (state = BLOCKED)\r\n - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)\r\n - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Interpreted frame)\r\n - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Interpreted frame)\r\n - java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=72, line=997 (Interpreted frame)\r\n - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) @bci=24, line=1304 (Interpreted frame)\r\n - java.util.concurrent.CountDownLatch.await() @bci=5, line=231 (Interpreted frame)\r\n - rx.internal.operators.BlockingOperatorToFuture$2.get() @bci=4, line=107 (Interpreted frame)\r\n - com.netflix.hystrix.HystrixCommand.execute() @bci=4, line=335 (Interpreted frame)\r\n - com.netflix.hystrix.contrib.javanica.command.CommandExecutor.execute(com.netflix.hystrix.HystrixExecutable, com.netflix.hystrix.contrib.javanica.command.ExecutionType, com.netflix.hystrix.contrib.jav\r\nanica.command.MetaHolder) @bci=45, line=45 (Interpreted frame)\r\n - com.netflix.hystrix.contrib.javanica.aop.aspectj.HystrixCommandAspect.methodsAnnotatedWithHystrixCommand(org.aspectj.lang.ProceedingJoinPoint) @bci=137, line=96 (Interpreted frame)\r\n - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Compiled frame)\r\n - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=100, line=62 (Compiled frame)\r\n - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)\r\n - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=497 (Compiled frame)\r\n - org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(java.lang.Object[]) @bci=36, line=621 (Interpreted frame)\r\n - org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(org.aspectj.lang.JoinPoint, org.aspectj.weaver.tools.JoinPointMatch, java.lang.Object, java.lang.Throwable) @bci=10, line=610\r\n(Interpreted frame)\r\n - org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(org.aopalliance.intercept.MethodInvocation) @bci=58, line=68 (Interpreted frame)\r\n - org.springframework.aop.framework.ReflectiveMethodInvocation.proceed() @bci=85, line=168 (Interpreted frame)\r\n - org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(org.aopalliance.intercept.MethodInvocation) @bci=18, line=92 (Interpreted frame)\r\n - org.springframework.aop.framework.ReflectiveMethodInvocation.proceed() @bci=101, line=179 (Interpreted frame)\r\n - org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(java.lang.Object, java.lang.reflect.Method, java.lang.Object[], org.springframework.cglib.proxy.MethodProxy) @bci=1\r\n20, line=655 (Interpreted frame)\r\n - org.springframework.cloud.sleuth.instrument.hystrix.SpanPassingForHystrixViaAnnotationsIntegrationTests$HystrixCommandInvocationSpanCatcher$$EnhancerBySpringCGLIB$$5b1dcfe1.invokeLogicWrappedInHystrixCommand() @bci=31 (Interpreted frame)\r\n - org.springframework.cloud.sleuth.instrument.hystrix.SpanPassingForHystrixViaAnnotationsIntegrationTests.whenHystrixCommandAnnotatedMethodGetsExecuted() @bci=4, line=47 (Interpreted frame)\r\n - org.springframework.cloud.sleuth.instrument.hystrix.SpanPassingForHystrixViaAnnotationsIntegrationTests.should_set_span_on_an_hystrix_command_annotated_method() @bci=6, line=35 (Interpreted frame)\r\n```\r\n\r\nThere is also another log that is stalling at the same place - https://build.spring.io/download/SCE2E-SCSE2E-JOB1/build_logs/SCE2E-SCSE2E-JOB1-323.log . Here the thread is waiting (by mistake I've done two subsequent thread dumps).\r\n\r\nI don't know if it's Hystrix related but looks that way (similar issues we had with spring-cloud-netflix). Here is the link to the test: https://github.com/spring-cloud/spring-cloud-sleuth/blob/master/spring-cloud-sleuth-core/src/test/java/org/springframework/cloud/sleuth/instrument/hystrix/SpanPassingForHystrixViaAnnotationsIntegrationTests.java"},"comment":{"url":"https://api.github.com/repos/Netflix/Hystrix/issues/comments/174397299","html_url":"https://github.com/Netflix/Hystrix/issues/1051#issuecomment-174397299","issue_url":"https://api.github.com/repos/Netflix/Hystrix/issues/1051","id":174397299,"user":{"login":"mattrjacobs","id":205579,"avatar_url":"https://avatars.githubusercontent.com/u/205579?v=3","gravatar_id":"","url":"https://api.github.com/users/mattrjacobs","html_url":"https://github.com/mattrjacobs","followers_url":"https://api.github.com/users/mattrjacobs/followers","following_url":"https://api.github.com/users/mattrjacobs/following{/other_user}","gists_url":"https://api.github.com/users/mattrjacobs/gists{/gist_id}","starred_url":"https://api.github.com/users/mattrjacobs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattrjacobs/subscriptions","organizations_url":"https://api.github.com/users/mattrjacobs/orgs","repos_url":"https://api.github.com/users/mattrjacobs/repos","events_url":"https://api.github.com/users/mattrjacobs/events{/privacy}","received_events_url":"https://api.github.com/users/mattrjacobs/received_events","type":"User","site_admin":false},"created_at":"2016-01-25T04:37:07Z","updated_at":"2016-01-25T04:37:07Z","body":"@marcingrzejszczak I haven't been able to reproduce this while running locally, which I think you said was your situation as well.\r\n\r\nWhat I can gather from the stack trace included in the link to your failed CI build is:\r\n\r\n* `org.springframework.cloud.sleuth.instrument.TraceDelegate.startSpan(TraceDelegate.java:59)` fails with `java.lang.IllegalStateException: org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@1d81e101 has been closed already`\r\n\r\nAFAICT, that failure is within the module you're testing, and Hystrix isn't responsible for that error occuring.\r\n\r\nOnce that error occurs, the HystrixTimer thread attempts to run and give the caller thread a Timeout OnError.  However, the `HystrixContextRunnable` throws an error before that can get delivered.  As a result, the caller thread waits forever.\r\n\r\nAre you registering a `HystrixConcurrencyStrategy`?\r\n\r\nThe code that is throwing is `com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:59)`  This implies that the following code is throwing: `concurrencyStrategy.wrapCallable(actual.run()).call()`.\r\n\r\nIn this case, `actual` refers to the OnError delivery, and should not be the source of the exception.  So I'm curious if you've got a `HystrixConcurrencyStrategy` that could be generating this error"}}