title: Zuul Server with High Volume fails
body: <p>I created a sample <strong>zuul server</strong> based on this: <a href="https://github.com/spring-cloud-samples/zuul-server" rel="nofollow">https://github.com/spring-cloud-samples/zuul-server</a>,  a <strong>demo-service</strong> that calls a <strong>dummy service</strong> with ribbon and an <strong>eureka server</strong>. From zuul goes to demo-service and then to dummy-service,  but when I throw at zuul 200 request per second (using jmeter) it starts failing.  </p>
  
  <blockquote>
    <p>forwarding error</p>
  </blockquote>
  
  <p>and the cause is:</p>
  
  <blockquote>
    <p>Caused by: java.net.SocketTimeoutException: Read timed out</p>
  </blockquote>
  
  <pre><code>    com.netflix.zuul.exception.ZuulException: Forwarding error
      at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:146)
      at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.run(RibbonRoutingFilter.java:110)
      at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:112)
      at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:197)
      at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:161)
      at com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:120)
      at com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:84)
      at com.netflix.zuul.http.ZuulServlet.route(ZuulServlet.java:111)
      at com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:77)
      at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:158)
      at org.springframework.cloud.netflix.zuul.web.ZuulController.handleRequestInternal(ZuulController.java:43)
      at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:146)
      at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
      at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943)
      at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877)
      at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966)
      at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:868)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:644)
      at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:725)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:291)
      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:102)
      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.springframework.boot.actuate.autoconfigure.MetricFilterAutoConfiguration$MetricsFilter.doFilterInternal(MetricFilterAutoConfiguration.java:90)
      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:516)
      at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1086)
      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:659)
      at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:223)
      at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1558)
      at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1515)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: demo-serviceRibbonCommand failed and no fallback available.
      at com.netflix.hystrix.AbstractCommand$20.call(AbstractCommand.java:816)
      at com.netflix.hystrix.AbstractCommand$20.call(AbstractCommand.java:798)
      at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$1.onError(OperatorOnErrorResumeNextViaFunction.java:76)
      at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70)
      at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70)
      at rx.Observable$ThrowObservable$1.call(Observable.java:10493)
      at rx.Observable$ThrowObservable$1.call(Observable.java:10483)
      at rx.Observable.unsafeSubscribe(Observable.java:8591)
      at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$1.onError(OperatorOnErrorResumeNextViaFunction.java:77)
      at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
      at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:318)
      at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:310)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable.unsafeSubscribe(Observable.java:8591)
      at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$1.onError(OperatorOnErrorResumeNextViaFunction.java:77)
      at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70)
      at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
      at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$3.onError(AbstractCommand.java:995)
      at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70)
      at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70)
      at rx.Observable$ThrowObservable$1.call(Observable.java:10493)
      at rx.Observable$ThrowObservable$1.call(Observable.java:10483)
      at rx.Observable.unsafeSubscribe(Observable.java:8591)
      at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$1.onError(OperatorOnErrorResumeNextViaFunction.java:77)
      at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
      at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:301)
      at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:293)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable.unsafeSubscribe(Observable.java:8591)
      at com.netflix.hystrix.AbstractCommand$1.call(AbstractCommand.java:388)
      at com.netflix.hystrix.AbstractCommand$1.call(AbstractCommand.java:368)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable.unsafeSubscribe(Observable.java:8591)
      at com.netflix.hystrix.AbstractCommand$ObservableCommand$1.call(AbstractCommand.java:1097)
      at com.netflix.hystrix.AbstractCommand$ObservableCommand$1.call(AbstractCommand.java:1093)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable.subscribe(Observable.java:8680)
      at rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:55)
      at rx.observables.BlockingObservable.toFuture(BlockingObservable.java:420)
      at com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:385)
      at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:339)
      at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:130)
      ... 60 common frames omitted
  Caused by: com.netflix.client.ClientException: null
      at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:123)
      at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:81)
      at org.springframework.cloud.netflix.zuul.filters.route.RibbonCommand.forward(RibbonCommand.java:129)
      at org.springframework.cloud.netflix.zuul.filters.route.RibbonCommand.run(RibbonCommand.java:103)
      at org.springframework.cloud.netflix.zuul.filters.route.RibbonCommand.run(RibbonCommand.java:48)
      at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:298)
      ... 103 common frames omitted
  Caused by: java.net.SocketTimeoutException: Read timed out
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      at java.net.SocketInputStream.read(SocketInputStream.java:170)
      at java.net.SocketInputStream.read(SocketInputStream.java:141)
      at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
      at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
      at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
      at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
      at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
      at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
      at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:223)
      at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
      at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
      at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
      at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
      at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:115)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
      at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:170)
      at com.sun.jersey.api.client.Client.handle(Client.java:648)
      at com.sun.jersey.api.client.WebResource.handle(WebResource.java:680)
      at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
      at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:568)
      at com.netflix.niws.client.http.RestClient.execute(RestClient.java:621)
      at com.netflix.niws.client.http.RestClient.execute(RestClient.java:527)
      at com.netflix.niws.client.http.RestClient.execute(RestClient.java:92)
      at com.netflix.client.AbstractLoadBalancerAwareClient$1.call(AbstractLoadBalancerAwareClient.java:109)
      at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:303)
      at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:287)
      at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:54)
      at rx.internal.util.ScalarSynchronousObservable$1.call(ScalarSynchronousObservable.java:41)
      at rx.internal.util.ScalarSynchronousObservable$1.call(ScalarSynchronousObservable.java:30)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable.unsafeSubscribe(Observable.java:8591)
      at rx.internal.operators.OperatorConcat$ConcatSubscriber.subscribeNext(OperatorConcat.java:163)
      at rx.internal.operators.OperatorConcat$ConcatSubscriber.onNext(OperatorConcat.java:128)
      at rx.internal.operators.OperatorConcat$ConcatSubscriber.onNext(OperatorConcat.java:65)
      at rx.internal.operators.OperatorMap$1.onNext(OperatorMap.java:54)
      at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:185)
      at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable.unsafeSubscribe(Observable.java:8591)
      at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber$1.call(OperatorRetryWithPredicate.java:112)
      at rx.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:85)
      at rx.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:65)
      at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:77)
      at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:45)
      at rx.internal.util.ScalarSynchronousObservable$1.call(ScalarSynchronousObservable.java:41)
      at rx.internal.util.ScalarSynchronousObservable$1.call(ScalarSynchronousObservable.java:30)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable$2.call(Observable.java:173)
      at rx.Observable$2.call(Observable.java:166)
      at rx.Observable.subscribe(Observable.java:8680)
      at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:452)
      at rx.observables.BlockingObservable.single(BlockingObservable.java:348)
      at enter code herecom.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:102)
      ... 108 common frames omitted
  </code></pre>
  
  <p>I have tried several configs but seems that nothing changes. The application.properties I have now is:</p>
  
  <pre><code>server.port= 8766    
  hystrix.command.default.execution.isolation.strategy= SEMAPHORE
  hystrix.command.default.fallback.isolation.semaphore.maxConcurrentRequests= 700
  
  hystrix.command.default.timeout.enabled= false
  hystrix.command.default.circuitBreaker.enabled=false
  hystrix.command.default.circuitBreaker.forceClosed=false
  
  logging.level.ROOT= INFO
  logging.level.org.springframework.web= INFO
  </code></pre>
  
  <p>I am not implementing any Hystrix fallback methods.
  When I take out Zuul from the equation and hit the demo-service directly, I can throw at it up to 2000 request per second and all are success</p>
  
  <p>The questions are:</p>
  
  <ol>
  <li>Why does Zuul support only a few transactions per second?</li>
  <li>I have tried without any configuration and is the same, what configs should be there in order to support higher volume?</li>
  </ol>
  
  <p>All I want is for zuul to route to a service and load balance between the instances of that service using ribbon, with success state at least from 200 threds per second or more.</p>
  
  <p>Any help will me much appreciated </p>