title: Indexing a large dataset in elastic search causes java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-223 error
body: <p>I am indexing a large dataset using Elasticsearch's high level REST client for JAVA. When I say &quot;indexing&quot;, I mean creating the index, adding a mapping, adding a lot of data (8 million documents) and then aliasing the index. I notice that this error often causes the indexing to crash. Most often the crash occurs after the index and mapping are created and during the process of adding documents to the index. However, the error sometimes occurs beforehand while creating the index or the mapping.</p>
  <blockquote>
  <p>java.net.SocketTimeoutException: 30,000 milliseconds timeout on
  connection http-outgoing-223 [ACTIVE]</p>
  <p>Occasionally there is a &quot;circuit-breaking&quot; error.</p>
  </blockquote>
  <p>I'm pasting both errors below</p>
  <p>I have stepped through my code and see this error does not happen at any consistent location in my code. The error also does not seem to occur when adding a document with any particularly unique values.</p>
  <p>I have increased the RAM in my elasticsearch cluster's aws.data.highio.i3 instance from 1GB to 8GB and the issue persists.</p>
  <p>Here is the entire stack trace for the socket timeout error...</p>
  <pre><code>    WARN  [2020-04-19 19:58:05,923] com.zaxxer.hikari.pool.PoolBase: HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@437eff49 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
  ERROR [2020-04-19 19:58:41,331] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: 1ed68c8eb0476ec6
  ! java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
  ! at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
  ! at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
  ! at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
  ! at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
  ! at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
  ! at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
  ! at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
  ! at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
  ! at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
  ! at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
  ! ... 1 common frames omitted
  ! Causing: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-0 [ACTIVE]
  ! at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:808)
  ! at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248)
  ! at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
  ! at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1514)
  ! at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1484)
  ! at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454)
  ! at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:871)
  ! at org.project.core.services.ElasticSearchService.createElasticIndex(ElasticSearchService.java:613)
  ! at org.project.core.services.ElasticSearchService.reindex(ElasticSearchService.java:726)
  ! at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  ! at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  ! at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  ! at java.base/java.lang.reflect.Method.invoke(Method.java:566)
  ! at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
  ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
  ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
  ! at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
  ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
  ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
  ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
  ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
  ! at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
  ! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
  ! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
  ! at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
  ! at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
  ! at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
  ! at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
  ! at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
  ! at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
  ! at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
  ! at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
  ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
  ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
  ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
  ! at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:50)
  ! at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
  ! at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)
  ! at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
  ! at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)
  ! at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)
  ! at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
  ! at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:310)
  ! at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:264)
  ! at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
  ! at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
  ! at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
  ! at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
  ! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
  ! at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
  ! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
  ! at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
  ! at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
  ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
  ! at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:249)
  ! at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
  ! at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)
  ! at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
  ! at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
  ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
  ! at org.eclipse.jetty.server.Server.handle(Server.java:500)
  ! at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
  ! at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
  ! at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
  ! at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
  ! at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
  ! at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
  ! at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
  ! at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
  ! at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
  ! at java.base/java.lang.Thread.run(Thread.java:834)
  </code></pre>
  <p>Here is the entire stack trace for the circuit breaking error...</p>
  <pre><code>INFO  [2020-04-19 19:34:09,206] org.project.core.services.ElasticSearchService: Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [&lt;http_request&gt;] would be [410582854/391.5mb], which is larger than the limit of [394910105/376.6mb], real usage: [392568768/374.3mb], new bytes reserved: [18014086/17.1mb], usages [request=0/0b, fielddata=341/341b, in_flight_requests=18014086/17.1mb, accounting=1124688/1mb]]
  ERROR [2020-04-19 19:34:09,223] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: 656d425f54800b53
  ! java.lang.NullPointerException: null
  ! at org.project.core.services.ElasticSearchService.updateDocuments(ElasticSearchService.java:489)
  ! at org.project.core.services.ElasticSearchService.addRecordsFromPostgresToElasticIndex(ElasticSearchService.java:684)
  ! at org.project.core.services.ElasticSearchService.reindex(ElasticSearchService.java:727)
  ! at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  ! at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  ! at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  ! at java.base/java.lang.reflect.Method.invoke(Method.java:566)
  ! at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
  ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
  ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
  ! at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
  ! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
  ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
  ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
  ! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
  ! at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
  ! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
  ! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
  ! at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
  ! at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
  ! at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
  ! at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
  ! at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
  ! at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
  ! at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
  ! at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
  ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
  ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
  ! at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
  ! at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:50)
  ! at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
  ! at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)
  ! at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
  ! at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)
  ! at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)
  ! at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
  ! at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:310)
  ! at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:264)
  ! at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
  ! at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
  ! at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
  ! at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
  ! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
  ! at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
  ! at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
  ! at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
  ! at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
  ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
  ! at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:249)
  ! at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52)
  ! at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)
  ! at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
  ! at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
  ! at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
  ! at org.eclipse.jetty.server.Server.handle(Server.java:500)
  ! at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
  ! at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
  ! at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
  ! at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
  ! at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
  ! at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
  ! at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
  ! at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
  ! at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
  ! at java.base/java.lang.Thread.run(Thread.java:834)
  </code></pre>