title: CAS Recordstore SocketTimeoutException
body: <p>While indexing from ATG, consistently we are getting the below error:</p>
  
  <blockquote>
    <p>Caused by: java.net.SocketTimeoutException: SocketTimeoutException
    invoking <a href="http://(endeca-server):8500/ExmProd_en_data/" rel="nofollow">http://(endeca-server):8500/ExmProd_en_data/</a>: Read timed out</p>
  </blockquote>
  
  <hr/>
  
  <p>Here are the logs:</p>
  
  <p><strong>JBoss server (ATG):</strong></p>
  
  <pre><code>2015-05-17 19:07:12,278 INFO  [nucleusNamespace.atg.endeca.index.DataDocumentSubmitter] (ProcessItemThread-482) Will attempt to contact record store "ExmProd_en_data" on host "&lt;endeca-server&gt;" on port 8,500
  ...
  (/atg/dynamo/service/Scheduler-reusablejobhandler-ProductCatalogSimpleIndexingAdmin - Baseline) Bulk load completed with "true" result in 6,097,145 milliseconds.
  2015-05-17 20:48:37,330 INFO  [nucleusNamespace.atg.endeca.index.SchemaDocumentSubmitter] (/atg/dynamo/service/Scheduler-reusablejobhandler-ProductCatalogSimpleIndexingAdmin - Baseline) Ending session of record store ExmProd_en_schema with transactionId 48 (submitted 11,531 records).
  2015-05-17 20:48:37,415 INFO  [nucleusNamespace.atg.endeca.index.DimensionDocumentSubmitter] (/atg/dynamo/service/Scheduler-reusablejobhandler-ProductCatalogSimpleIndexingAdmin - Baseline) Ending session of record store ExmProd_en_dimvals with transactionId 48 (submitted 178,870 records).
  2015-05-17 20:48:38,112 INFO  [nucleusNamespace.atg.endeca.index.DataDocumentSubmitter] (/atg/dynamo/service/Scheduler-reusablejobhandler-ProductCatalogSimpleIndexingAdmin - Baseline) Ending session of record store ExmProd_en_data with transactionId 48 (submitted 1,127,618 records).
  ...
  2015-05-17 21:18:38,164 WARN  [org.apache.cxf.phase.PhaseInterceptorChain] (/atg/dynamo/service/Scheduler-reusablejobhandler-ProductCatalogSimpleIndexingAdmin - Baseline) Interceptor for {http://recordstore.itl.endeca.com/}RecordStoreService#{http://recordstore.itl.endeca.com/}commitTransaction has thrown exception, unwinding now
  org.apache.cxf.interceptor.Fault: Could not send Message.
      at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)
      at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)
      at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:532)
      at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:464)
      at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:367)
      at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:320)
      at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:89)
      at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:134)
      at com.sun.proxy.$Proxy179.commitTransaction(Unknown Source)
      at atg.endeca.index.RecordStoreDocumentSubmitterSessionImpl.endSession(RecordStoreDocumentSubmitterSessionImpl.java:593)
      at atg.endeca.index.AbstractRecordStoreAggregateSession.commitPending(AbstractRecordStoreAggregateSession.java:494)
      at atg.endeca.index.AbstractRecordStoreAggregateSession.endSession(AbstractRecordStoreAggregateSession.java:432)
      at atg.endeca.index.RecordSubmitterImpl.endSession(RecordSubmitterImpl.java:231)
      at atg.endeca.index.admin.IndexingPhase.doPhaseCleanup(IndexingPhase.java:376)
      at atg.endeca.index.admin.IndexingPhase.performPhaseTasks(IndexingPhase.java:324)
      at atg.endeca.index.admin.IndexingPhase.performPhaseTasksSerially(IndexingPhase.java:243)
      at atg.endeca.index.admin.IndexingJob.performJobSerially(IndexingJob.java:281)
      at com.example.commerce.endeca.index.admin.EXMSimpleIndexingAdmin.indexBaseline(EXMSimpleIndexingAdmin.java:428)
      at atg.endeca.index.admin.SimpleIndexingAdmin.indexBaseline(SimpleIndexingAdmin.java:637)
      at com.example.commerce.endeca.index.admin.EXMSimpleIndexingAdmin.indexBaseline(EXMSimpleIndexingAdmin.java:262)
      at atg.endeca.index.admin.SimpleIndexingAdmin.performScheduledTask(SimpleIndexingAdmin.java:867)
      at atg.service.scheduler.ScheduledJob.runJobs(ScheduledJob.java:479)
      at atg.service.scheduler.Scheduler$2handler.run(Scheduler.java:780)
  Caused by: java.net.SocketTimeoutException: SocketTimeoutException invoking http://&lt;endeca-server&gt;:8500/ExmProd_en_data/: Read timed out
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1457)
      at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1442)
      at org.apache.cxf.io.AbstractWrappedOutputStream.close(AbstractWrappedOutputStream.java:72)
      at org.apache.cxf.io.AbstractThresholdOutputStream.close(AbstractThresholdOutputStream.java:102)
      at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
      at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:658)
      at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
      ... 22 more
  Caused by: java.net.SocketTimeoutException: Read timed out
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
      at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
      at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:698)
      at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:641)
      at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1218)
      at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
      at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1577)
      at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1520)
      at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1428)
      ... 27 more
  </code></pre>
  
  <p><strong>Endeca CAS server (cas-service.log):</strong>
  <br/>(No ERRORs were found in the log, only the below WARNing is getting repeated often)</p>
  
  <pre><code>2015-05-15 03:13:02,486 WARN [ExmProd_en_data] [ExmProd_en_data-cleaner-1] com.endeca.itl.recordstore.impl.Cleaner: Exception caught while performing cleanup
  java.lang.UnsupportedOperationException: Cleans while transactions are active not implemented yet
      at com.endeca.itl.recordstore.impl.RecordStoreImpl.clean(RecordStoreImpl.java:566)
      at com.endeca.itl.recordstore.impl.Cleaner$1.run(Cleaner.java:30)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)
      at com.endeca.itl.util.LoggingContextAwareThread.run(LoggingContextAwareThread.java:26)
  </code></pre>
  
  <hr/>
  
  <p>ATG Version: <strong>10.2</strong> (ProductCatalogSimpleIndexingAdmin)<br/>
  Endeca Version: <strong>3.1.2</strong> (pipeline based, i.e., Forge -> Dgidx).</p>
  
  <p>Any help in fixing this issue is much appreciated.</p>