{"action":"opened","issue":{"url":"https://api.github.com/repos/dotCMS/core/issues/15230","repository_url":"https://api.github.com/repos/dotCMS/core","labels_url":"https://api.github.com/repos/dotCMS/core/issues/15230/labels{/name}","comments_url":"https://api.github.com/repos/dotCMS/core/issues/15230/comments","events_url":"https://api.github.com/repos/dotCMS/core/issues/15230/events","html_url":"https://github.com/dotCMS/core/issues/15230","id":356894285,"node_id":"MDU6SXNzdWUzNTY4OTQyODU=","number":15230,"title":"Elastic Threads stuck in dotCMS","user":{"login":"wezell","id":934364,"node_id":"MDQ6VXNlcjkzNDM2NA==","avatar_url":"https://avatars2.githubusercontent.com/u/934364?v=4","gravatar_id":"","url":"https://api.github.com/users/wezell","html_url":"https://github.com/wezell","followers_url":"https://api.github.com/users/wezell/followers","following_url":"https://api.github.com/users/wezell/following{/other_user}","gists_url":"https://api.github.com/users/wezell/gists{/gist_id}","starred_url":"https://api.github.com/users/wezell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wezell/subscriptions","organizations_url":"https://api.github.com/users/wezell/orgs","repos_url":"https://api.github.com/users/wezell/repos","events_url":"https://api.github.com/users/wezell/events{/privacy}","received_events_url":"https://api.github.com/users/wezell/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-09-04T16:51:27Z","updated_at":"2018-09-04T16:51:27Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Running ElasticSearch 6.1.3 Node Client in our Tomcat Application server, connecting to ElasticSearch 6.1.3 cluster in another JVM on the same machine.  Our Tomcat Application quits handling requests because request threads are all parked waiting for response from ElasticSearch index search; however, the threaddumps from the ElasticSearch server shows that it does not realize anyone is waiting on results.\r\n\r\nProblem seems to me to be just like this problem that was fixed with the TransportClient:  https://github.com/elastic/elasticsearch/issues/10766\r\n\r\nThe main reason I say that is this thread stack that shows our code running in the elasticsearch transport_client_boss thread:\r\n    \"elasticsearch[eaf33a49-7544-4620-ac14-c52189b85885][transport_client_boss][T#1]\" #693 daemon prio=5 os_prio=0 tid=0x0000557d91beb000 nid=0x8bc waiting on condition [0x00007fb4e587c000]\r\n       java.lang.Thread.State: WAITING (parking)\r\n        at sun.misc.Unsafe.park(Native Method)\r\n        - parking to wait for  <0x00000000b77303a8> (a org.elasticsearch.common.util.concurrent.BaseFuture$Sync)\r\n        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)\r\n        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)\r\n        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)\r\n        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)\r\n        at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:251)\r\n        at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:94)\r\n        at org.elasticsearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:39)\r\n        at com.dotcms.content.elasticsearch.business.ESContentFactoryImpl.indexSearch(ESContentFactoryImpl.java:1414)\r\n        at com.dotcms.content.elasticsearch.business.ESContentletAPIImpl.isInodeIndexedWithQuery(ESContentletAPIImpl.java:5599)\r\n        at com.dotcms.content.elasticsearch.business.ESContentletAPIImpl.isInodeIndexedWithQuery(ESContentletAPIImpl.java:5585)\r\n        at com.dotcms.content.elasticsearch.business.ESContentletAPIImpl.isInodeIndexed(ESContentletAPIImpl.java:5535)\r\n        at com.dotcms.content.elasticsearch.business.ESContentletAPIImpl.isInodeIndexed(ESContentletAPIImpl.java:5526)\r\n        at com.dotmarketing.portlets.contentlet.business.ContentletAPIInterceptor.isInodeIndexed(ContentletAPIInterceptor.java:1958)\r\n        at com.dotmarketing.portlets.contentlet.business.web.ContentletWebAPIImpl.lambda$6(ContentletWebAPIImpl.java:176)\r\n        at com.dotmarketing.portlets.contentlet.business.web.ContentletWebAPIImpl$$Lambda$2222/1923093073.run(Unknown Source)\r\n        at com.dotmarketing.db.HibernateUtil$DotAsyncRunnable.run(HibernateUtil.java:898)\r\n        at com.dotmarketing.db.DotRunnableThread$2$$Lambda$2223/525496524.accept(Unknown Source)\r\n        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)\r\n        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)\r\n        at com.dotmarketing.db.DotRunnableThread$2.onResponse(DotRunnableThread.java:108)\r\n        at com.dotmarketing.db.DotRunnableThread$2.onResponse(DotRunnableThread.java:1)\r\n        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85)\r\n        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81)\r\n        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:380)\r\n        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:361)\r\n        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:350)\r\n        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85)\r\n        at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81)\r\n        at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishOnSuccess(TransportReplicationAction.java:936)\r\n        at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:846)\r\n        at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:832)\r\n        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1049)\r\n        at org.elasticsearch.transport.TcpTransport$2.doRun(TcpTransport.java:1450)\r\n        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n        at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:135)\r\n        at org.elasticsearch.transport.TcpTransport.handleResponse(TcpTransport.java:1442)\r\n        at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1398)\r\n        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:64)\r\n    ...\r\n\r\n\r\nFull thread dumps available here for reference:  https://gist.github.com/brentgriffin/80fccab5be0c0a040425c7dbd983072a\r\n\r\nThe code base has changed alot in the 3.5 years since the fix was made, but if I followed things correctly, I believe the current implementation for this fix with the transport client is here:  https://github.com/elastic/elasticsearch/blob/v6.1.3/core/src/main/java/org/elasticsearch/action/support/ThreadedActionListener.java#L52-L54\r\n\r\nIf this is the correct code, then it does not seem that there is a way for me to configure ES to enable this functionality for a node client.  Is that correct?\r\n\r\nDefinitely interested in thoughts and suggestions here.  From my limited viewpoint, the options I see are as follows:\r\n* Change type of client used from Node Client to REST API or TransportClient (been deprecated in recent releases)\r\n* Refactor our code to prevent it from executing ES queries inside of the transport_client_boss thread\r\n* Add a timeout parameter to our call to actionGet() - this seems like a hack and while I think it would prevent the server from parking all threads waiting on ES index searches, it would not permit these requests to successfully search the index\r\n\r\nWhat am I missing?\r\n\r\nIs it a bug that the threadListener cannot be configured for a node client or is this by design?"}}
{"actor":{"display_login":"wezell"}}