{"action":"closed","issue":{"url":"https://api.github.com/repos/apache/skywalking/issues/2658","repository_url":"https://api.github.com/repos/apache/skywalking","labels_url":"https://api.github.com/repos/apache/skywalking/issues/2658/labels{/name}","comments_url":"https://api.github.com/repos/apache/skywalking/issues/2658/comments","events_url":"https://api.github.com/repos/apache/skywalking/issues/2658/events","html_url":"https://github.com/apache/skywalking/issues/2658","id":443682842,"node_id":"MDU6SXNzdWU0NDM2ODI4NDI=","number":2658,"title":"There are numbers of \"TIMED WAITING\" & \"BLOCKED\" threads in jstack log","user":{"login":"chenmutime","id":13652596,"node_id":"MDQ6VXNlcjEzNjUyNTk2","avatar_url":"https://avatars0.githubusercontent.com/u/13652596?v=4","gravatar_id":"","url":"https://api.github.com/users/chenmutime","html_url":"https://github.com/chenmutime","followers_url":"https://api.github.com/users/chenmutime/followers","following_url":"https://api.github.com/users/chenmutime/following{/other_user}","gists_url":"https://api.github.com/users/chenmutime/gists{/gist_id}","starred_url":"https://api.github.com/users/chenmutime/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chenmutime/subscriptions","organizations_url":"https://api.github.com/users/chenmutime/orgs","repos_url":"https://api.github.com/users/chenmutime/repos","events_url":"https://api.github.com/users/chenmutime/events{/privacy}","received_events_url":"https://api.github.com/users/chenmutime/received_events","type":"User","site_admin":false},"labels":[{"id":830850453,"node_id":"MDU6TGFiZWw4MzA4NTA0NTM=","url":"https://api.github.com/repos/apache/skywalking/labels/OAP-backend","name":"OAP-backend","color":"54ceff","default":false},{"id":283421099,"node_id":"MDU6TGFiZWwyODM0MjEwOTk=","url":"https://api.github.com/repos/apache/skywalking/labels/question","name":"question","color":"cc317c","default":true}],"state":"closed","locked":false,"assignee":{"login":"wu-sheng","id":5441976,"node_id":"MDQ6VXNlcjU0NDE5NzY=","avatar_url":"https://avatars2.githubusercontent.com/u/5441976?v=4","gravatar_id":"","url":"https://api.github.com/users/wu-sheng","html_url":"https://github.com/wu-sheng","followers_url":"https://api.github.com/users/wu-sheng/followers","following_url":"https://api.github.com/users/wu-sheng/following{/other_user}","gists_url":"https://api.github.com/users/wu-sheng/gists{/gist_id}","starred_url":"https://api.github.com/users/wu-sheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wu-sheng/subscriptions","organizations_url":"https://api.github.com/users/wu-sheng/orgs","repos_url":"https://api.github.com/users/wu-sheng/repos","events_url":"https://api.github.com/users/wu-sheng/events{/privacy}","received_events_url":"https://api.github.com/users/wu-sheng/received_events","type":"User","site_admin":false},"assignees":[{"login":"wu-sheng","id":5441976,"node_id":"MDQ6VXNlcjU0NDE5NzY=","avatar_url":"https://avatars2.githubusercontent.com/u/5441976?v=4","gravatar_id":"","url":"https://api.github.com/users/wu-sheng","html_url":"https://github.com/wu-sheng","followers_url":"https://api.github.com/users/wu-sheng/followers","following_url":"https://api.github.com/users/wu-sheng/following{/other_user}","gists_url":"https://api.github.com/users/wu-sheng/gists{/gist_id}","starred_url":"https://api.github.com/users/wu-sheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wu-sheng/subscriptions","organizations_url":"https://api.github.com/users/wu-sheng/orgs","repos_url":"https://api.github.com/users/wu-sheng/repos","events_url":"https://api.github.com/users/wu-sheng/events{/privacy}","received_events_url":"https://api.github.com/users/wu-sheng/received_events","type":"User","site_admin":false},{"login":"peng-yongsheng","id":16147247,"node_id":"MDQ6VXNlcjE2MTQ3MjQ3","avatar_url":"https://avatars0.githubusercontent.com/u/16147247?v=4","gravatar_id":"","url":"https://api.github.com/users/peng-yongsheng","html_url":"https://github.com/peng-yongsheng","followers_url":"https://api.github.com/users/peng-yongsheng/followers","following_url":"https://api.github.com/users/peng-yongsheng/following{/other_user}","gists_url":"https://api.github.com/users/peng-yongsheng/gists{/gist_id}","starred_url":"https://api.github.com/users/peng-yongsheng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/peng-yongsheng/subscriptions","organizations_url":"https://api.github.com/users/peng-yongsheng/orgs","repos_url":"https://api.github.com/users/peng-yongsheng/repos","events_url":"https://api.github.com/users/peng-yongsheng/events{/privacy}","received_events_url":"https://api.github.com/users/peng-yongsheng/received_events","type":"User","site_admin":false}],"milestone":null,"comments":4,"created_at":"2019-05-14T02:38:10Z","updated_at":"2019-07-02T14:58:48Z","closed_at":"2019-07-02T14:58:48Z","author_association":"NONE","body":"[environment]\r\n1 single virtual machine, 4C/8GB, with es & skywalking server deployed.\r\nstandalone one-node elasticsearch, assigned Xmx4g for jvm.\r\nskywalking server deployed on the same vm, assigned Xmx2g for jvm.\r\n2 spring boot applications (deployed on another vm) connected to skywalking server.\r\n\r\n[symptom]\r\n1. After restarting es & skywalking server, skywalking server receives continued traces from application.\r\n2. It lasts 15-30 minutes. after then, no traces come into skywalking server any more.\r\n\r\n[effort taken]\r\n1. I've checked elasticsearch.log. There are no obvious error messages in elasticsearch.log.\r\n2. I've checked gc.log. There are no gc overhead messages in gc.log.\r\n3. I've printed stacktrace using jstack, log shows as below:\r\n\r\n```\r\n\"DataCarrier.IndicatorPersistentWorker.all_p50_day.Consumser.0.Thread\" #20 daemon prio=5 os_prio=0 tid=0x00007f3e9507f000 nid=0x2e99 waiting on condition [0x00007f3e58acd000]\r\njava.lang.Thread.State: TIMED_WAITING (sleeping)\r\nat java.lang.Thread.sleep(Native Method)\r\nat org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.run(ConsumerThread.java:72)\r\n```\r\n```\r\n\"DataCarrier.IndicatorPersistentWorker.endpoint_avg.Consumser.0.Thread\" #43 daemon prio=5 os_prio=0 tid=0x00007f3e950e2800 nid=0x2eb0 waiting for monitor entry [0x00007f3dfb2f1000]\r\njava.lang.Thread.State: BLOCKED (on object monitor)\r\nat org.elasticsearch.action.bulk.BulkProcessor.internalAdd(BulkProcessor.java:286)\r\n- waiting to lock <0x00000000813c8440> (a org.elasticsearch.action.bulk.BulkProcessor)\r\nat org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:271)\r\nat org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:267)\r\nat org.elasticsearch.action.bulk.BulkProcessor.add(BulkProcessor.java:253)\r\nat org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.BatchProcessEsDAO.lambda$batchPersistence$0(BatchProcessEsDAO.java:64)\r\nat org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.BatchProcessEsDAO$$Lambda$278/81782502.accept(Unknown Source)\r\nat java.lang.Iterable.forEach(Iterable.java:75)\r\nat org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.BatchProcessEsDAO.batchPersistence(BatchProcessEsDAO.java:62)\r\nat org.apache.skywalking.oap.server.core.analysis.worker.PersistenceWorker.onWork(PersistenceWorker.java:51)\r\nat org.apache.skywalking.oap.server.core.analysis.worker.IndicatorPersistentWorker.onWork(IndicatorPersistentWorker.java:63)\r\nat org.apache.skywalking.oap.server.core.analysis.worker.IndicatorPersistentWorker$PersistentConsumer.consume(IndicatorPersistentWorker.java:153)\r\nat org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.consume(ConsumerThread.java:101)\r\nat org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.run(ConsumerThread.java:68)\r\n```\r\n\r\n```\r\n\"DataCarrier.IndicatorAggregateWorker.instance_jvm_cpu.Consumser.0.Thread\" #152 daemon prio=5 os_prio=0 tid=0x00007f3e951c1000 nid=0x2f1d sleeping[0x00007f3df4584000]\r\njava.lang.Thread.State: TIMED_WAITING (sleeping)\r\nat java.lang.Thread.sleep(Native Method)\r\nat org.apache.skywalking.apm.commons.datacarrier.buffer.Buffer.save(Buffer.java:64)\r\nat org.apache.skywalking.apm.commons.datacarrier.buffer.Channels.save(Channels.java:52)\r\nat org.apache.skywalking.apm.commons.datacarrier.DataCarrier.produce(DataCarrier.java:88)\r\nat org.apache.skywalking.oap.server.core.analysis.worker.IndicatorPersistentWorker.in(IndicatorPersistentWorker.java:68)\r\nat org.apache.skywalking.oap.server.core.analysis.worker.IndicatorTransWorker.in(IndicatorTransWorker.java:89)\r\nat org.apache.skywalking.oap.server.core.analysis.worker.IndicatorTransWorker.in(IndicatorTransWorker.java:32)\r\nat org.apache.skywalking.oap.server.core.remote.client.SelfRemoteClient.push(SelfRemoteClient.java:55)\r\nat org.apache.skywalking.oap.server.core.remote.RemoteSenderService.send(RemoteSenderService.java:51)\r\nat org.apache.skywalking.oap.server.core.analysis.worker.IndicatorRemoteWorker.in(IndicatorRemoteWorker.java:51)\r\nat org.apache.skywalking.oap.server.core.analysis.worker.IndicatorRemoteWorker.in(IndicatorRemoteWorker.java:33)\r\nat org.apache.skywalking.oap.server.core.analysis.worker.IndicatorAggregateWorker.lambda$sendToNext$0(IndicatorAggregateWorker.java:93)\r\nat org.apache.skywalking.oap.server.core.analysis.worker.IndicatorAggregateWorker$$Lambda$203/1364775767.accept(Unknown Source)\r\nat java.util.HashMap$Values.forEach(HashMap.java:981)\r\nat org.apache.skywalking.oap.server.core.analysis.worker.IndicatorAggregateWorker.sendToNext(IndicatorAggregateWorker.java:88)\r\nat org.apache.skywalking.oap.server.core.analysis.worker.IndicatorAggregateWorker.onWork(IndicatorAggregateWorker.java:73)\r\nat org.apache.skywalking.oap.server.core.analysis.worker.IndicatorAggregateWorker.access$100(IndicatorAggregateWorker.java:38)\r\nat org.apache.skywalking.oap.server.core.analysis.worker.IndicatorAggregateWorker$AggregatorConsumer.consume(IndicatorAggregateWorker.java:131)\r\nat org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.consume(ConsumerThread.java:101)\r\nat org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.run(ConsumerThread.java:68)\r\n```\r\n\r\n4. I've checked es cluster stat and node stat.\r\nheap percent is in normal range, and cluster keeps yellow.\r\n\r\n[target]\r\nI wonder if you could give some hint on:\r\n- if there is recommended hardware for skywalking deployment. Like recommend 1C/2GB for single application tracing.\r\n- if there is any log or troubleshooting methods I could take for more deeply looking into the problem."}}
{"actor":{"display_login":"wu-sheng"}}