{"action":"created","issue":{"url":"https://api.github.com/repos/BroadSoft-Xtended/BroadWorks-Dashboards-and-Discovery/issues/47","repository_url":"https://api.github.com/repos/BroadSoft-Xtended/BroadWorks-Dashboards-and-Discovery","labels_url":"https://api.github.com/repos/BroadSoft-Xtended/BroadWorks-Dashboards-and-Discovery/issues/47/labels{/name}","comments_url":"https://api.github.com/repos/BroadSoft-Xtended/BroadWorks-Dashboards-and-Discovery/issues/47/comments","events_url":"https://api.github.com/repos/BroadSoft-Xtended/BroadWorks-Dashboards-and-Discovery/issues/47/events","html_url":"https://github.com/BroadSoft-Xtended/BroadWorks-Dashboards-and-Discovery/issues/47","id":327502111,"node_id":"MDU6SXNzdWUzMjc1MDIxMTE=","number":47,"title":"Lastest logsender/receiver 1020","user":{"login":"kdotson3263k","id":36305072,"node_id":"MDQ6VXNlcjM2MzA1MDcy","avatar_url":"https://avatars0.githubusercontent.com/u/36305072?v=4","gravatar_id":"","url":"https://api.github.com/users/kdotson3263k","html_url":"https://github.com/kdotson3263k","followers_url":"https://api.github.com/users/kdotson3263k/followers","following_url":"https://api.github.com/users/kdotson3263k/following{/other_user}","gists_url":"https://api.github.com/users/kdotson3263k/gists{/gist_id}","starred_url":"https://api.github.com/users/kdotson3263k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kdotson3263k/subscriptions","organizations_url":"https://api.github.com/users/kdotson3263k/orgs","repos_url":"https://api.github.com/users/kdotson3263k/repos","events_url":"https://api.github.com/users/kdotson3263k/events{/privacy}","received_events_url":"https://api.github.com/users/kdotson3263k/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":22,"created_at":"2018-05-29T21:57:17Z","updated_at":"2018-06-19T19:37:36Z","closed_at":null,"author_association":"NONE","body":"Hello,\r\nI have recently upgraded to logsender/receiver 1020.  I also upgraded to elasticsearch 6.24 and added a kafka cluster.\r\n\r\nI have six hot data servers in my cluster and each has a kafka, bwlogreceiver and elasticsearch instance. \r\n\r\nI have two AS and six XSP's currently logging to the cluster, all appear to be doing well except my more heavily utilized AS server where the XSLogs consistantly fall behind during peak traffic.  I have always had trouble with this server keeping up with the logging but most of the troubles were cleared up by adding additional servers, a load balancer and removing all receiving nodes outside of the datacenter with high latency.  \r\n\r\nAfter reading through another thread I also update the bwlogreceiver with the following:\r\nnumsimultaneousrequests:8\r\nuseindexthread:true\r\n\r\nAlthough I believe this may be a bottle neck on the AS server, unfortunately, logging seems to be falling behind quicker since the changes (upgrade,kafka, settings) and I am not sure where to look next for a bottle neck.\r\n\r\n**logsender.props  (Note: I am aware the ports are reversed, I did this so that I could keep my LB settings the same)**\r\nenabled:true\r\nservertype:AS\r\nreceiverserver:XXXXXX\r\nreceiverport:9092\r\nkafkaserver:XXXXXXXX\r\nkafkaport:9072\r\ntopicname:av1\r\ntopicgroup:av1\r\nusekafka:true\r\nsenderlogqueuesize:1000\r\nnumbersenderwritethreads:8\r\nzipblocksize:8192\r\njvmheapsize:1024m\r\nprocessmonitormaxthreads:100\r\nprocessmonitormaxfiles:100\r\nlogsenderdebugging:true\r\n\r\n\r\nNOTE:  The broadsoft XSLog size is set to 31 MB and they are rotated ~3.5 seconds during peak.  You will see below that the bwlogsender is report >6 seconds to process. \r\n\r\n**Here are some logs from the receiver:**\r\n\r\n2018-05-29_21:41:01.755 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Starting\r\n2018-05-29_21:41:01.800 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Number of Open File Descriptors -> 69\r\n2018-05-29_21:41:01.800 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Using -> 69 of -> 4096 allowed.\r\n2018-05-29_21:41:01.800 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Number of Running Threads -> 31\r\n2018-05-29_21:41:01.800 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Finished\r\n2018-05-29_21:41:05.861 [I/O dispatcher 2] INFO  c.b.e.ElasticLogIndexerThreadownerImpl - onSuccess response from ES  numreqs= 0 indexcount 21029 - took 890\r\n2018-05-29_21:41:11.340 [I/O dispatcher 2] INFO  c.b.e.ElasticLogIndexerThreadownerImpl - onSuccess response from ES  numreqs= 0 indexcount 21467 - took 1312\r\n2018-05-29_21:41:15.802 [I/O dispatcher 2] INFO  c.b.e.ElasticLogIndexerThreadownerImpl - onSuccess response from ES  numreqs= 0 indexcount 19700 - took 751\r\n2018-05-29_21:41:21.112 [I/O dispatcher 2] INFO  c.b.e.ElasticLogIndexerThreadownerImpl - onSuccess response from ES  numreqs= 0 indexcount 16357 - took 1044\r\n2018-05-29_21:41:26.021 [I/O dispatcher 2] INFO  c.b.e.ElasticLogIndexerThreadownerImpl - onSuccess response from ES  numreqs= 0 indexcount 13551 - took 939\r\n2018-05-29_21:41:30.804 [I/O dispatcher 2] INFO  c.b.e.ElasticLogIndexerThreadownerImpl - onSuccess response from ES  numreqs= 0 indexcount 18065 - took 709\r\n2018-05-29_21:41:35.984 [I/O dispatcher 2] INFO  c.b.e.ElasticLogIndexerThreadownerImpl - onSuccess response from ES  numreqs= 0 indexcount 20781 - took 815\r\n2018-05-29_21:41:41.391 [I/O dispatcher 2] INFO  c.b.e.ElasticLogIndexerThreadownerImpl - onSuccess response from ES  numreqs= 0 indexcount 25000 - took 1299\r\n2018-05-29_21:41:45.855 [I/O dispatcher 2] INFO  c.b.e.ElasticLogIndexerThreadownerImpl - onSuccess response from ES  numreqs= 0 indexcount 20016 - took 740\r\n2018-05-29_21:41:51.010 [I/O dispatcher 2] INFO  c.b.e.ElasticLogIndexerThreadownerImpl - onSuccess response from ES  numreqs= 0 indexcount 16194 - took 877\r\n2018-05-29_21:41:56.149 [I/O dispatcher 2] INFO  c.b.e.ElasticLogIndexerThreadownerImpl - onSuccess response from ES  numreqs= 0 indexcount 16699 - took 995\r\n2018-05-29_21:42:01.800 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Starting\r\n2018-05-29_21:42:01.845 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Number of Open File Descriptors -> 69\r\n2018-05-29_21:42:01.845 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Using -> 69 of -> 4096 allowed.\r\n2018-05-29_21:42:01.845 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Number of Running Threads -> 30\r\n2018-05-29_21:42:01.845 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Finished\r\n2018-05-29_21:42:01.953 [I/O dispatcher 2] INFO  c.b.e.ElasticLogIndexerThreadownerImpl - onSuccess response from ES  numreqs= 0 indexcount 19024 - took 1785\r\n2018-05-29_21:42:06.055 [I/O dispatcher 2] INFO  c.b.e.ElasticLogIndexerThreadownerImpl - onSuccess response from ES  numreqs= 0 indexcount 22253 - took 826\r\n2018-05-29_21:42:11.119 [I/O dispatcher 2] INFO  c.b.e.ElasticLogIndexerThreadownerImpl - onSuccess response from ES  numreqs= 0 indexcount 22191 - took 853\r\n2018-05-29_21:42:16.072 [I/O dispatcher 2] INFO  c.b.e.ElasticLogIndexerThreadownerImpl - onSuccess response from ES  numreqs= 0 indexcount 21982 - took 764\r\n\r\n\r\n\r\n**Here are some logs from the AS in question:**\r\n\r\n2018-05-29_21:40:53.596 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Starting\r\n2018-05-29_21:40:53.950 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Number of Open File Descriptors -> 47\r\n2018-05-29_21:40:53.951 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Using -> 47 of -> 64000 allowed.\r\n2018-05-29_21:40:53.952 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Number of Running Threads -> 38\r\n2018-05-29_21:40:53.952 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Finished\r\n2018-05-29_21:40:54.403 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File XSLog2018.05.29-21.23.22.txt processed in 6\r\n2018-05-29_21:40:54.403 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File => XSLog2018.05.29-21.23.22.txt processed <100973> logs and processed <671861> lines.\r\n2018-05-29_21:40:54.403 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Starting file => XSLog2018.05.29-21.23.28.txt\r\n2018-05-29_21:40:54.404 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Creating new Tailer for XSLog2018.05.29-21.23.28.txt\r\n2018-05-29_21:41:00.409 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File XSLog2018.05.29-21.23.28.txt processed in 6\r\n2018-05-29_21:41:00.409 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File => XSLog2018.05.29-21.23.28.txt processed <101746> logs and processed <671777> lines.\r\n2018-05-29_21:41:00.409 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Starting file => XSLog2018.05.29-21.23.33.txt\r\n2018-05-29_21:41:00.409 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Creating new Tailer for XSLog2018.05.29-21.23.33.txt\r\n2018-05-29_21:41:06.417 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File XSLog2018.05.29-21.23.33.txt processed in 6\r\n2018-05-29_21:41:06.417 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File => XSLog2018.05.29-21.23.33.txt processed <100770> logs and processed <671620> lines.\r\n2018-05-29_21:41:06.417 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Starting file => XSLog2018.05.29-21.23.38.txt\r\n2018-05-29_21:41:06.418 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Creating new Tailer for XSLog2018.05.29-21.23.38.txt\r\n2018-05-29_21:41:12.421 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File XSLog2018.05.29-21.23.38.txt processed in 6\r\n2018-05-29_21:41:12.422 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File => XSLog2018.05.29-21.23.38.txt processed <102473> logs and processed <670689> lines.\r\n2018-05-29_21:41:12.422 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Starting file => XSLog2018.05.29-21.23.43.txt\r\n2018-05-29_21:41:12.422 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Creating new Tailer for XSLog2018.05.29-21.23.43.txt\r\n2018-05-29_21:41:18.426 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File XSLog2018.05.29-21.23.43.txt processed in 6\r\n2018-05-29_21:41:18.427 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File => XSLog2018.05.29-21.23.43.txt processed <101822> logs and processed <673091> lines.\r\n2018-05-29_21:41:18.427 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Starting file => XSLog2018.05.29-21.23.48.txt\r\n2018-05-29_21:41:18.427 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Creating new Tailer for XSLog2018.05.29-21.23.48.txt\r\n2018-05-29_21:41:24.431 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File XSLog2018.05.29-21.23.48.txt processed in 6\r\n2018-05-29_21:41:24.432 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File => XSLog2018.05.29-21.23.48.txt processed <101353> logs and processed <672166> lines.\r\n2018-05-29_21:41:24.432 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Starting file => XSLog2018.05.29-21.23.53.txt\r\n2018-05-29_21:41:24.432 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Creating new Tailer for XSLog2018.05.29-21.23.53.txt\r\n2018-05-29_21:41:30.437 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File XSLog2018.05.29-21.23.53.txt processed in 6\r\n2018-05-29_21:41:30.438 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File => XSLog2018.05.29-21.23.53.txt processed <102074> logs and processed <672274> lines.\r\n2018-05-29_21:41:30.438 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Starting file => XSLog2018.05.29-21.23.59.txt\r\n2018-05-29_21:41:30.438 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Creating new Tailer for XSLog2018.05.29-21.23.59.txt\r\n2018-05-29_21:41:36.443 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File XSLog2018.05.29-21.23.59.txt processed in 6\r\n2018-05-29_21:41:36.443 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File => XSLog2018.05.29-21.23.59.txt processed <101738> logs and processed <669930> lines.\r\n2018-05-29_21:41:36.443 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Starting file => XSLog2018.05.29-21.24.04.txt\r\n2018-05-29_21:41:36.444 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Creating new Tailer for XSLog2018.05.29-21.24.04.txt\r\n2018-05-29_21:41:42.449 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File XSLog2018.05.29-21.24.04.txt processed in 6\r\n2018-05-29_21:41:42.595 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File => XSLog2018.05.29-21.24.04.txt processed <101341> logs and processed <668683> lines.\r\n2018-05-29_21:41:42.596 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Starting file => XSLog2018.05.29-21.24.09.txt\r\n2018-05-29_21:41:42.622 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Creating new Tailer for XSLog2018.05.29-21.24.09.txt\r\n2018-05-29_21:41:48.628 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File XSLog2018.05.29-21.24.09.txt processed in 6\r\n2018-05-29_21:41:48.628 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File => XSLog2018.05.29-21.24.09.txt processed <100695> logs and processed <671904> lines.\r\n2018-05-29_21:41:48.628 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Starting file => XSLog2018.05.29-21.24.13.txt\r\n2018-05-29_21:41:48.629 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Creating new Tailer for XSLog2018.05.29-21.24.13.txt\r\n2018-05-29_21:41:48.993 [TailRunner => PSLog] INFO  com.broadsoft.zipsender.TailerRunner - File PSLog2018.05.29-21.37.09.txt processed in 276\r\n2018-05-29_21:41:48.994 [TailRunner => PSLog] INFO  com.broadsoft.zipsender.TailerRunner - File => PSLog2018.05.29-21.37.09.txt processed <27610> logs and processed <172782> lines.\r\n2018-05-29_21:41:48.994 [TailRunner => PSLog] INFO  com.broadsoft.zipsender.TailerRunner - Starting file => PSLog2018.05.29-21.41.44.txt\r\n2018-05-29_21:41:48.994 [TailRunner => PSLog] INFO  com.broadsoft.zipsender.TailerRunner - Creating new Tailer for PSLog2018.05.29-21.41.44.txt\r\n2018-05-29_21:41:53.952 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Starting\r\n2018-05-29_21:41:53.962 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Number of Open File Descriptors -> 47\r\n2018-05-29_21:41:53.962 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Using -> 47 of -> 64000 allowed.\r\n2018-05-29_21:41:53.964 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Number of Running Threads -> 38\r\n2018-05-29_21:41:53.964 [ProcessMonitor] INFO  c.b.eslogzipshared.ProcessMonitor - ProcessMonitor: Finished\r\n2018-05-29_21:41:54.634 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File XSLog2018.05.29-21.24.13.txt processed in 6\r\n2018-05-29_21:41:54.634 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File => XSLog2018.05.29-21.24.13.txt processed <101781> logs and processed <675330> lines.\r\n2018-05-29_21:41:54.634 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Starting file => XSLog2018.05.29-21.24.18.txt\r\n2018-05-29_21:41:54.635 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Creating new Tailer for XSLog2018.05.29-21.24.18.txt\r\n2018-05-29_21:42:00.642 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File XSLog2018.05.29-21.24.18.txt processed in 6\r\n2018-05-29_21:42:00.642 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - File => XSLog2018.05.29-21.24.18.txt processed <101782> logs and processed <670794> lines.\r\n2018-05-29_21:42:00.643 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Starting file => XSLog2018.05.29-21.24.23.txt\r\n2018-05-29_21:42:00.643 [TailRunner => XSLog] INFO  com.broadsoft.zipsender.TailerRunner - Creating new Tailer for XSLog2018.05.29-21.24.23.txt\r\n"},"comment":{"url":"https://api.github.com/repos/BroadSoft-Xtended/BroadWorks-Dashboards-and-Discovery/issues/comments/398519651","html_url":"https://github.com/BroadSoft-Xtended/BroadWorks-Dashboards-and-Discovery/issues/47#issuecomment-398519651","issue_url":"https://api.github.com/repos/BroadSoft-Xtended/BroadWorks-Dashboards-and-Discovery/issues/47","id":398519651,"node_id":"MDEyOklzc3VlQ29tbWVudDM5ODUxOTY1MQ==","user":{"login":"kdotson3263k","id":36305072,"node_id":"MDQ6VXNlcjM2MzA1MDcy","avatar_url":"https://avatars0.githubusercontent.com/u/36305072?v=4","gravatar_id":"","url":"https://api.github.com/users/kdotson3263k","html_url":"https://github.com/kdotson3263k","followers_url":"https://api.github.com/users/kdotson3263k/followers","following_url":"https://api.github.com/users/kdotson3263k/following{/other_user}","gists_url":"https://api.github.com/users/kdotson3263k/gists{/gist_id}","starred_url":"https://api.github.com/users/kdotson3263k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kdotson3263k/subscriptions","organizations_url":"https://api.github.com/users/kdotson3263k/orgs","repos_url":"https://api.github.com/users/kdotson3263k/repos","events_url":"https://api.github.com/users/kdotson3263k/events{/privacy}","received_events_url":"https://api.github.com/users/kdotson3263k/received_events","type":"User","site_admin":false},"created_at":"2018-06-19T19:37:36Z","updated_at":"2018-06-19T19:37:36Z","author_association":"NONE","body":"Thanks so much, it looks like I will implement tonight and provide an update sometime tomorrow afternoon."}}
{"actor":{"display_login":"kdotson3263k"}}