{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/beats/issues/878","repository_url":"https://api.github.com/repos/elastic/beats","labels_url":"https://api.github.com/repos/elastic/beats/issues/878/labels{/name}","comments_url":"https://api.github.com/repos/elastic/beats/issues/878/comments","events_url":"https://api.github.com/repos/elastic/beats/issues/878/events","html_url":"https://github.com/elastic/beats/issues/878","id":129440435,"number":878,"title":"Filebeat not resuming after EOF","user":{"login":"matejzero","id":4689252,"avatar_url":"https://avatars.githubusercontent.com/u/4689252?v=3","gravatar_id":"","url":"https://api.github.com/users/matejzero","html_url":"https://github.com/matejzero","followers_url":"https://api.github.com/users/matejzero/followers","following_url":"https://api.github.com/users/matejzero/following{/other_user}","gists_url":"https://api.github.com/users/matejzero/gists{/gist_id}","starred_url":"https://api.github.com/users/matejzero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matejzero/subscriptions","organizations_url":"https://api.github.com/users/matejzero/orgs","repos_url":"https://api.github.com/users/matejzero/repos","events_url":"https://api.github.com/users/matejzero/events{/privacy}","received_events_url":"https://api.github.com/users/matejzero/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/beats/labels/Filebeat","name":"Filebeat","color":"008000"},{"url":"https://api.github.com/repos/elastic/beats/labels/under%20investigation","name":"under investigation","color":"cc317c"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2016-01-28T12:42:22Z","updated_at":"2016-02-04T07:32:59Z","closed_at":"2016-02-01T09:58:04Z","body":"Hello,\r\n\r\nI'm having troubles with filebeat not resuming after EOF. My filebeat is sending logs to 2 logstash nodes in loadbalance mode. Every day at random time, it stops sending with EOF error in log file:\r\n```\r\n2016-01-27T17:15:01+01:00 INFO Events sent: 918\r\n2016-01-27T17:15:01+01:00 INFO Registry file updated. 491 states written.\r\n2016-01-27T17:15:06+01:00 INFO Events sent: 490\r\n2016-01-27T17:15:06+01:00 INFO Registry file updated. 491 states written.\r\n2016-01-27T17:15:18+01:00 INFO Error publishing events (retrying): EOF\r\n2016-01-27T17:15:19+01:00 INFO Events sent: 865\r\n2016-01-27T17:15:19+01:00 INFO Registry file updated. 491 states written.\r\n2016-01-28T00:00:05+01:00 INFO Harvester started for file: /logs/server1/httpd/20160128-httpd_access.log\r\n2016-01-28T00:00:05+01:00 INFO Harvester started for file: /logs/server2/radius/20160128-radius.log\r\n```\r\n\r\nIf I look at logstash logs, I see this\r\n```\r\n{:timestamp=>\"2016-01-27T16:04:26.772000+0100\", :message=>\"Beats input: the pipeline is blocked, temporary refusing new connection.\", :reconnect_backoff_sleep=>0.5, :level=>:warn}\r\n{:timestamp=>\"2016-01-27T17:15:18.720000+0100\", :message=>\"CircuitBreaker::rescuing exceptions\", :name=>\"Beats input\", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}\r\n{:timestamp=>\"2016-01-27T17:15:18.720000+0100\", :message=>\"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.\", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}\r\n{:timestamp=>\"2016-01-27T17:15:18.815000+0100\", :message=>\"Beats input: the pipeline is blocked, temporary refusing new connection.\", :reconnect_backoff_sleep=>0.5, :level=>:warn}\r\n{:timestamp=>\"2016-01-27T17:15:18.816000+0100\", :message=>\"CircuitBreaker::Open\", :name=>\"Beats input\", :level=>:warn}\r\n{:timestamp=>\"2016-01-27T17:15:18.817000+0100\", :message=>\"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.\", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::OpenBreaker, :level=>:warn}\r\n{:timestamp=>\"2016-01-27T17:15:19.316000+0100\", :message=>\"Beats input: the pipeline is blocked, temporary refusing new connection.\", :reconnect_backoff_sleep=>0.5, :level=>:warn}\r\n```\r\n\r\nPipeline blocked logs continue for another 20s and then they stop. Logstash is still reading logs from redis, but filebeat never resumes. The only way to get logs flowing again is to restart filebeat.\r\n\r\nShouldn't filebeat resume sending of logs? At least that is the idea a got on filebeat irc channel."},"comment":{"url":"https://api.github.com/repos/elastic/beats/issues/comments/179690781","html_url":"https://github.com/elastic/beats/issues/878#issuecomment-179690781","issue_url":"https://api.github.com/repos/elastic/beats/issues/878","id":179690781,"user":{"login":"matejzero","id":4689252,"avatar_url":"https://avatars.githubusercontent.com/u/4689252?v=3","gravatar_id":"","url":"https://api.github.com/users/matejzero","html_url":"https://github.com/matejzero","followers_url":"https://api.github.com/users/matejzero/followers","following_url":"https://api.github.com/users/matejzero/following{/other_user}","gists_url":"https://api.github.com/users/matejzero/gists{/gist_id}","starred_url":"https://api.github.com/users/matejzero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matejzero/subscriptions","organizations_url":"https://api.github.com/users/matejzero/orgs","repos_url":"https://api.github.com/users/matejzero/repos","events_url":"https://api.github.com/users/matejzero/events{/privacy}","received_events_url":"https://api.github.com/users/matejzero/received_events","type":"User","site_admin":false},"created_at":"2016-02-04T07:32:59Z","updated_at":"2016-02-04T07:32:59Z","body":"Hello,\r\n\r\nproblem occurred again today and yesterday.\r\n\r\n# filebeat -version\r\nfilebeat version 1.2.0-SNAPSHOT (amd64) (it's a release from Jan 28 2016)\r\n\r\nIn log files, all I see is:\r\n```\r\n2016-02-04T07:52:16+01:00 DBG  Try to publish 1024 events to logstash with window size 1\r\n2016-02-04T07:52:16+01:00 DBG  0 events out of 1024 events sent to logstash. Continue sending ...\r\n2016-02-04T07:52:16+01:00 DBG  Try to publish 1024 events to logstash with window size 1\r\n2016-02-04T07:52:16+01:00 DBG  0 events out of 1024 events sent to logstash. Continue sending ...\r\n```\r\n\r\nI did run it with httprof and here is the output of debug:\r\n```\r\n# curl http://localhost:6060/debug/pprof/goroutine?debug=1\r\ngoroutine profile: total 83\r\n1 @ 0x636ee8 0x636cc3 0x6325a4 0x67288e 0x672aa0 0x65117a 0x6529dd 0x65349e 0x650c3e 0x45e581\r\n#   0x636ee8    runtime/pprof.writeRuntimeProfile+0xb8  /usr/local/go/src/runtime/pprof/pprof.go:545\r\n#   0x636cc3    runtime/pprof.writeGoroutine+0x93   /usr/local/go/src/runtime/pprof/pprof.go:507\r\n#   0x6325a4    runtime/pprof.(*Profile).WriteTo+0xd4   /usr/local/go/src/runtime/pprof/pprof.go:236\r\n#   0x67288e    net/http/pprof.handler.ServeHTTP+0x37e  /usr/local/go/src/net/http/pprof/pprof.go:199\r\n#   0x672aa0    net/http/pprof.Index+0x200      /usr/local/go/src/net/http/pprof/pprof.go:211\r\n#   0x65117a    net/http.HandlerFunc.ServeHTTP+0x3a /usr/local/go/src/net/http/server.go:1422\r\n#   0x6529dd    net/http.(*ServeMux).ServeHTTP+0x17d    /usr/local/go/src/net/http/server.go:1699\r\n#   0x65349e    net/http.serverHandler.ServeHTTP+0x19e  /usr/local/go/src/net/http/server.go:1862\r\n#   0x650c3e    net/http.(*conn).serve+0xbee        /usr/local/go/src/net/http/server.go:1361\r\n\r\n1 @ 0x42d443 0x42d504 0x404082 0x403bcb 0x463642 0x401099 0x42d060 0x45e581\r\n#   0x463642    github.com/elastic/beats/libbeat/beat.Run+0xc2  /go/src/github.com/elastic/beats/libbeat/beat/beat.go:136\r\n#   0x401099    main.main+0x99                  /go/src/github.com/elastic/beats/filebeat/main.go:20\r\n#   0x42d060    runtime.main+0x2b0              /usr/local/go/src/runtime/proc.go:111\r\n\r\n1 @ 0x45e581\r\n\r\n1 @ 0x42d443 0x42d504 0x42d248 0x45e581\r\n#   0x42d443    runtime.gopark+0x163        /usr/local/go/src/runtime/proc.go:186\r\n#   0x42d504    runtime.goparkunlock+0x54   /usr/local/go/src/runtime/proc.go:191\r\n#   0x42d248    runtime.forcegchelper+0xb8  /usr/local/go/src/runtime/proc.go:152\r\n\r\n1 @ 0x42d2d4 0x41d149 0x45e581\r\n#   0x42d2d4    runtime.Gosched+0x14    /usr/local/go/src/runtime/proc.go:167\r\n#   0x41d149    runtime.bgsweep+0xd9    /usr/local/go/src/runtime/mgcsweep.go:56\r\n\r\n1 @ 0x42d443 0x42d504 0x414e6a 0x45e581\r\n#   0x42d443    runtime.gopark+0x163        /usr/local/go/src/runtime/proc.go:186\r\n#   0x42d504    runtime.goparkunlock+0x54   /usr/local/go/src/runtime/proc.go:191\r\n#   0x414e6a    runtime.runfinq+0xaa        /usr/local/go/src/runtime/mfinal.go:154\r\n\r\n1 @ 0x40e12e 0x440072 0x6314a8 0x45e581\r\n#   0x6314a8    os/signal.loop+0x18 /usr/local/go/src/os/signal/signal_unix.go:22\r\n\r\n24 @ 0x42d443 0x418557 0x45e581\r\n#   0x42d443    runtime.gopark+0x163        /usr/local/go/src/runtime/proc.go:186\r\n#   0x418557    runtime.gcBgMarkWorker+0xf7 /usr/local/go/src/runtime/mgc.go:1289\r\n\r\n1 @ 0x42d443 0x42d504 0x404082 0x403bcb 0x46032c 0x4647ab 0x4637c3 0x464c7c 0x45e581\r\n#   0x46032c    github.com/elastic/beats/filebeat/beat.(*Filebeat).Run+0x6dc    /go/src/github.com/elastic/beats/filebeat/beat/filebeat.go:99\r\n#   0x4647ab    github.com/elastic/beats/libbeat/beat.(*Beat).Run+0x33b     /go/src/github.com/elastic/beats/libbeat/beat/beat.go:263\r\n#   0x4637c3    github.com/elastic/beats/libbeat/beat.(*Beat).Start+0x113   /go/src/github.com/elastic/beats/libbeat/beat/beat.go:173\r\n#   0x464c7c    github.com/elastic/beats/libbeat/beat.Run.func1+0x3c        /go/src/github.com/elastic/beats/libbeat/beat/beat.go:120\r\n\r\n1 @ 0x42d443 0x43bbb4 0x43b112 0x6a8af6 0x45e581\r\n#   0x6a8af6    github.com/elastic/beats/libbeat/outputs/mode.(*LoadBalancerMode).start.func1+0x3a6 /go/src/github.com/elastic/beats/libbeat/outputs/mode/balance.go:164\r\n\r\n1 @ 0x42d443 0x42d504 0x448219 0x41a41e 0x40f215 0x40f8b2 0x448157 0x41a41e 0x40f215 0x40f8b2 0x448157 0x41a41e 0x40f215 0x40f8b2 0x448157 0x41a41e 0x40f215 0x40f8b2 0x448157 0x41a41e 0x40f215 0x40f8b2 0x448157 0x41a41e 0x40f215 0x40f8b2 0x448157 0x41a41e 0x40f215 0x40f8b2 0x448157 0x41a41e\r\n#   0x448219    time.Sleep+0xf9         /usr/local/go/src/runtime/time.go:59\r\n#   0x41a41e    runtime.gcAssistAlloc+0x23e /usr/local/go/src/runtime/mgcmark.go:295\r\n#   0x40f215    runtime.mallocgc+0x535      /usr/local/go/src/runtime/malloc.go:711\r\n#   0x40f8b2    runtime.newobject+0x42      /usr/local/go/src/runtime/malloc.go:760\r\n#   0x448157    time.Sleep+0x37         /usr/local/go/src/runtime/time.go:53\r\n#   0x41a41e    runtime.gcAssistAlloc+0x23e /usr/local/go/src/runtime/mgcmark.go:295\r\n#   0x40f215    runtime.mallocgc+0x535      /usr/local/go/src/runtime/malloc.go:711\r\n#   0x40f8b2    runtime.newobject+0x42      /usr/local/go/src/runtime/malloc.go:760\r\n#   0x448157    time.Sleep+0x37         /usr/local/go/src/runtime/time.go:53\r\n#   0x41a41e    runtime.gcAssistAlloc+0x23e /usr/local/go/src/runtime/mgcmark.go:295\r\n#   0x40f215    runtime.mallocgc+0x535      /usr/local/go/src/runtime/malloc.go:711\r\n#   0x40f8b2    runtime.newobject+0x42      /usr/local/go/src/runtime/malloc.go:760\r\n#   0x448157    time.Sleep+0x37         /usr/local/go/src/runtime/time.go:53\r\n#   0x41a41e    runtime.gcAssistAlloc+0x23e /usr/local/go/src/runtime/mgcmark.go:295\r\n#   0x40f215    runtime.mallocgc+0x535      /usr/local/go/src/runtime/malloc.go:711\r\n#   0x40f8b2    runtime.newobject+0x42      /usr/local/go/src/runtime/malloc.go:760\r\n#   0x448157    time.Sleep+0x37         /usr/local/go/src/runtime/time.go:53\r\n#   0x41a41e    runtime.gcAssistAlloc+0x23e /usr/local/go/src/runtime/mgcmark.go:295\r\n#   0x40f215    runtime.mallocgc+0x535      /usr/local/go/src/runtime/malloc.go:711\r\n#   0x40f8b2    runtime.newobject+0x42      /usr/local/go/src/runtime/malloc.go:760\r\n#   0x448157    time.Sleep+0x37         /usr/local/go/src/runtime/time.go:53\r\n#   0x41a41e    runtime.gcAssistAlloc+0x23e /usr/local/go/src/runtime/mgcmark.go:295\r\n#   0x40f215    runtime.mallocgc+0x535      /usr/local/go/src/runtime/malloc.go:711\r\n#   0x40f8b2    runtime.newobject+0x42      /usr/local/go/src/runtime/malloc.go:760\r\n#   0x448157    time.Sleep+0x37         /usr/local/go/src/runtime/time.go:53\r\n#   0x41a41e    runtime.gcAssistAlloc+0x23e /usr/local/go/src/runtime/mgcmark.go:295\r\n#   0x40f215    runtime.mallocgc+0x535      /usr/local/go/src/runtime/malloc.go:711\r\n#   0x40f8b2    runtime.newobject+0x42      /usr/local/go/src/runtime/malloc.go:760\r\n#   0x448157    time.Sleep+0x37         /usr/local/go/src/runtime/time.go:53\r\n#   0x41a41e    runtime.gcAssistAlloc+0x23e /usr/local/go/src/runtime/mgcmark.go:295\r\n\r\n2 @ 0x42d443 0x43bbb4 0x43b112 0x4b0409 0x45e581\r\n#   0x4b0409    github.com/elastic/beats/libbeat/publisher.(*messageWorker).run+0x449   /go/src/github.com/elastic/beats/libbeat/publisher/worker.go:61\r\n\r\n32 @ 0x42d443 0x42d504 0x4034e0 0x402c13 0x5456c3 0x45e581\r\n#   0x5456c3    github.com/elastic/beats/filebeat/harvester.(*Harvester).Harvest+0xce3  /go/src/github.com/elastic/beats/filebeat/harvester/log.go:178\r\n\r\n1 @ 0x40e12e 0x44887e 0x45e581\r\n#   0x40e12e    runtime.notetsleepg+0x4e    /usr/local/go/src/runtime/lock_futex.go:203\r\n#   0x44887e    runtime.timerproc+0xde      /usr/local/go/src/runtime/time.go:209\r\n\r\n1 @ 0x42d443 0x43bbb4 0x43b112 0x4ab2fa 0x45e581\r\n#   0x4ab2fa    github.com/elastic/beats/libbeat/publisher.(*bulkWorker).run+0x3ca  /go/src/github.com/elastic/beats/libbeat/publisher/bulk.go:57\r\n\r\n1 @ 0x42d443 0x42d504 0x41735b 0x45e581\r\n#   0x42d443    runtime.gopark+0x163        /usr/local/go/src/runtime/proc.go:186\r\n#   0x42d504    runtime.goparkunlock+0x54   /usr/local/go/src/runtime/proc.go:191\r\n#   0x41735b    runtime.backgroundgc+0x8b   /usr/local/go/src/runtime/mgc.go:900\r\n\r\n1 @ 0x42d443 0x42756e 0x426a30 0x584cca 0x584d36 0x5888bc 0x5a626d 0x654371 0x653773 0x653636 0x653ccf 0x4bd399 0x45e581\r\n#   0x426a30    net.runtime_pollWait+0x60                   /usr/local/go/src/runtime/netpoll.go:157\r\n#   0x584cca    net.(*pollDesc).Wait+0x3a                   /usr/local/go/src/net/fd_poll_runtime.go:73\r\n#   0x584d36    net.(*pollDesc).WaitRead+0x36                   /usr/local/go/src/net/fd_poll_runtime.go:78\r\n#   0x5888bc    net.(*netFD).accept+0x27c                   /usr/local/go/src/net/fd_unix.go:408\r\n#   0x5a626d    net.(*TCPListener).AcceptTCP+0x4d               /usr/local/go/src/net/tcpsock_posix.go:254\r\n#   0x654371    net/http.tcpKeepAliveListener.Accept+0x41           /usr/local/go/src/net/http/server.go:2135\r\n#   0x653773    net/http.(*Server).Serve+0xb3                   /usr/local/go/src/net/http/server.go:1887\r\n#   0x653636    net/http.(*Server).ListenAndServe+0x136             /usr/local/go/src/net/http/server.go:1877\r\n#   0x653ccf    net/http.ListenAndServe+0x8f                    /usr/local/go/src/net/http/server.go:1967\r\n#   0x4bd399    github.com/elastic/beats/libbeat/service.BeforeRun.func1+0x69   /go/src/github.com/elastic/beats/libbeat/service/service.go:72\r\n\r\n1 @ 0x42d443 0x43bbb4 0x43b112 0x452813 0x45e581\r\n#   0x42d443    runtime.gopark+0x163        /usr/local/go/src/runtime/proc.go:186\r\n#   0x43bbb4    runtime.selectgoImpl+0xa64  /usr/local/go/src/runtime/select.go:392\r\n#   0x43b112    runtime.selectgo+0x12       /usr/local/go/src/runtime/select.go:212\r\n#   0x452813    runtime.ensureSigM.func1+0x353  /usr/local/go/src/runtime/signal1_unix.go:227\r\n\r\n1 @ 0x42d443 0x42d504 0x404511 0x403bcb 0x4bd23a 0x45e581\r\n#   0x4bd23a    github.com/elastic/beats/libbeat/service.HandleSignals.func1+0x3a   /go/src/github.com/elastic/beats/libbeat/service/service.go:29\r\n\r\n1 @ 0x42d443 0x42d504 0x4034e0 0x402c13 0x461cc8 0x461908 0x461622 0x45e581\r\n#   0x461cc8    github.com/elastic/beats/filebeat/beat.(*Spooler).flush+0x128   /go/src/github.com/elastic/beats/filebeat/beat/spooler.go:148\r\n#   0x461908    github.com/elastic/beats/filebeat/beat.(*Spooler).queue+0x148   /go/src/github.com/elastic/beats/filebeat/beat/spooler.go:124\r\n#   0x461622    github.com/elastic/beats/filebeat/beat.(*Spooler).run+0x392 /go/src/github.com/elastic/beats/filebeat/beat/spooler.go:85\r\n\r\n1 @ 0x42d443 0x43bbb4 0x43b112 0x4892b4 0x45e581\r\n#   0x4892b4    github.com/elastic/beats/filebeat/crawler.(*Registrar).Run+0x4d4    /go/src/github.com/elastic/beats/filebeat/crawler/registrar.go:88\r\n\r\n6 @ 0x42d443 0x42d504 0x448219 0x484f3d 0x48afaa 0x483c8d 0x45e581\r\n#   0x448219    time.Sleep+0xf9                             /usr/local/go/src/runtime/time.go:59\r\n#   0x484f3d    github.com/elastic/beats/filebeat/crawler.ProspectorLog.Run+0x21d   /go/src/github.com/elastic/beats/filebeat/crawler/prospector_log.go:62\r\n#   0x48afaa    github.com/elastic/beats/filebeat/crawler.(*ProspectorLog).Run+0xaa <autogenerated>:6\r\n#   0x483c8d    github.com/elastic/beats/filebeat/crawler.(*Prospector).Run+0x1ad   /go/src/github.com/elastic/beats/filebeat/crawler/prospector.go:95\r\n\r\n1 @ 0x42d443 0x42d504 0x404511 0x403bcb 0x4afc11 0x4af8eb 0x4ac0a7 0x4621a1 0x45e581\r\n#   0x4afc11    github.com/elastic/beats/libbeat/publisher.(*syncPublisher).send+0x301      /go/src/github.com/elastic/beats/libbeat/publisher/sync.go:51\r\n#   0x4af8eb    github.com/elastic/beats/libbeat/publisher.(*syncPublisher).PublishEvents+0xbb  /go/src/github.com/elastic/beats/libbeat/publisher/sync.go:29\r\n#   0x4ac0a7    github.com/elastic/beats/libbeat/publisher.(*client).PublishEvents+0x197    /go/src/github.com/elastic/beats/libbeat/publisher/client.go:104\r\n#   0x4621a1    github.com/elastic/beats/filebeat/beat.(*syncLogPublisher).Start.func1+0x3c1    /go/src/github.com/elastic/beats/filebeat/beat/publish.go:103\r\n```\r\n\r\nI can telnet to logstash port without problem:\r\n```\r\n# telnet logstash2 5055\r\nTrying 193.2.18.183...\r\nConnected to logstash2.arnes.si.\r\n```\r\nIn logstash, I see the following errors around the time filebeat stopped sending logs:\r\n```\r\n:timestamp=>\"2016-02-03T11:15:45.642000+0100\", :message=>\"Beats input: the pipeline is blocked, temporary refusing new connection.\", :reconnect_backoff_sleep=>0.5, :level=>:warn}\r\n{:timestamp=>\"2016-02-03T13:15:20.553000+0100\", :message=>\"CircuitBreaker::rescuing exceptions\", :name=>\"Beats input\", :exception=>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=>:warn}\r\n{:timestamp=>\"2016-02-03T13:15:20.554000+0100\", :message=>\"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.\", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=>:warn}\r\n{:timestamp=>\"2016-02-03T13:15:20.605000+0100\", :message=>\"Beats input: the pipeline is blocked, temporary refusing new connection.\", :reconnect_backoff_sleep=>0.5, :level=>:warn}\r\n{:timestamp=>\"2016-02-03T13:15:20.606000+0100\", :message=>\"CircuitBreaker::Open\", :name=>\"Beats input\", :level=>:warn}\r\n{:timestamp=>\"2016-02-03T13:15:20.606000+0100\", :message=>\"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.\", :exception=>LogStash::Inputs::BeatsSupport::CircuitBreaker::OpenBreaker, :level=>:warn}\r\n{:timestamp=>\"2016-02-03T13:15:21.105000+0100\", :message=>\"Beats input: the pipeline is blocked, temporary refusing new connection.\", :reconnect_backoff_sleep=>0.5, :level=>:warn}\r\n{:timestamp=>\"2016-02-03T13:15:21.606000+0100\", :message=>\"Beats input: the pipeline is blocked, temporary refusing new connection.\", :reconnect_backoff_sleep=>0.5, :level=>:warn}\r\n```\r\nAccording to grafana metrics, logs stopped flowing to logstash at around 13:16:00. The last error message in logstash logs is at 13:50:50:\r\n```\r\n{:timestamp=>\"2016-02-03T13:15:50.136000+0100\", :message=>\"Beats input: the pipeline is blocked, temporary refusing new connection.\", :reconnect_backoff_sleep=>0.5, :level=>:warn}\r\n```\r\n\r\nLet me know if you need more information."}}