{"action":"created","issue":{"url":"https://api.github.com/repos/arturictus/sidekiq_alive/issues/10","repository_url":"https://api.github.com/repos/arturictus/sidekiq_alive","labels_url":"https://api.github.com/repos/arturictus/sidekiq_alive/issues/10/labels{/name}","comments_url":"https://api.github.com/repos/arturictus/sidekiq_alive/issues/10/comments","events_url":"https://api.github.com/repos/arturictus/sidekiq_alive/issues/10/events","html_url":"https://github.com/arturictus/sidekiq_alive/issues/10","id":378942544,"node_id":"MDU6SXNzdWUzNzg5NDI1NDQ=","number":10,"title":"Runaway checks hit redis at 2600 requests per second within 30 minutes of starting","user":{"login":"rosskevin","id":136564,"node_id":"MDQ6VXNlcjEzNjU2NA==","avatar_url":"https://avatars1.githubusercontent.com/u/136564?v=4","gravatar_id":"","url":"https://api.github.com/users/rosskevin","html_url":"https://github.com/rosskevin","followers_url":"https://api.github.com/users/rosskevin/followers","following_url":"https://api.github.com/users/rosskevin/following{/other_user}","gists_url":"https://api.github.com/users/rosskevin/gists{/gist_id}","starred_url":"https://api.github.com/users/rosskevin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rosskevin/subscriptions","organizations_url":"https://api.github.com/users/rosskevin/orgs","repos_url":"https://api.github.com/users/rosskevin/repos","events_url":"https://api.github.com/users/rosskevin/events{/privacy}","received_events_url":"https://api.github.com/users/rosskevin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-11-08T22:23:46Z","updated_at":"2018-11-09T22:56:34Z","closed_at":null,"author_association":"NONE","body":"I've just integrated this but found a single instance escalated within 30 minutes to 2600 requests **per second**.  This is a no load/new architecture and this is the only job running.\r\n\r\nI did notice an environment variable/config error on my part, but not sure it would cause this - I had set ttl to `config.time_to_live = 8000`.\r\n\r\n![redis_labs](https://user-images.githubusercontent.com/136564/48230981-5a908e00-e372-11e8-9293-05290f169b5f.png)\r\n\r\nI did happen to lose a kubernetes cluster node in the pool, so the sidekiq pod may have been restarted on another host.  Could this be the same as @fruwe is discussing in #7?\r\n\r\nCould this TTL 8000 cause it to spin out of control (wouldn't think so)?\r\n"},"comment":{"url":"https://api.github.com/repos/arturictus/sidekiq_alive/issues/comments/437521371","html_url":"https://github.com/arturictus/sidekiq_alive/issues/10#issuecomment-437521371","issue_url":"https://api.github.com/repos/arturictus/sidekiq_alive/issues/10","id":437521371,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNzUyMTM3MQ==","user":{"login":"rosskevin","id":136564,"node_id":"MDQ6VXNlcjEzNjU2NA==","avatar_url":"https://avatars1.githubusercontent.com/u/136564?v=4","gravatar_id":"","url":"https://api.github.com/users/rosskevin","html_url":"https://github.com/rosskevin","followers_url":"https://api.github.com/users/rosskevin/followers","following_url":"https://api.github.com/users/rosskevin/following{/other_user}","gists_url":"https://api.github.com/users/rosskevin/gists{/gist_id}","starred_url":"https://api.github.com/users/rosskevin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rosskevin/subscriptions","organizations_url":"https://api.github.com/users/rosskevin/orgs","repos_url":"https://api.github.com/users/rosskevin/repos","events_url":"https://api.github.com/users/rosskevin/events{/privacy}","received_events_url":"https://api.github.com/users/rosskevin/received_events","type":"User","site_admin":false},"created_at":"2018-11-09T22:56:34Z","updated_at":"2018-11-09T22:56:34Z","author_association":"NONE","body":"This has just happened again.  As I take down pods and deploy new ones it just runs away:\r\n\r\n```\r\n2018-11-09T22:54:08.709Z 11 TID-go5eoizov SidekiqAlive::Worker JID-13c5f2051107f52a25ef5f1e INFO: start\r\n2018-11-09T22:54:08.710Z 11 TID-go5eoj2d3 SidekiqAlive::Worker JID-29372509d901455f06733abb INFO: done: 0.005 sec\r\n2018-11-09T22:54:08.713Z 11 TID-go5eoizov SidekiqAlive::Worker JID-13c5f2051107f52a25ef5f1e INFO: done: 0.003 sec\r\n2018-11-09T22:54:08.713Z 11 TID-go5eoj1z3 SidekiqAlive::Worker JID-cad550f796b0ee11ea24af10 INFO: start\r\n2018-11-09T22:54:08.716Z 11 TID-go5eoj1z3 SidekiqAlive::Worker JID-cad550f796b0ee11ea24af10 INFO: done: 0.003 sec\r\n2018-11-09T22:54:08.716Z 11 TID-go5eoizef SidekiqAlive::Worker JID-cc66acbdebf274f111b40c58 INFO: start\r\n2018-11-09T22:54:08.719Z 11 TID-go5eoizef SidekiqAlive::Worker JID-cc66acbdebf274f111b40c58 INFO: done: 0.003 sec\r\n2018-11-09T22:54:08.719Z 11 TID-go5eoj4xr SidekiqAlive::Worker JID-31c532b2ec2d1aa57b37d0d2 INFO: start\r\n2018-11-09T22:54:08.725Z 11 TID-go5eoj4xr SidekiqAlive::Worker JID-31c532b2ec2d1aa57b37d0d2 INFO: done: 0.005 sec\r\n2018-11-09T22:54:08.725Z 11 TID-go5eoiytv SidekiqAlive::Worker JID-1c4f6526cadc2a67b52b5ce0 INFO: start\r\n2018-11-09T22:54:08.729Z 11 TID-go5eoiytv SidekiqAlive::Worker JID-1c4f6526cadc2a67b52b5ce0 INFO: done: 0.004 sec\r\n2018-11-09T22:54:08.730Z 11 TID-go5eoj0p3 SidekiqAlive::Worker JID-e7447ac5992ca48667b2aa0d INFO: start\r\n2018-11-09T22:54:08.735Z 11 TID-go5eoj0p3 SidekiqAlive::Worker JID-e7447ac5992ca48667b2aa0d INFO: done: 0.005 sec\r\n2018-11-09T22:54:08.735Z 11 TID-go5eoj1ar SidekiqAlive::Worker JID-48fa855268670a1c77584efe INFO: start\r\n2018-11-09T22:54:08.738Z 11 TID-go5eoj1ar SidekiqAlive::Worker JID-48fa855268670a1c77584efe INFO: done: 0.003 sec\r\n2018-11-09T22:54:08.739Z 11 TID-go5eoj0fn SidekiqAlive::Worker JID-dbb66602fac45881bec1ff59 INFO: start\r\n2018-11-09T22:54:08.743Z 11 TID-go5eoj3dn SidekiqAlive::Worker JID-be3aa1aa7ec2db3402005825 INFO: start\r\n2018-11-09T22:54:08.744Z 11 TID-go5eoj0fn SidekiqAlive::Worker JID-dbb66602fac45881bec1ff59 INFO: done: 0.005 sec\r\n2018-11-09T22:54:08.746Z 11 TID-go5eoj3dn SidekiqAlive::Worker JID-be3aa1aa7ec2db3402005825 INFO: done: 0.003 sec\r\n2018-11-09T22:54:08.746Z 11 TID-go5eoj2d3 SidekiqAlive::Worker JID-e053558a23400b40139cf321 INFO: start\r\n2018-11-09T22:54:08.750Z 11 TID-go5eoizov SidekiqAlive::Worker JID-2dec473f542077fcf88ea7d1 INFO: start\r\n2018-11-09T22:54:08.751Z 11 TID-go5eoj2d3 SidekiqAlive::Worker JID-e053558a23400b40139cf321 INFO: done: 0.004 sec\r\n2018-11-09T22:54:08.753Z 11 TID-go5eoizov SidekiqAlive::Worker JID-2dec473f542077fcf88ea7d1 INFO: done: 0.003 sec\r\n2018-11-09T22:54:08.754Z 11 TID-go5eoj1z3 SidekiqAlive::Worker JID-9abdc83f275f941e630dfa49 INFO: start\r\n2018-11-09T22:54:08.757Z 11 TID-go5eoj1z3 SidekiqAlive::Worker JID-9abdc83f275f941e630dfa49 INFO: done: 0.003 sec\r\n2018-11-09T22:54:08.757Z 11 TID-go5eoizef SidekiqAlive::Worker JID-3f5ed59a06e7e926c9c26e6d INFO: start\r\n2018-11-09T22:54:08.760Z 11 TID-go5eoj4xr SidekiqAlive::Worker JID-19e69736237e3f93faefc95e INFO: start\r\n2018-11-09T22:54:08.768Z 11 TID-go5eoizef SidekiqAlive::Worker JID-3f5ed59a06e7e926c9c26e6d INFO: done: 0.011 sec\r\n2018-11-09T22:54:08.771Z 11 TID-go5eoiytv SidekiqAlive::Worker JID-ad2c19842bb736bbaddd0406 INFO: start\r\n2018-11-09T22:54:08.771Z 11 TID-go5eoj4xr SidekiqAlive::Worker JID-19e69736237e3f93faefc95e INFO: done: 0.011 sec\r\n2018-11-09T22:54:08.773Z 11 TID-go5eoiytv SidekiqAlive::Worker JID-ad2c19842bb736bbaddd0406 INFO: done: 0.003 sec\r\n2018-11-09T22:54:08.774Z 11 TID-go5eoj0p3 SidekiqAlive::Worker JID-dffe4b1ae40923aefde8098d INFO: start\r\n2018-11-09T22:54:08.776Z 11 TID-go5eoj0p3 SidekiqAlive::Worker JID-dffe4b1ae40923aefde8098d INFO: done: 0.003 sec\r\n2018-11-09T22:54:08.777Z 11 TID-go5eoj1ar SidekiqAlive::Worker JID-eda2ac20cd709bf30792d59a INFO: start\r\n2018-11-09T22:54:08.779Z 11 TID-go5eoj0fn SidekiqAlive::Worker JID-50eaefb18564c5780ecc802c INFO: start\r\n2018-11-09T22:54:08.780Z 11 TID-go5eoj1ar SidekiqAlive::Worker JID-eda2ac20cd709bf30792d59a INFO: done: 0.003 sec\r\n2018-11-09T22:54:08.782Z 11 TID-go5eoj0fn SidekiqAlive::Worker JID-50eaefb18564c5780ecc802c INFO: done: 0.002 sec\r\n2018-11-09T22:54:08.782Z 11 TID-go5eoj3dn SidekiqAlive::Worker JID-e599764cdacd0fe43481b5cb INFO: start\r\n2018-11-09T22:54:08.785Z 11 TID-go5eoj2d3 SidekiqAlive::Worker JID-63e250a424575821374ad89c INFO: start\r\n2018-11-09T22:54:08.786Z 11 TID-go5eoj3dn SidekiqAlive::Worker JID-e599764cdacd0fe43481b5cb INFO: done: 0.003 sec\r\n2018-11-09T22:54:08.789Z 11 TID-go5eoj2d3 SidekiqAlive::Worker JID-63e250a424575821374ad89c INFO: done: 0.004 sec\r\n2018-11-09T22:54:08.789Z 11 TID-go5eoizov SidekiqAlive::Worker JID-ceca7482d9946899dc8c80f9 INFO: start\r\n2018-11-09T22:54:08.792Z 11 TID-go5eoizov SidekiqAlive::Worker JID-ceca7482d9946899dc8c80f9 INFO: done: 0.003 sec\r\n2018-11-09T22:54:08.793Z 11 TID-go5eoj1z3 SidekiqAlive::Worker JID-6d532f1c87cb3d6a5ac73391 INFO: start\r\n2018-11-09T22:54:08.796Z 11 TID-go5eoizef SidekiqAlive::Worker JID-368216db9b0db2bc3d8d073b INFO: start\r\n2018-11-09T22:54:08.796Z 11 TID-go5eoj1z3 SidekiqAlive::Worker JID-6d532f1c87cb3d6a5ac73391 INFO: done: 0.003 sec\r\n2018-11-09T22:54:08.815Z 11 TID-go5eoizef SidekiqAlive::Worker JID-368216db9b0db2bc3d8d073b INFO: done: 0.019 sec\r\n2018-11-09T22:54:08.818Z 11 TID-go5eoj4xr SidekiqAlive::Worker JID-b905c7eb0f0b8ab58613e5e3 INFO: start\r\n2018-11-09T22:54:08.822Z 11 TID-go5eoiytv SidekiqAlive::Worker JID-50699c6860cff70fa3384955 INFO: start\r\n2018-11-09T22:54:08.823Z 11 TID-go5eoj4xr SidekiqAlive::Worker JID-b905c7eb0f0b8ab58613e5e3 INFO: done: 0.005 sec\r\n2018-11-09T22:54:08.826Z 11 TID-go5eoj0p3 SidekiqAlive::Worker JID-df0acb5dae800558f88efbf1 INFO: start\r\n2018-11-09T22:54:08.826Z 11 TID-go5eoiytv SidekiqAlive::Worker JID-50699c6860cff70fa3384955 INFO: done: 0.004 sec\r\n2018-11-09T22:54:08.829Z 11 TID-go5eoj1ar SidekiqAlive::Worker JID-256f96301792de537316b1e5 INFO: start\r\n2018-11-09T22:54:08.829Z 11 TID-go5eoj0p3 SidekiqAlive::Worker JID-df0acb5dae800558f88efbf1 INFO: done: 0.004 sec\r\n```\r\n\r\n"}}
{"actor":{"display_login":"rosskevin"}}