{"action":"opened","issue":{"url":"https://api.github.com/repos/kostya/eye/issues/165","repository_url":"https://api.github.com/repos/kostya/eye","labels_url":"https://api.github.com/repos/kostya/eye/issues/165/labels{/name}","comments_url":"https://api.github.com/repos/kostya/eye/issues/165/comments","events_url":"https://api.github.com/repos/kostya/eye/issues/165/events","html_url":"https://github.com/kostya/eye/issues/165","id":139499967,"number":165,"title":"Sidekiq reported as crashed, spamming processes","user":{"login":"milgner","id":160025,"avatar_url":"https://avatars.githubusercontent.com/u/160025?v=3","gravatar_id":"","url":"https://api.github.com/users/milgner","html_url":"https://github.com/milgner","followers_url":"https://api.github.com/users/milgner/followers","following_url":"https://api.github.com/users/milgner/following{/other_user}","gists_url":"https://api.github.com/users/milgner/gists{/gist_id}","starred_url":"https://api.github.com/users/milgner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/milgner/subscriptions","organizations_url":"https://api.github.com/users/milgner/orgs","repos_url":"https://api.github.com/users/milgner/repos","events_url":"https://api.github.com/users/milgner/events{/privacy}","received_events_url":"https://api.github.com/users/milgner/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-03-09T07:41:03Z","updated_at":"2016-03-09T07:41:03Z","closed_at":null,"body":"We have been using eye for a couple of months but twice in the last few weeks it started spamming Sidekiq instances when it somehow thought that the process had crashed.\r\n\r\nFrom the logs:\r\n\r\n```\r\n24.02.2016 06:52:32 WARN  -- [app:sidekiq] compare_identity: fail, pid_file: '24 Feb 06:51', process: '24 Feb 06:52' (sidekiq 3.5.4 app [3 of 20 busy])\r\n24.02.2016 06:52:32 INFO  -- [app:sidekiq] switch :crashed [:up => :down] (reason: crashed_by_identity)\r\n24.02.2016 06:52:32 INFO  -- [app:sidekiq] check:cpu(<100%) [0%] => OK\r\n24.02.2016 06:52:32 INFO  -- [app:sidekiq] check:memory(<400Mb) [182Mb, 202Mb] => OK\r\n24.02.2016 06:52:32 INFO  -- [app:sidekiq] schedule :check_crash (reason: crashed)\r\n24.02.2016 06:52:32 INFO  -- [app:sidekiq] delete pid_file: PIDFILE_PATH\r\n24.02.2016 06:52:32 INFO  -- [app:sidekiq] => check_crash  (reason: crashed)\r\n24.02.2016 06:52:32 WARN  -- [app:sidekiq] check crashed: process is down\r\n24.02.2016 06:52:32 INFO  -- [app:sidekiq] schedule :restore (reason: crashed)\r\n24.02.2016 06:52:32 INFO  -- [app:sidekiq] <= check_crash\r\n24.02.2016 06:52:32 INFO  -- [app:sidekiq] => restore  (reason: crashed)\r\n24.02.2016 06:52:32 INFO  -- [app:sidekiq] load_external_pid_file: pid_file not found\r\n24.02.2016 06:52:32 INFO  -- [app:sidekiq] switch :starting [:down => :starting] (reason: crashed)\r\n```\r\n\r\nOur configuration is pretty standard and it usually(tm) just works, too :)\r\n\r\n```\r\n  process 'sidekiq' do\r\n    start_command \"bin/bundle exec sidekiq -e #{RAILS_ENV} -C #{APP_DIR}/config/sidekiq.yml\"\r\n    pid_file \"#{APP_DIR}/tmp/pids/#{name}.pid\"\r\n    daemonize true\r\n    stop_signals [:USR1, 0, :TERM, 10.seconds, :KILL]\r\n    check :cpu, every: 60, below: 100, times: 5\r\n    check :memory, every: 30, below: 400.megabytes, times: 5\r\n  end\r\n```\r\n\r\nAnother thing I noticed is that when I ran `eye stop sidekiq`, it suddenly forgot about all the other processes it was monitoring: running `eye info` afterwards only showed:\r\n\r\n```\r\napp                    \r\n  sidekiq ......................... unmonitored (stop by user at 09 Mar 08:16)\r\n```\r\n\r\nLuckily, the other processes were still running, only it looks like Eye forgot about them.\r\n"}}