{"action":"created","issue":{"url":"https://api.github.com/repos/mperham/sidekiq/issues/4029","repository_url":"https://api.github.com/repos/mperham/sidekiq","labels_url":"https://api.github.com/repos/mperham/sidekiq/issues/4029/labels{/name}","comments_url":"https://api.github.com/repos/mperham/sidekiq/issues/4029/comments","events_url":"https://api.github.com/repos/mperham/sidekiq/issues/4029/events","html_url":"https://github.com/mperham/sidekiq/issues/4029","id":385310424,"node_id":"MDU6SXNzdWUzODUzMTA0MjQ=","number":4029,"title":"ThreadError: can't alloc thread","user":{"login":"benoittgt","id":8417720,"node_id":"MDQ6VXNlcjg0MTc3MjA=","avatar_url":"https://avatars3.githubusercontent.com/u/8417720?v=4","gravatar_id":"","url":"https://api.github.com/users/benoittgt","html_url":"https://github.com/benoittgt","followers_url":"https://api.github.com/users/benoittgt/followers","following_url":"https://api.github.com/users/benoittgt/following{/other_user}","gists_url":"https://api.github.com/users/benoittgt/gists{/gist_id}","starred_url":"https://api.github.com/users/benoittgt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benoittgt/subscriptions","organizations_url":"https://api.github.com/users/benoittgt/orgs","repos_url":"https://api.github.com/users/benoittgt/repos","events_url":"https://api.github.com/users/benoittgt/events{/privacy}","received_events_url":"https://api.github.com/users/benoittgt/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-11-28T15:05:04Z","updated_at":"2018-11-28T20:53:58Z","closed_at":null,"author_association":"NONE","body":"Ruby version:\r\n- sidekiq-pro (4.0.2)\r\n- sidekiq (5.1.3)\r\n\r\n```\r\n13:29:02.713804+00:00 app worker_notifications.1 - - 4 TID-1u06zw SendNotificationsWorker JID-754eb57d86009b5a6baee320 INFO: SidekiqMemorySampling queue=notifications worker=SendNotificationsWorker jid=754eb57d86009b5a6baee320 \r\n13:29:02.719922+00:00 app worker_notifications.1 - - 4 TID-1u06no SendNotificationsWorker JID-fdd856cc57c54585cd4aab19 INFO: SidekiqMemorySampling queue=notifications worker=SendNotificationsWorker jid=fdd856cc57c54585cd4aab19 \r\n13:29:02.727448+00:00 app worker_notifications.1 - - 4 TID-1u06zw WARN: ERR only (P)SUBSCRIBE / (P)UNSUBSCRIBE / PING / QUIT allowed in this context\r\n13:29:02.727736+00:00 app worker_notifications.1 - - 4 TID-1u06zw WARN: {\"context\":\"processor\"}\r\n13:29:02.727917+00:00 app worker_notifications.1 - - 4 TID-1u06zw WARN: ThreadError: can't alloc thread\r\n13:29:02.728000+00:00 app worker_notifications.1 - - 4 TID-1u06zw WARN: /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:23:in `new'\r\n13:29:02.728003+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:23:in `safe_thread'\r\n13:29:02.728004+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:65:in `start'\r\n13:29:02.728005+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/manager.rb:96:in `block in processor_died'\r\n13:29:02.728007+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/manager.rb:91:in `synchronize'\r\n13:29:02.728008+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/manager.rb:91:in `processor_died'\r\n13:29:02.728009+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:79:in `rescue in run'\r\n13:29:02.728010+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:70:in `run'\r\n13:29:02.728012+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:16:in `watchdog'\r\n13:29:02.728013+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:25:in `block in safe_thread'\r\n13:29:02.728151+00:00 app worker_notifications.1 - - W, [2018-11-28T13:29:02.728053 #4] WARN -- : **Airbrake: falling back to sync delivery because there are no running async workers\r\n13:29:02.728239+00:00 app worker_notifications.1 - - 4 TID-1u06zw ERROR: !!! ERROR HANDLER THREW AN ERROR !!!\r\n13:29:02.728289+00:00 app worker_notifications.1 - - 4 TID-1u06zw ERROR: attempted to build can't alloc thread with closed Airbrake instance\r\n13:29:02.728369+00:00 app worker_notifications.1 - - 4 TID-1u06zw ERROR: /app/vendor/bundle/ruby/2.5.0/gems/airbrake-ruby-2.10.0/lib/airbrake-ruby/notifier.rb:59:in `build_notice'\r\n13:29:02.728371+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/airbrake-ruby-2.10.0/lib/airbrake-ruby/notifier.rb:117:in `send_notice'\r\n13:29:02.728372+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/airbrake-ruby-2.10.0/lib/airbrake-ruby/notifier.rb:43:in `notify'\r\n13:29:02.728374+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/airbrake-ruby-2.10.0/lib/airbrake-ruby.rb:199:in `notify'\r\n13:29:02.728375+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/airbrake-7.3.3/lib/airbrake/sidekiq.rb:19:in `notify_airbrake'\r\n13:29:02.728377+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/exception_handler.rb:20:in `call'\r\n13:29:02.728378+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/exception_handler.rb:20:in `block in handle_exception'\r\n13:29:02.728379+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/exception_handler.rb:18:in `each'\r\n13:29:02.728380+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/exception_handler.rb:18:in `handle_exception'\r\n13:29:02.728382+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:18:in `rescue in watchdog'\r\n13:29:02.728383+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:15:in `watchdog'\r\n13:29:02.728384+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:25:in `block in safe_thread'\r\n13:29:02.729120+00:00 app worker_notifications.1 - - #<Thread:0x000000000d374ad0@/app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:23 aborting> terminated with exception (report_on_exception is true):\r\n28 Nov 2018 14:29:02.991225 <190>1 2018-11-28T13:29:02.729212+00:00 app worker_notifications.1 - - /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:23:in `new': can't alloc thread (ThreadError)\r\n\tfrom /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:23:in `safe_thread'\r\n\tfrom /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:65:in `start'\r\n\tfrom /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/manager.rb:96:in `block in processor_died'\r\n\tfrom /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/manager.rb:91:in `synchronize'\r\n\tfrom /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/manager.rb:91:in `processor_died'\r\n\tfrom /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:79:in `rescue in run'\r\n\tfrom /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/processor.rb:70:in `run'\r\n\tfrom /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:16:in `watchdog'\r\nfrom /app/vendor/bundle/ruby/2.5.0/gems/sidekiq-5.1.3/lib/sidekiq/util.rb:25:in `block in safe_thread'\r\n```\r\n\r\nI found this comment https://github.com/mperham/sidekiq/issues/2732#issuecomment-166706814 but we don't have complicated custom initializer with Redis. For the moment I am not able to find where this could happen. When we go on `/app/vendor/bundle/ruby/2.5.0/gems/airbrake-ruby-2.10.0/lib/airbrake-ruby/notifier.rb:59:in `build_notice'` https://github.com/airbrake/airbrake-ruby/blob/v2.10.0/lib/airbrake-ruby/notifier.rb#L59 we only raise an error.\r\n\r\nWe don't have a peak of threads or memory in NewRelic. \r\n![screenshot 2018-11-28 at 15 58 40](https://user-images.githubusercontent.com/8417720/49160299-c5f0ce80-f326-11e8-974c-8e648400f251.png)\r\n\r\n![screenshot 2018-11-28 at 16 04 13](https://user-images.githubusercontent.com/8417720/49160562-5202f600-f327-11e8-88f0-36cbd42e4b58.png)\r\n\r\nWe are on a standard-2x Heroku dyno with 512 threads available.\r\n\r\n```ruby\r\nrequire 'sidekiq'\r\n\r\nSidekiq.configure_client do |config|\r\n  config.redis = { url: ***, size: (ENV['SIDEKIQ_REDIS_CLIENT_SIZE'] || 1) }\r\n  config.error_handlers << proc { |ex, ctx_hash| Airbrake.notify(ex, ctx_hash) }\r\nend\r\n\r\nSidekiq::Client.reliable_push! unless Rails.env.test?\r\n\r\nSidekiq::Extensions.enable_delay!\r\n\r\nSidekiq.logger.level = Logger::WARN if Rails.env.test?\r\n\r\nSidekiq.configure_server do |config|\r\n  config.super_fetch!\r\n  config.reliable_scheduler!\r\n  config.server_middleware do |chain|\r\n    chain.add Sidekiq::MemorySamplingMiddleware\r\n    chain.add Sidekiq::RetryMonitoringMiddleware\r\n    chain.add Sidekiq::ExceptionMonitoringMiddleware\r\n  end\r\nend\r\n```\r\n\r\nDo you have any ideas?"},"comment":{"url":"https://api.github.com/repos/mperham/sidekiq/issues/comments/442600809","html_url":"https://github.com/mperham/sidekiq/issues/4029#issuecomment-442600809","issue_url":"https://api.github.com/repos/mperham/sidekiq/issues/4029","id":442600809,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MjYwMDgwOQ==","user":{"login":"benoittgt","id":8417720,"node_id":"MDQ6VXNlcjg0MTc3MjA=","avatar_url":"https://avatars3.githubusercontent.com/u/8417720?v=4","gravatar_id":"","url":"https://api.github.com/users/benoittgt","html_url":"https://github.com/benoittgt","followers_url":"https://api.github.com/users/benoittgt/followers","following_url":"https://api.github.com/users/benoittgt/following{/other_user}","gists_url":"https://api.github.com/users/benoittgt/gists{/gist_id}","starred_url":"https://api.github.com/users/benoittgt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benoittgt/subscriptions","organizations_url":"https://api.github.com/users/benoittgt/orgs","repos_url":"https://api.github.com/users/benoittgt/repos","events_url":"https://api.github.com/users/benoittgt/events{/privacy}","received_events_url":"https://api.github.com/users/benoittgt/received_events","type":"User","site_admin":false},"created_at":"2018-11-28T20:53:58Z","updated_at":"2018-11-28T20:53:58Z","author_association":"NONE","body":"Thanks Mike for your answer.\r\n\r\nWe are trying to reproduce the issue in a small app but this is not easy. We will share it if we can.\r\n\r\nFor Redis gem (4.0.2) and Redis instance (on Redis Cloud) 4.0.10.\r\n"}}
{"actor":{"display_login":"benoittgt"}}