{"action":"created","issue":{"url":"https://api.github.com/repos/quantum-elixir/quantum-core/issues/368","repository_url":"https://api.github.com/repos/quantum-elixir/quantum-core","labels_url":"https://api.github.com/repos/quantum-elixir/quantum-core/issues/368/labels{/name}","comments_url":"https://api.github.com/repos/quantum-elixir/quantum-core/issues/368/comments","events_url":"https://api.github.com/repos/quantum-elixir/quantum-core/issues/368/events","html_url":"https://github.com/quantum-elixir/quantum-core/issues/368","id":353903679,"node_id":"MDU6SXNzdWUzNTM5MDM2Nzk=","number":368,"title":"Scheduler dies during scaling events in container orchestration environments","user":{"login":"doughsay","id":221693,"node_id":"MDQ6VXNlcjIyMTY5Mw==","avatar_url":"https://avatars0.githubusercontent.com/u/221693?v=4","gravatar_id":"","url":"https://api.github.com/users/doughsay","html_url":"https://github.com/doughsay","followers_url":"https://api.github.com/users/doughsay/followers","following_url":"https://api.github.com/users/doughsay/following{/other_user}","gists_url":"https://api.github.com/users/doughsay/gists{/gist_id}","starred_url":"https://api.github.com/users/doughsay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/doughsay/subscriptions","organizations_url":"https://api.github.com/users/doughsay/orgs","repos_url":"https://api.github.com/users/doughsay/repos","events_url":"https://api.github.com/users/doughsay/events{/privacy}","received_events_url":"https://api.github.com/users/doughsay/received_events","type":"User","site_admin":false},"labels":[{"id":204725630,"node_id":"MDU6TGFiZWwyMDQ3MjU2MzA=","url":"https://api.github.com/repos/quantum-elixir/quantum-core/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":{"login":"maennchen","id":333918,"node_id":"MDQ6VXNlcjMzMzkxOA==","avatar_url":"https://avatars1.githubusercontent.com/u/333918?v=4","gravatar_id":"","url":"https://api.github.com/users/maennchen","html_url":"https://github.com/maennchen","followers_url":"https://api.github.com/users/maennchen/followers","following_url":"https://api.github.com/users/maennchen/following{/other_user}","gists_url":"https://api.github.com/users/maennchen/gists{/gist_id}","starred_url":"https://api.github.com/users/maennchen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maennchen/subscriptions","organizations_url":"https://api.github.com/users/maennchen/orgs","repos_url":"https://api.github.com/users/maennchen/repos","events_url":"https://api.github.com/users/maennchen/events{/privacy}","received_events_url":"https://api.github.com/users/maennchen/received_events","type":"User","site_admin":false},"assignees":[{"login":"maennchen","id":333918,"node_id":"MDQ6VXNlcjMzMzkxOA==","avatar_url":"https://avatars1.githubusercontent.com/u/333918?v=4","gravatar_id":"","url":"https://api.github.com/users/maennchen","html_url":"https://github.com/maennchen","followers_url":"https://api.github.com/users/maennchen/followers","following_url":"https://api.github.com/users/maennchen/following{/other_user}","gists_url":"https://api.github.com/users/maennchen/gists{/gist_id}","starred_url":"https://api.github.com/users/maennchen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maennchen/subscriptions","organizations_url":"https://api.github.com/users/maennchen/orgs","repos_url":"https://api.github.com/users/maennchen/repos","events_url":"https://api.github.com/users/maennchen/events{/privacy}","received_events_url":"https://api.github.com/users/maennchen/received_events","type":"User","site_admin":false}],"milestone":null,"comments":3,"created_at":"2018-08-24T19:37:14Z","updated_at":"2018-08-24T21:57:12Z","closed_at":null,"author_association":"NONE","body":"We run our elixir apps in docker-compose for testing and development and in kubernetes for production.  We also use libcluster to help our nodes discover one another.  The new swarm-based version of quantum does't handle scaling in these environments very well.\r\n\r\nI've reproduced a minimal test-case showing this behavior, please see my repo here: https://github.com/peek-travel/quantum_swarm\r\nIf you have docker + compose installed it should be really easy to reproduce the error.\r\n\r\nThe setup is pretty simple: one global scheduler with one job that just prints \"PING!\" to stdout once every 3 seconds.\r\n\r\nThe error I get when scaling up the number of nodes is this:\r\n\r\n```\r\n19:26:11.646 [error] GenServer QuantumSwarm.Scheduler.ExecutorSupervisor terminating\r\n** (FunctionClauseError) no function clause matching in Quantum.ExecutionBroadcaster.add_job_to_state/2\r\n    (quantum) lib/quantum/execution_broadcaster.ex:271: Quantum.ExecutionBroadcaster.add_job_to_state({~N[2018-08-24 19:26:12], [%Quantum.Job{name: #Reference<0.1786720193.3425435649.14848>, overlap: true, run_strategy: %Quantum.RunStrategy.Random{nodes: :cluster}, schedule: ~e[*/3 * * * * * *]e, state: :active, task: {QuantumSwarm.Pinger, :ping, []}, timezone: :utc}]}, %Quantum.ExecutionBroadcaster.State{debug_logging: true, jobs: [], scheduler: QuantumSwarm.Scheduler, storage: Quantum.Storage.Noop, time: ~N[2018-08-24 19:26:10], timer: nil})\r\n    (elixir) lib/enum.ex:1925: Enum.\"-reduce/3-lists^foldl/2-0-\"/3\r\n    (quantum) lib/quantum/execution_broadcaster.ex:241: Quantum.ExecutionBroadcaster.handle_cast/2\r\n    (gen_stage) lib/gen_stage.ex:2039: GenStage.noreply_callback/3\r\n    (stdlib) gen_server.erl:637: :gen_server.try_dispatch/4\r\n    (stdlib) gen_server.erl:711: :gen_server.handle_msg/6\r\n    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3\r\nLast message: {:DOWN, #Reference<0.1786720193.3425435656.15142>, :process, #PID<32743.1366.0>, {:function_clause, [{Quantum.ExecutionBroadcaster, :add_job_to_state, [{~N[2018-08-24 19:26:12], [%Quantum.Job{name: #Reference<0.1786720193.3425435649.14848>, overlap: true, run_strategy: %Quantum.RunStrategy.Random{nodes: :cluster}, schedule: ~e[*/3 * * * * * *]e, state: :active, task: {QuantumSwarm.Pinger, :ping, []}, timezone: :utc}]}, %Quantum.ExecutionBroadcaster.State{debug_logging: true, jobs: [], scheduler: QuantumSwarm.Scheduler, storage: Quantum.Storage.Noop, time: ~N[2018-08-24 19:26:10], timer: nil}], [file: 'lib/quantum/execution_broadcaster.ex', line: 271]}, {Enum, :\"-reduce/3-lists^foldl/2-0-\", 3, [file: 'lib/enum.ex', line: 1925]}, {Quantum.ExecutionBroadcaster, :handle_cast, 2, [file: 'lib/quantum/execution_broadcaster.ex', line: 241]}, {GenStage, :noreply_callback, 3, [file: 'lib/gen_stage.ex', line: 2039]}, {:gen_server, :try_dispatch, 4, [file: 'gen_server.erl', line: 637]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 711]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}}\r\nState: %ConsumerSupervisor{args: %Quantum.ExecutorSupervisor.InitOpts{cluster_task_supervisor_registry_reference: QuantumSwarm.Scheduler.ClusterTaskSupervisorRegistry, debug_logging: true, execution_broadcaster_reference: {:via, :swarm, QuantumSwarm.Scheduler.ExecutionBroadcaster}, task_registry_reference: {:via, :swarm, QuantumSwarm.Scheduler.TaskRegistry}, task_supervisor_reference: QuantumSwarm.Scheduler.Task.Supervisor}, children: %{}, max_restarts: 3, max_seconds: 5, mod: Quantum.ExecutorSupervisor, name: QuantumSwarm.Scheduler.ExecutorSupervisor, producers: %{}, restarting: 0, restarts: [], strategy: :one_for_one, template: {Quantum.Executor, {Quantum.Executor, :start_link, [%Quantum.Executor.StartOpts{cluster_task_supervisor_registry_reference: QuantumSwarm.Scheduler.ClusterTaskSupervisorRegistry, debug_logging: true, task_registry_reference: {:via, :swarm, QuantumSwarm.Scheduler.TaskRegistry}, task_supervisor_reference: QuantumSwarm.Scheduler.Task.Supervisor}]}, :temporary, 5000, :worker, [Quantum.Executor]}}\r\n```\r\n\r\nPlease have a look at the repo and let me know if you have any questions about what I did to set it up.  I'll also be trying to look into the error above and see if I can figure out what's going on.  If I do, I'll send a PR!\r\n\r\nIt should also be noted that quantum 2.2.7 does not have this bug, but has others, like while scaling up the number of nodes you might run the same job multiple times or when scaling down you might miss some.  This isn't such a big deal in our case, so we are reverting to 2.2.7 until we can get to the bottom of the above error."},"comment":{"url":"https://api.github.com/repos/quantum-elixir/quantum-core/issues/comments/415893174","html_url":"https://github.com/quantum-elixir/quantum-core/issues/368#issuecomment-415893174","issue_url":"https://api.github.com/repos/quantum-elixir/quantum-core/issues/368","id":415893174,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNTg5MzE3NA==","user":{"login":"maennchen","id":333918,"node_id":"MDQ6VXNlcjMzMzkxOA==","avatar_url":"https://avatars1.githubusercontent.com/u/333918?v=4","gravatar_id":"","url":"https://api.github.com/users/maennchen","html_url":"https://github.com/maennchen","followers_url":"https://api.github.com/users/maennchen/followers","following_url":"https://api.github.com/users/maennchen/following{/other_user}","gists_url":"https://api.github.com/users/maennchen/gists{/gist_id}","starred_url":"https://api.github.com/users/maennchen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/maennchen/subscriptions","organizations_url":"https://api.github.com/users/maennchen/orgs","repos_url":"https://api.github.com/users/maennchen/repos","events_url":"https://api.github.com/users/maennchen/events{/privacy}","received_events_url":"https://api.github.com/users/maennchen/received_events","type":"User","site_admin":false},"created_at":"2018-08-24T21:57:12Z","updated_at":"2018-08-24T21:57:12Z","author_association":"MEMBER","body":"@doughsay I think I found and solved the issue. I'm a little low on time though and that's why I couldn't test it. Would you mind testing again with #369 ?"}}
{"actor":{"display_login":"maennchen"}}