{"action":"created","issue":{"url":"https://api.github.com/repos/spotify/luigi/issues/1538","repository_url":"https://api.github.com/repos/spotify/luigi","labels_url":"https://api.github.com/repos/spotify/luigi/issues/1538/labels{/name}","comments_url":"https://api.github.com/repos/spotify/luigi/issues/1538/comments","events_url":"https://api.github.com/repos/spotify/luigi/issues/1538/events","html_url":"https://github.com/spotify/luigi/pull/1538","id":132303976,"number":1538,"title":"Enables running of multiple tasks in batches","user":{"login":"daveFNbuck","id":2091885,"avatar_url":"https://avatars.githubusercontent.com/u/2091885?v=3","gravatar_id":"","url":"https://api.github.com/users/daveFNbuck","html_url":"https://github.com/daveFNbuck","followers_url":"https://api.github.com/users/daveFNbuck/followers","following_url":"https://api.github.com/users/daveFNbuck/following{/other_user}","gists_url":"https://api.github.com/users/daveFNbuck/gists{/gist_id}","starred_url":"https://api.github.com/users/daveFNbuck/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/daveFNbuck/subscriptions","organizations_url":"https://api.github.com/users/daveFNbuck/orgs","repos_url":"https://api.github.com/users/daveFNbuck/repos","events_url":"https://api.github.com/users/daveFNbuck/events{/privacy}","received_events_url":"https://api.github.com/users/daveFNbuck/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-02-09T01:11:41Z","updated_at":"2016-02-22T03:04:22Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/spotify/luigi/pulls/1538","html_url":"https://github.com/spotify/luigi/pull/1538","diff_url":"https://github.com/spotify/luigi/pull/1538.diff","patch_url":"https://github.com/spotify/luigi/pull/1538.patch"},"body":"Sometimes it's more efficient to run a group of tasks all at once rather than\r\none at a time. With luigi, it's difficult to take advantage of this because your\r\nbatch size will also be the minimum granularity you're able to compute. So if\r\nyou have a job that runs hourly, you can't combine their computation when many\r\nof them get backlogged. When you have a task that runs daily, you can't get\r\nhourly runs.\r\n\r\nIn order to gain efficiency when many jobs are queued up, this change allows\r\nworkers to provide details of how jobs can be batched to the scheduler. If you\r\nhave several hourly jobs of the same type in the scheduler, it can combine them\r\ninto a single job for the worker. We allow parameters to be combined in three\r\nways: we can combine all the arguments in a csv, take the min and max to form\r\na range, or just provide the min or max. The csv gives the most specificity,\r\nbut range and min/max are available for when that's all you need. In particular,\r\nthe max function provides an implementation of #570, allowing for jobs that\r\noverwrite eachother to be grouped by just running the largest one.\r\n\r\nIn order to implement this, the scheduler will create a new task based on the\r\ninformation sent by the worker. It's possible (as in the max/min case) that the\r\nnew task already exists, but if it doesn't it will be cleaned up at the end of\r\nthe run. While this new task is running, any other tasks will be marked as\r\nBATCH_RUNNING. When the head task becomes DONE or FAILED, the BATCH_RUNNING\r\ntasks will also be updated accordingly. They'll also have their tracking urls\r\nupdated to match the batch task.\r\n\r\nThis is a fairly big change to how the scheduler works, so there are a few\r\nissues with it in the initial implementation:\r\n  - newly created batch tasks don't show up in dependency graphs\r\n  - the run summary doesn't know what happened to the batched tasks\r\n  - we can't limit how big batches can be (how should we handle ranges?)\r\n  - batching takes quadratic time for simplicity of implementation\r\n  - I'm not sure what would happen if there was a yield in a batch run function\r\n\r\nOn the worker side, batching is accomplished by setting a batch_class,\r\nbatcher_args and batcher_aggregate_args. The batch class is the Python class\r\nthat runs the batched version of the job. This can be set equal to the current\r\nclass by overriding the class method get_batch_class.\r\n\r\nThe batcher_args are the arguments passed from the current class to the batch\r\nclass. These come in pairs. So if the original class has parameters machine and\r\nfilename that need to go to host and files in the batcher, you'll use\r\n  [('machine', 'host'), ('filename', 'files')]\r\nfor batcher_args.\r\n\r\nThe final value is batcher_aggregate_args, which explains which arguments are to\r\nbe aggregated and how. So using the machine, filename example, we might want to\r\nbatch multiple files together for the same machine. For that, we could do\r\nsomething like\r\n  {'filename': 'csv'}\r\nto combine them all as comma-separated values. Now if we have multiple machine,\r\nfilename pairs such as ('m1', 'f1'), ('m1', 'f2'), ('m2', 'f3'), ('m2', 'f4'),\r\nwe'd end up with batch jobs with host, files pairs of ('m1', 'f1,f2') and\r\n('m2', 'f3,f4').\r\n\r\nThe worker will send the batch class, batcher args and batcher aggregate args to\r\nthe worker once per class, which is why these are class methods. It doesn't make\r\nsense to have different ways to batch per individual task, so that's not\r\nallowed."},"comment":{"url":"https://api.github.com/repos/spotify/luigi/issues/comments/186979529","html_url":"https://github.com/spotify/luigi/pull/1538#issuecomment-186979529","issue_url":"https://api.github.com/repos/spotify/luigi/issues/1538","id":186979529,"user":{"login":"erikbern","id":1027979,"avatar_url":"https://avatars.githubusercontent.com/u/1027979?v=3","gravatar_id":"","url":"https://api.github.com/users/erikbern","html_url":"https://github.com/erikbern","followers_url":"https://api.github.com/users/erikbern/followers","following_url":"https://api.github.com/users/erikbern/following{/other_user}","gists_url":"https://api.github.com/users/erikbern/gists{/gist_id}","starred_url":"https://api.github.com/users/erikbern/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erikbern/subscriptions","organizations_url":"https://api.github.com/users/erikbern/orgs","repos_url":"https://api.github.com/users/erikbern/repos","events_url":"https://api.github.com/users/erikbern/events{/privacy}","received_events_url":"https://api.github.com/users/erikbern/received_events","type":"User","site_admin":false},"created_at":"2016-02-22T03:04:22Z","updated_at":"2016-02-22T03:04:22Z","body":"Sorry for the delay. Have been busy at work. Will try to allocate some time for this"}}