{"action":"created","issue":{"url":"https://api.github.com/repos/spotify/luigi/issues/1750","repository_url":"https://api.github.com/repos/spotify/luigi","labels_url":"https://api.github.com/repos/spotify/luigi/issues/1750/labels{/name}","comments_url":"https://api.github.com/repos/spotify/luigi/issues/1750/comments","events_url":"https://api.github.com/repos/spotify/luigi/issues/1750/events","html_url":"https://github.com/spotify/luigi/issues/1750","id":164071871,"number":1750,"title":"Scalability problem: unworkable scheduler slowdown when requiring many tasks","user":{"login":"proycon","id":75427,"avatar_url":"https://avatars.githubusercontent.com/u/75427?v=3","gravatar_id":"","url":"https://api.github.com/users/proycon","html_url":"https://github.com/proycon","followers_url":"https://api.github.com/users/proycon/followers","following_url":"https://api.github.com/users/proycon/following{/other_user}","gists_url":"https://api.github.com/users/proycon/gists{/gist_id}","starred_url":"https://api.github.com/users/proycon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/proycon/subscriptions","organizations_url":"https://api.github.com/users/proycon/orgs","repos_url":"https://api.github.com/users/proycon/repos","events_url":"https://api.github.com/users/proycon/events{/privacy}","received_events_url":"https://api.github.com/users/proycon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-07-06T13:10:34Z","updated_at":"2016-07-07T10:13:34Z","closed_at":null,"body":"I have run into a scalability problem where the scheduler causes an unworkable delay when many tasks are scheduled. Have I run into an inherent limitation in Luigi? In my use case I have a task that reads an input directory and requires() child tasks for each file in that input directory. I have a simple example case that just lowercases all files and reproduces the problem:\r\n\r\n```python\r\nimport os\r\nimport glob\r\nimport luigi\r\n\r\nclass LowercaseFile(luigi.ExternalTask):\r\n    inputfile = luigi.Parameter()\r\n    outputdir = luigi.Parameter()\r\n\r\n    def run(self):\r\n        with self.output().open('w') as f_out:\r\n            with open(self.inputfile,'r',encoding='utf-8') as f_in:\r\n                f_out.write(f_in.read().lower())\r\n\r\n    def output(self):\r\n        return luigi.LocalTarget(os.path.join(self.outputdir, os.path.basename(self.inputfile)))\r\n\r\nclass LowercaseDir(luigi.Task):\r\n    inputdir = luigi.Parameter()\r\n    outputdir = luigi.Parameter()\r\n\r\n    def requires(self):\r\n        return [ LowercaseFile(inputfile=inputfile, outputdir=self.outputdir) for inputfile in glob.glob(self.inputdir + '/*.txt') ]\r\n\r\n    def run(self):\r\n        with self.output().open('w') as f:\r\n            f.write(\"done\")\r\n\r\n    def output(self):\r\n        return luigi.LocalTarget(\"done\")\r\n```\r\n\r\nWhen I run this with five workers and feed it a directory with 300,000 input files. The scheduler only manages to get new work about each every two seconds (so +- 166 hours to do it all), the child tasks finish way faster than two seconds, which implies only one of my five workers is effectively doing something. On a low number of files (1000 or less) everything works fast and fine.\r\n\r\nIt doesn't really matter whether I use the local scheduler or the central one, and whether I enable ``--parallel-scheduling`` on the local one or not.\r\n\r\nAm I missing something? Is there a way to remedy this scalability problem, or have I hit a limit in Luigi itself? I was planning on using it on big data collections (300,000 files is still fairly small even).\r\n\r\nSome output:\r\n\r\n```\r\nINFO: Informed scheduler that task   LowercaseFile_scaletest_txtdir_out2_7c1f2aea5e   has status   PENDING\r\nINFO: Done scheduling tasks\r\nINFO: Running Worker with 5 processes\r\nDEBUG: Asking scheduler for work...\r\nDEBUG: Pending tasks: 299981\r\nDEBUG: Asking scheduler for work...\r\nINFO: [pid 31350] Worker Worker(salt=482450273, workers=5, host=mhysa, username=proycon, pid=28147) running   LowercaseFile(inputfile=scaletest.txtdir/153157.txt, outputdir=out2)\r\nINFO: [pid 31350] Worker Worker(salt=482450273, workers=5, host=mhysa, username=proycon, pid=28147) done      LowercaseFile(inputfile=scaletest.txtdir/153157.txt, outputdir=out2)\r\nDEBUG: Pending tasks: 299980\r\nDEBUG: Asking scheduler for work...\r\nINFO: [pid 31356] Worker Worker(salt=482450273, workers=5, host=mhysa, username=proycon, pid=28147) running   LowercaseFile(inputfile=scaletest.txtdir/29528.txt, outputdir=out2)\r\nINFO: [pid 31356] Worker Worker(salt=482450273, workers=5, host=mhysa, username=proycon, pid=28147) done      LowercaseFile(inputfile=scaletest.txtdir/29528.txt, outputdir=out2)\r\nDEBUG: Pending tasks: 299979\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/spotify/luigi/issues/comments/231038688","html_url":"https://github.com/spotify/luigi/issues/1750#issuecomment-231038688","issue_url":"https://api.github.com/repos/spotify/luigi/issues/1750","id":231038688,"user":{"login":"proycon","id":75427,"avatar_url":"https://avatars.githubusercontent.com/u/75427?v=3","gravatar_id":"","url":"https://api.github.com/users/proycon","html_url":"https://github.com/proycon","followers_url":"https://api.github.com/users/proycon/followers","following_url":"https://api.github.com/users/proycon/following{/other_user}","gists_url":"https://api.github.com/users/proycon/gists{/gist_id}","starred_url":"https://api.github.com/users/proycon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/proycon/subscriptions","organizations_url":"https://api.github.com/users/proycon/orgs","repos_url":"https://api.github.com/users/proycon/repos","events_url":"https://api.github.com/users/proycon/events{/privacy}","received_events_url":"https://api.github.com/users/proycon/received_events","type":"User","site_admin":false},"created_at":"2016-07-07T10:13:34Z","updated_at":"2016-07-07T10:13:34Z","body":"That is rather unfortunate..  I have done some profiling of ``Scheduler.get_work()`` and wonder if there aren't ways to overcome this limitation by not considering all pending tasks every time, but partitioning them into smaller queues first and moving on to the next when a queue is done. This may not combine well with certain other features such has task priorities though.\r\n\r\n(300k tasks, intel i7-4770K):\r\n\r\n```\r\nget_pending_tasks: 1.1205673217773438e-05s\r\nlist(relevant_tasks): 0.012180805206298828s\r\ntasks.sort():0.39243364334106445s\r\niteration over tasks:0.9191298484802246s\r\n```"}}
{"actor":{"display_login":"proycon"}}