{"action":"created","issue":{"url":"https://api.github.com/repos/spotify/luigi/issues/2543","repository_url":"https://api.github.com/repos/spotify/luigi","labels_url":"https://api.github.com/repos/spotify/luigi/issues/2543/labels{/name}","comments_url":"https://api.github.com/repos/spotify/luigi/issues/2543/comments","events_url":"https://api.github.com/repos/spotify/luigi/issues/2543/events","html_url":"https://github.com/spotify/luigi/issues/2543","id":365222547,"node_id":"MDU6SXNzdWUzNjUyMjI1NDc=","number":2543,"title":"Per-task retry_count doesn't seem to work","user":{"login":"deneb84","id":8522197,"node_id":"MDQ6VXNlcjg1MjIxOTc=","avatar_url":"https://avatars1.githubusercontent.com/u/8522197?v=4","gravatar_id":"","url":"https://api.github.com/users/deneb84","html_url":"https://github.com/deneb84","followers_url":"https://api.github.com/users/deneb84/followers","following_url":"https://api.github.com/users/deneb84/following{/other_user}","gists_url":"https://api.github.com/users/deneb84/gists{/gist_id}","starred_url":"https://api.github.com/users/deneb84/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deneb84/subscriptions","organizations_url":"https://api.github.com/users/deneb84/orgs","repos_url":"https://api.github.com/users/deneb84/repos","events_url":"https://api.github.com/users/deneb84/events{/privacy}","received_events_url":"https://api.github.com/users/deneb84/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-09-30T12:54:23Z","updated_at":"2018-10-01T19:12:06Z","closed_at":null,"author_association":"NONE","body":"<!---\r\nWe use GitHub issues mainly for tracking bugs and feature requests.\r\nQuestions for how to use luigi can be sent to the mailing list.\r\n\r\nCurrently, there are no strict procedures or guidelines for submitting issues.\r\nIn short, please just use common sense.\r\n\r\nCommon sense includes this at bare-minimum:\r\n\r\n * search for similar issues posted before creating a new issue.\r\n * Use markdown to format all code/logs. Issues which are hard to read\r\n   when rendered on GitHub might be closed with a friendly reminder of this.\r\n * If applicable, reading relevant parts of the documentation.\r\n\r\nAlso, add steps to reproduce the bug, if applicable. Sample code would be nice too :)\r\n\r\nFor more information on how to submit valuable contributions,\r\nsee https://opensource.guide/how-to-contribute/#how-to-submit-a-contribution\r\n-->\r\n\r\nI was trying to set a `retry_count` for a specific task, the idea is to wait for external dependencies but for a limited period of time.\r\n\r\nI tried default luigi configuration with this sample code:\r\n\r\n```\r\nimport luigi\r\n\r\nfrom luigi import Parameter\r\n\r\n\r\nclass InputDataTask(luigi.ExternalTask):\r\n    retry_count = 2\r\n\r\n    target = Parameter()\r\n    def output(self):\r\n        return [luigi.LocalTarget(self.target)]\r\n\r\nclass CleanTask(luigi.Task):\r\n    retry_count = 2\r\n\r\n    def output(self):\r\n        return luigi.LocalTarget(\"cleaned/cleaned\")\r\n\r\n    def requires(self):\r\n        reqs = []\r\n        reqs.append(InputDataTask(target = \"input/file1\"))\r\n        reqs.append(InputDataTask(target = \"input/file2\"))\r\n        reqs.append(InputDataTask(target = \"input/file3\"))\r\n        return reqs\r\n\r\n    def run(self):\r\n        with self.output().open(\"w\") as output_file:\r\n            output_file.write(\"pipeline\")\r\n\r\nif __name__ == '__main__':\r\n    luigi.run()\r\n```\r\nI run this script with the following command:\r\n`python luigi-example-easy.py CleanTask --worker-retry-external-tasks --worker-keep-alive`\r\n\r\nAnithing seem to work fine except the fact that the process never ends in case the external dependencies are not satisfied.\r\n\r\nThe same settings works if I modify luigi.cfg adding: \r\n```\r\n[scheduler]\r\nretry_count = 2\r\n```\r\nFurthermore, adding `--scheduler-retry-count 2` to the command line args returns:\r\n`luigi-example-easy.py: error: unrecognized arguments: --scheduler-retry-count 5`\r\neven though:\r\n```\r\n$ luigi --help-all | grep retry-count\r\n             [--scheduler-retry-count SCHEDULER_RETRY_COUNT]\r\n  --scheduler-retry-count SCHEDULER_RETRY_COUNT\r\n\r\n```\r\ncalling the function using directly `luigi` seems to work again:\r\n`luigi --module luigi-example-easy CleanTask --worker-retry-external-tasks --worker-keep-alive --scheduler-retry-count 2`\r\n\r\nit works also without `--scheduler-retry-count 2` since the configuration is specified in the task"},"comment":{"url":"https://api.github.com/repos/spotify/luigi/issues/comments/426027603","html_url":"https://github.com/spotify/luigi/issues/2543#issuecomment-426027603","issue_url":"https://api.github.com/repos/spotify/luigi/issues/2543","id":426027603,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNjAyNzYwMw==","user":{"login":"deneb84","id":8522197,"node_id":"MDQ6VXNlcjg1MjIxOTc=","avatar_url":"https://avatars1.githubusercontent.com/u/8522197?v=4","gravatar_id":"","url":"https://api.github.com/users/deneb84","html_url":"https://github.com/deneb84","followers_url":"https://api.github.com/users/deneb84/followers","following_url":"https://api.github.com/users/deneb84/following{/other_user}","gists_url":"https://api.github.com/users/deneb84/gists{/gist_id}","starred_url":"https://api.github.com/users/deneb84/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deneb84/subscriptions","organizations_url":"https://api.github.com/users/deneb84/orgs","repos_url":"https://api.github.com/users/deneb84/repos","events_url":"https://api.github.com/users/deneb84/events{/privacy}","received_events_url":"https://api.github.com/users/deneb84/received_events","type":"User","site_admin":false},"created_at":"2018-10-01T19:12:06Z","updated_at":"2018-10-01T19:12:06Z","author_association":"NONE","body":"sure, I think is @javrasya. I tried and it seems that all the parameters added  [here](https://github.com/spotify/luigi/pull/1791) are available only if the task is started using directly luigi.\r\n\r\n```\r\n\"retry_count\"\r\n\"disable_hard_timeout\"\r\n\"disable_window\"\r\n```"}}
{"actor":{"display_login":"deneb84"}}