{"action":"created","issue":{"url":"https://api.github.com/repos/Azure/batch-shipyard/issues/173","repository_url":"https://api.github.com/repos/Azure/batch-shipyard","labels_url":"https://api.github.com/repos/Azure/batch-shipyard/issues/173/labels{/name}","comments_url":"https://api.github.com/repos/Azure/batch-shipyard/issues/173/comments","events_url":"https://api.github.com/repos/Azure/batch-shipyard/issues/173/events","html_url":"https://github.com/Azure/batch-shipyard/issues/173","id":304269430,"number":173,"title":"Support \"depends_on\" in task_factory","user":{"login":"whataride","id":6028927,"avatar_url":"https://avatars3.githubusercontent.com/u/6028927?v=4","gravatar_id":"","url":"https://api.github.com/users/whataride","html_url":"https://github.com/whataride","followers_url":"https://api.github.com/users/whataride/followers","following_url":"https://api.github.com/users/whataride/following{/other_user}","gists_url":"https://api.github.com/users/whataride/gists{/gist_id}","starred_url":"https://api.github.com/users/whataride/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/whataride/subscriptions","organizations_url":"https://api.github.com/users/whataride/orgs","repos_url":"https://api.github.com/users/whataride/repos","events_url":"https://api.github.com/users/whataride/events{/privacy}","received_events_url":"https://api.github.com/users/whataride/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-03-12T07:49:49Z","updated_at":"2018-03-13T07:56:09Z","closed_at":null,"author_association":"NONE","body":"Right now one cannot specify dependency of tasks generated by task_factory on other tasks. For example, I have a split task, which will split a file to smaller ones, and want to generate tasks to process each of the files created in the split task. This type of dependency is fairly common in jobs that requires map and reduce."},"comment":{"url":"https://api.github.com/repos/Azure/batch-shipyard/issues/comments/372577354","html_url":"https://github.com/Azure/batch-shipyard/issues/173#issuecomment-372577354","issue_url":"https://api.github.com/repos/Azure/batch-shipyard/issues/173","id":372577354,"user":{"login":"whataride","id":6028927,"avatar_url":"https://avatars3.githubusercontent.com/u/6028927?v=4","gravatar_id":"","url":"https://api.github.com/users/whataride","html_url":"https://github.com/whataride","followers_url":"https://api.github.com/users/whataride/followers","following_url":"https://api.github.com/users/whataride/following{/other_user}","gists_url":"https://api.github.com/users/whataride/gists{/gist_id}","starred_url":"https://api.github.com/users/whataride/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/whataride/subscriptions","organizations_url":"https://api.github.com/users/whataride/orgs","repos_url":"https://api.github.com/users/whataride/repos","events_url":"https://api.github.com/users/whataride/events{/privacy}","received_events_url":"https://api.github.com/users/whataride/received_events","type":"User","site_admin":false},"created_at":"2018-03-13T07:56:09Z","updated_at":"2018-03-13T07:56:09Z","author_association":"NONE","body":"Yes, what we need is that each task specification generated from task_factory can depend on some other task.\r\n\r\nTo be more specific, here is an example:\r\n\r\n`job_specifications:\r\n- id: some-job\r\n  tasks:\r\n  - id: split_file\r\n    output_data:\r\n      azure_storage:\r\n      - storage_account_settings: mystorageaccount\r\n        remote_path: engressed-files\r\n        local_path: $AZ_BATCH_TASK_WORKING_DIR\r\n        include:\r\n        - 'chunks/x*'\r\n        blobxfer_extra_options: null\r\n    command: mkdir -p chunks && split -n l/10 -d ./test/urls.txt && mv x* chunks\r\n  - id: null\r\n    # depends_on doesn't work for task_factory\r\n    # depends_on:\r\n    # - split_file\r\n    task_factory:\r\n      file:\r\n        azure_storage:\r\n          storage_account_settings: mystorageaccount\r\n          remote_path: engressed-files\r\n          is_file_share: false\r\n          include:\r\n          - 'chunks/x*'\r\n        task_filepath: file_name\r\n    command: cp {file_name} {file_name}_copied\r\n`\r\nDo we have a workaround for this kind of dependency right now? "}}
{"actor":{"display_login":"whataride"}}