{"action":"created","issue":{"url":"https://api.github.com/repos/Azure/BatchAI/issues/15","repository_url":"https://api.github.com/repos/Azure/BatchAI","labels_url":"https://api.github.com/repos/Azure/BatchAI/issues/15/labels{/name}","comments_url":"https://api.github.com/repos/Azure/BatchAI/issues/15/comments","events_url":"https://api.github.com/repos/Azure/BatchAI/issues/15/events","html_url":"https://github.com/Azure/BatchAI/issues/15","id":283800008,"node_id":"MDU6SXNzdWUyODM4MDAwMDg=","number":15,"title":"Provide TF_CONFIG environment variable for distributed TensorFlow","user":{"login":"damienpontifex","id":1321276,"node_id":"MDQ6VXNlcjEzMjEyNzY=","avatar_url":"https://avatars1.githubusercontent.com/u/1321276?v=4","gravatar_id":"","url":"https://api.github.com/users/damienpontifex","html_url":"https://github.com/damienpontifex","followers_url":"https://api.github.com/users/damienpontifex/followers","following_url":"https://api.github.com/users/damienpontifex/following{/other_user}","gists_url":"https://api.github.com/users/damienpontifex/gists{/gist_id}","starred_url":"https://api.github.com/users/damienpontifex/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/damienpontifex/subscriptions","organizations_url":"https://api.github.com/users/damienpontifex/orgs","repos_url":"https://api.github.com/users/damienpontifex/repos","events_url":"https://api.github.com/users/damienpontifex/events{/privacy}","received_events_url":"https://api.github.com/users/damienpontifex/received_events","type":"User","site_admin":false},"labels":[{"id":691899852,"node_id":"MDU6TGFiZWw2OTE4OTk4NTI=","url":"https://api.github.com/repos/Azure/BatchAI/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2017-12-21T07:54:35Z","updated_at":"2018-07-25T02:50:14Z","closed_at":null,"author_association":"NONE","body":"The TensorFlow `ClusterConfig` can parse worker and parameter server settings from a `TF_CONFIG` environment variable (see https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/learn/python/learn/estimators/run_config.py#L64-L156)\r\n\r\nI was trying to pass it via an environment variable in the job configuration file like so:\r\n```json\r\n\"{ 'cluster': { 'ps': $AZ_BATCHAI_PS_HOSTS, 'worker': $AZ_BATCHAI_WORKER_HOSTS }, 'task': { 'index': $AZ_BATCHAI_TASK_INDEX, 'type': '' } }\"\r\n```\r\n\r\nWhich is kind of fine, but falls down for a few cases:\r\n1. When there are no parameter servers (i.e. single node) the ps hosts should be an empty array, but in this case it's just an empty string.\r\n2. The variables for hosts and workers are comma separated and the TF code parses it as JSON, so would ideally be an array type inside this string.\r\n2. The 'task.type' property can be 'master', 'worker' or 'ps' but that doesn't seem to have an appropriate environment variable and I had to pass the option via command line args\r\n\r\nMore generally though, providing this configuration via a `TF_CONFIG` environment variable would significantly lower the bar to get distributed training working in TensorFlow and Azure Batch. It would also simplify command line arg parameters and mean just the appropriate data directories would need to be passed and mean the same arguments could be used across master, worker and ps potentially simplify the tensorflowSettings property further."},"comment":{"url":"https://api.github.com/repos/Azure/BatchAI/issues/comments/407616451","html_url":"https://github.com/Azure/BatchAI/issues/15#issuecomment-407616451","issue_url":"https://api.github.com/repos/Azure/BatchAI/issues/15","id":407616451,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNzYxNjQ1MQ==","user":{"login":"yangsiyu007","id":9201966,"node_id":"MDQ6VXNlcjkyMDE5NjY=","avatar_url":"https://avatars3.githubusercontent.com/u/9201966?v=4","gravatar_id":"","url":"https://api.github.com/users/yangsiyu007","html_url":"https://github.com/yangsiyu007","followers_url":"https://api.github.com/users/yangsiyu007/followers","following_url":"https://api.github.com/users/yangsiyu007/following{/other_user}","gists_url":"https://api.github.com/users/yangsiyu007/gists{/gist_id}","starred_url":"https://api.github.com/users/yangsiyu007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yangsiyu007/subscriptions","organizations_url":"https://api.github.com/users/yangsiyu007/orgs","repos_url":"https://api.github.com/users/yangsiyu007/repos","events_url":"https://api.github.com/users/yangsiyu007/events{/privacy}","received_events_url":"https://api.github.com/users/yangsiyu007/received_events","type":"User","site_admin":false},"created_at":"2018-07-25T02:50:14Z","updated_at":"2018-07-25T02:50:14Z","author_association":"NONE","body":"I don't think so - not when I tried it the week before last... @lliimsft updates?"}}
{"actor":{"display_login":"yangsiyu007"}}