{"action":"created","issue":{"url":"https://api.github.com/repos/Microsoft/pai/issues/1244","repository_url":"https://api.github.com/repos/Microsoft/pai","labels_url":"https://api.github.com/repos/Microsoft/pai/issues/1244/labels{/name}","comments_url":"https://api.github.com/repos/Microsoft/pai/issues/1244/comments","events_url":"https://api.github.com/repos/Microsoft/pai/issues/1244/events","html_url":"https://github.com/Microsoft/pai/issues/1244","id":356287783,"node_id":"MDU6SXNzdWUzNTYyODc3ODM=","number":1244,"title":"How to load data like code from HDFS to container?","user":{"login":"DengZeshuai","id":30120222,"node_id":"MDQ6VXNlcjMwMTIwMjIy","avatar_url":"https://avatars2.githubusercontent.com/u/30120222?v=4","gravatar_id":"","url":"https://api.github.com/users/DengZeshuai","html_url":"https://github.com/DengZeshuai","followers_url":"https://api.github.com/users/DengZeshuai/followers","following_url":"https://api.github.com/users/DengZeshuai/following{/other_user}","gists_url":"https://api.github.com/users/DengZeshuai/gists{/gist_id}","starred_url":"https://api.github.com/users/DengZeshuai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DengZeshuai/subscriptions","organizations_url":"https://api.github.com/users/DengZeshuai/orgs","repos_url":"https://api.github.com/users/DengZeshuai/repos","events_url":"https://api.github.com/users/DengZeshuai/events{/privacy}","received_events_url":"https://api.github.com/users/DengZeshuai/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-09-02T14:42:23Z","updated_at":"2018-09-03T08:50:02Z","closed_at":null,"author_association":"NONE","body":"For code, I upload the code in HDFS directory `/Home/admin/dcgan`, and submit job with configuration ```\"codeDir\": $PAI_DEFAULT_FS_URI/Home/admin/dcgan```. After create the container, the code will be copy into `/root/dcgan` in the container.\r\n\r\nHowever, for dataset, I upload the code in the directory `/Home/admin/flowers` of HDFS, and configure the ```\"dataDir\": \"$PAI_DEFAULT_FS_URI/Home/admin/flowers\"```, but this dataset will not be copy into container. I want to know how can I load the data into container? \r\n\r\nIn addition, in the past version of OpenPAI, I have put the data and code in the same directory of HDFS, and copy the directory into container, and it will work. But in the new version of OpenPAI, it will report anerror ```hdfs://master-ip:9000/Home/admin/dcgan is larger than 200MB, exit.```"},"comment":{"url":"https://api.github.com/repos/Microsoft/pai/issues/comments/418046397","html_url":"https://github.com/Microsoft/pai/issues/1244#issuecomment-418046397","issue_url":"https://api.github.com/repos/Microsoft/pai/issues/1244","id":418046397,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODA0NjM5Nw==","user":{"login":"DengZeshuai","id":30120222,"node_id":"MDQ6VXNlcjMwMTIwMjIy","avatar_url":"https://avatars2.githubusercontent.com/u/30120222?v=4","gravatar_id":"","url":"https://api.github.com/users/DengZeshuai","html_url":"https://github.com/DengZeshuai","followers_url":"https://api.github.com/users/DengZeshuai/followers","following_url":"https://api.github.com/users/DengZeshuai/following{/other_user}","gists_url":"https://api.github.com/users/DengZeshuai/gists{/gist_id}","starred_url":"https://api.github.com/users/DengZeshuai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DengZeshuai/subscriptions","organizations_url":"https://api.github.com/users/DengZeshuai/orgs","repos_url":"https://api.github.com/users/DengZeshuai/repos","events_url":"https://api.github.com/users/DengZeshuai/events{/privacy}","received_events_url":"https://api.github.com/users/DengZeshuai/received_events","type":"User","site_admin":false},"created_at":"2018-09-03T08:50:02Z","updated_at":"2018-09-03T08:50:02Z","author_association":"NONE","body":"I have ask Mr. Yuan Hao to solve my question，that is the data will not load into container automatically, and it need to use command `hdfs dfs -get $PAI_DATA_DIR ./data ` to load data actively.\r\n\r\nThanks for you help. @YanjieGao "}}
{"actor":{"display_login":"DengZeshuai"}}