{"action":"created","issue":{"url":"https://api.github.com/repos/pachyderm/pachyderm/issues/666","repository_url":"https://api.github.com/repos/pachyderm/pachyderm","labels_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/666/labels{/name}","comments_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/666/comments","events_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/666/events","html_url":"https://github.com/pachyderm/pachyderm/issues/666","id":165713132,"number":666,"title":"Problem met in the example Web Scraper","user":{"login":"ShengjieLuo","id":20234415,"avatar_url":"https://avatars.githubusercontent.com/u/20234415?v=3","gravatar_id":"","url":"https://api.github.com/users/ShengjieLuo","html_url":"https://github.com/ShengjieLuo","followers_url":"https://api.github.com/users/ShengjieLuo/followers","following_url":"https://api.github.com/users/ShengjieLuo/following{/other_user}","gists_url":"https://api.github.com/users/ShengjieLuo/gists{/gist_id}","starred_url":"https://api.github.com/users/ShengjieLuo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ShengjieLuo/subscriptions","organizations_url":"https://api.github.com/users/ShengjieLuo/orgs","repos_url":"https://api.github.com/users/ShengjieLuo/repos","events_url":"https://api.github.com/users/ShengjieLuo/events{/privacy}","received_events_url":"https://api.github.com/users/ShengjieLuo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-07-15T05:18:44Z","updated_at":"2016-07-15T05:57:03Z","closed_at":null,"body":"Hi, I have a met a problem in the example Web Scraper, I guess it is led by the proxy server problem.\r\n\r\nI follow the guide of Web Scraper.  The pachyderm job's status is running, but with no result.\r\n`pachctl list-job`\r\n`ID                                 OUTPUT                                             STARTED             DURATION            STATE`\r\n`4823900e201664353385d198bd6c7b86   scraper/4cf15ae67e5d43d39177a590ea1ee876           23 minutes ago      -                   running`\r\n\r\n` ls ./pfs5/scraper/4cf15ae67e5d43d39177a590ea1ee876/`\r\n`Nothing in the output file folder`\r\n\r\nIt confuses me so I directly use the command in the `pipeline.json`\r\n`wget  --recursive --level 1 --accept jpg,jpeg,png,gif,bmp  --page-requisites --adjust-extension --span-hosts --no-check-certificate  --timestamping --directory-prefix /root  --input-file /root/pfs5/urls/a996ffb34c6742a6a3368d9715330686/urls`\r\n\r\nThe command works successfully,\r\n`--2016-07-15 12:53:52--  http://www.google.com/`\r\n`Resolving child-prc.intel.com (child-prc.intel.com)... 10.239.4.80`\r\n`Connecting to child-prc.intel.com (child-prc.intel.com)|10.239.4.80|:914... connected.`\r\n`Proxy request sent, awaiting response... 200 OK`\r\n`Length: unspecified [text/html]`\r\n`Saving to: ‘/root/www.google.com/index.html’`\r\n  `  [ <=>                ] 10,427      --.-K/s   in 0.002s`\r\n\r\nNote that my server used a proxy to connect the google.com. The proxy works in my server, but does it work in the pachyderm container as well? Do I need to review the Dockerfile used in this example to add the proxy configuration? Or  the network configuration and environment variables are shared in dockers and servers ? And if this is necessary to add the proxy configuration, would you mind tell me how to deal with it?\r\n"},"comment":{"url":"https://api.github.com/repos/pachyderm/pachyderm/issues/comments/232866206","html_url":"https://github.com/pachyderm/pachyderm/issues/666#issuecomment-232866206","issue_url":"https://api.github.com/repos/pachyderm/pachyderm/issues/666","id":232866206,"user":{"login":"ShengjieLuo","id":20234415,"avatar_url":"https://avatars.githubusercontent.com/u/20234415?v=3","gravatar_id":"","url":"https://api.github.com/users/ShengjieLuo","html_url":"https://github.com/ShengjieLuo","followers_url":"https://api.github.com/users/ShengjieLuo/followers","following_url":"https://api.github.com/users/ShengjieLuo/following{/other_user}","gists_url":"https://api.github.com/users/ShengjieLuo/gists{/gist_id}","starred_url":"https://api.github.com/users/ShengjieLuo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ShengjieLuo/subscriptions","organizations_url":"https://api.github.com/users/ShengjieLuo/orgs","repos_url":"https://api.github.com/users/ShengjieLuo/repos","events_url":"https://api.github.com/users/ShengjieLuo/events{/privacy}","received_events_url":"https://api.github.com/users/ShengjieLuo/received_events","type":"User","site_admin":false},"created_at":"2016-07-15T05:57:03Z","updated_at":"2016-07-15T05:57:03Z","body":"To address this issue, I change the command in the pipeline.json as following,\r\n`\"pipeline\": {`\r\n`    \"name\": \"scraper_export_proxy\"`\r\n`  },`\r\n ` \"transform\": {`\r\n    `\"cmd\": [`\r\n       ` \"export\",\"http_proxy=http://username:password@child-prc.intel.com:914\",`\r\n       ` \"export\",\"https_proxy=https://username:password@child-prc.intel.com:914\",`\r\n      `  \"echo\",\"$http_proxy\",\">\",\"/pfs/out/test.txt\",`\r\n       ` \"echo\",\"$https_proxy\",\">\",\"/pfs/out/test.txt\",`\r\n       ` \"wget\",`\r\n       ` \"--recursive\",`\r\n       ` \"--level\", \"1\",`\r\n       `... ...`\r\n\r\nUnfortunately, the change doesn't work.\r\nSee job status here,\r\n`ID                                 OUTPUT                                                  STARTED             DURATION            STATE`\r\n`9e172109fe3522527883473a01688ab9   scraper_export_proxy/6bada8e255f446cd8d60d6bb2ab04ec8   7 minutes ago       -                   pulling`\r\n\r\nSee logs here,\r\n`pachctl get-logs 9e172109fe3522527883473a01688ab9`\r\n`container \"user\" in pod \"9e172109fe3522527883473a01688ab9-6uqeu\" is waiting to start: ContainerCreating`\r\n\r\nSee kubectl status here,\r\n`NAME                                     READY        STATUS              RESTARTS                       AGE            IP           NODE`\r\n`9e172109fe3522527883473a01688ab9-6uqeu   0/1          ContainerCreating   0                              9m             <none>       127.0.0.1`\r\n\r\nSo this method don't works for the proxy problem. "}}
{"actor":{"display_login":"ShengjieLuo"}}