{"action":"created","issue":{"url":"https://api.github.com/repos/linkedin/TonY/issues/74","repository_url":"https://api.github.com/repos/linkedin/TonY","labels_url":"https://api.github.com/repos/linkedin/TonY/issues/74/labels{/name}","comments_url":"https://api.github.com/repos/linkedin/TonY/issues/74/comments","events_url":"https://api.github.com/repos/linkedin/TonY/issues/74/events","html_url":"https://github.com/linkedin/TonY/issues/74","id":379501472,"node_id":"MDU6SXNzdWUzNzk1MDE0NzI=","number":74,"title":"Failed to run mnist example in GCP","user":{"login":"spicyramen","id":2833776,"node_id":"MDQ6VXNlcjI4MzM3NzY=","avatar_url":"https://avatars2.githubusercontent.com/u/2833776?v=4","gravatar_id":"","url":"https://api.github.com/users/spicyramen","html_url":"https://github.com/spicyramen","followers_url":"https://api.github.com/users/spicyramen/followers","following_url":"https://api.github.com/users/spicyramen/following{/other_user}","gists_url":"https://api.github.com/users/spicyramen/gists{/gist_id}","starred_url":"https://api.github.com/users/spicyramen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spicyramen/subscriptions","organizations_url":"https://api.github.com/users/spicyramen/orgs","repos_url":"https://api.github.com/users/spicyramen/repos","events_url":"https://api.github.com/users/spicyramen/events{/privacy}","received_events_url":"https://api.github.com/users/spicyramen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2018-11-11T08:30:32Z","updated_at":"2018-11-13T08:02:39Z","closed_at":null,"author_association":"NONE","body":"Unable to run mnist example in Dataproc.\r\n\r\n```\r\nsudo java -cp `hadoop classpath`:/usr/local/src/MyJob/tony-cli-0.1.5-all.jar com.linkedin.tony.cli.ClusterSubmitter --python_venv=/usr/lo\r\ncal/src/MyJob/venv.zip --src_dir=/usr/local/src/TonY/mnist/ --executes=/usr/local/src/TonY/mnist/src/mnist_distributed.py --conf_file=/usr/local/src/tony.xml --python_binary_path\r\n=venv/bin/python3.5\r\n18/11/11 08:23:02 INFO cli.ClusterSubmitter: Starting ClusterSubmitter..\r\nSLF4J: Class path contains multiple SLF4J bindings.\r\nSLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]\r\nSLF4J: Found binding in [jar:file:/usr/local/src/MyJob/tony-cli-0.1.5-all.jar!/org/slf4j/impl/StaticLoggerBinder.class]\r\nSLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.\r\nSLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]\r\n18/11/11 08:23:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r\n18/11/11 08:23:02 INFO cli.ClusterSubmitter: Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, null/core-site.\r\nxml, null/hdfs-site.xml\r\nNov 11, 2018 8:23:02 AM com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase <clinit>\r\nINFO: GHFS version: hadoop2-1.9.8\r\n18/11/11 08:23:03 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.\r\n18/11/11 08:23:03 INFO cli.ClusterSubmitter: Copying /usr/local/src/MyJob/tony-cli-0.1.5-all.jar to: hdfs://tony-dev-m/user/root/.tony/6665ca2a-fd31-4f61-a947-33f895517302\r\nException in thread \"main\" java.lang.ClassCastException: [Ljava.lang.Object; cannot be cast to [Ljava.lang.String;\r\n        at com.linkedin.tony.cli.ClusterSubmitter.main(ClusterSubmitter.java:60)\r\n```\r\n\r\n```\r\nhadoop version\r\nHadoop 2.9.0\r\nSubversion Unknown -r Unknown\r\nCompiled by bigtop on 2018-08-17T12:00Z\r\nCompiled with protoc 2.5.0\r\nFrom source with checksum f510b6e8bafb2ddfd660aeb7454e7c30\r\nThis command was run using /usr/lib/hadoop/hadoop-common-2.9.0.jar\r\n```\r\nJava version\r\n```\r\njava -version\r\nopenjdk version \"1.8.0_181\"\r\nOpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-1~deb9u1-b13)\r\nOpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)\r\n```\r\nCommand run:\r\n```\r\njava -cp `hadoop classpath`:/usr/local/src/MyJob/tony-cli-0.1.5-all.jar com.linkedin.tony.cli.ClusterSubmitter \\\r\n--python_venv=/usr/local/src/MyJob/venv.zip \\\r\n--src_dir=/usr/local/src/TonY/mnist/ \\\r\n--executes=/usr/local/src/TonY/mnist/src/mnist_distributed.py \\\r\n--conf_file=/usr/local/src/tony.xml \\\r\n--python_binary_path=venv/bin/python3.5\r\n```\r\n\r\nDirectory structure:\r\n\r\n```\r\n.\r\n├── src\r\n│   └── mnist_distributed.py\r\n├── tony-cli-0.1.5-all.jar\r\n├── tony.xml\r\n└── venv.zip\r\n```\r\n\r\ntony.xml contents:\r\n\r\n```\r\n<configuration>\r\n  <property>\r\n    <name>tony.application.security.enabled</name>\r\n   <value>false</value>\r\n  </property>    \r\n  <property>\r\n    <name>tony.worker.instances</name>\r\n    <value>2</value>\r\n  </property>\r\n  <property>\r\n    <name>tony.worker.memory</name>\r\n    <value>15g</value>\r\n  </property>\r\n  <property>\r\n    <name>tony.worker.gpus</name>\r\n    <value>0</value>\r\n  </property>\r\n  <property>\r\n    <name>tony.ps.memory</name>\r\n    <value>3g</value>\r\n  </property>\r\n</configuration>\r\n```"},"comment":{"url":"https://api.github.com/repos/linkedin/TonY/issues/comments/438172066","html_url":"https://github.com/linkedin/TonY/issues/74#issuecomment-438172066","issue_url":"https://api.github.com/repos/linkedin/TonY/issues/74","id":438172066,"node_id":"MDEyOklzc3VlQ29tbWVudDQzODE3MjA2Ng==","user":{"login":"gogasca","id":30065079,"node_id":"MDQ6VXNlcjMwMDY1MDc5","avatar_url":"https://avatars3.githubusercontent.com/u/30065079?v=4","gravatar_id":"","url":"https://api.github.com/users/gogasca","html_url":"https://github.com/gogasca","followers_url":"https://api.github.com/users/gogasca/followers","following_url":"https://api.github.com/users/gogasca/following{/other_user}","gists_url":"https://api.github.com/users/gogasca/gists{/gist_id}","starred_url":"https://api.github.com/users/gogasca/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gogasca/subscriptions","organizations_url":"https://api.github.com/users/gogasca/orgs","repos_url":"https://api.github.com/users/gogasca/repos","events_url":"https://api.github.com/users/gogasca/events{/privacy}","received_events_url":"https://api.github.com/users/gogasca/received_events","type":"User","site_admin":false},"created_at":"2018-11-13T08:02:39Z","updated_at":"2018-11-13T08:02:39Z","author_association":"NONE","body":"Thanks, I was able to edit:\r\n\r\n```yarn-site.xml```\r\n\r\nAnd modify:\r\n\r\n```\r\n<property>\r\n    <name>yarn.application.classpath</name>\r\n    <value>$HADOOP_CONF_DIR,\r\n      $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,\r\n      $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,\r\n      $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,\r\n      $HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*'</value>\r\n</property>\r\n```\r\n\r\nAs you can see, there was a ' by default. I have contacted the Dataproc team to find out if its a known issue. Job is running now. Will update thread later. Thanks again."}}
{"actor":{"display_login":"gogasca"}}