{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/cloudml-samples/issues/258","repository_url":"https://api.github.com/repos/GoogleCloudPlatform/cloudml-samples","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/cloudml-samples/issues/258/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/cloudml-samples/issues/258/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/cloudml-samples/issues/258/events","html_url":"https://github.com/GoogleCloudPlatform/cloudml-samples/issues/258","id":364352221,"node_id":"MDU6SXNzdWUzNjQzNTIyMjE=","number":258,"title":"Movielens cloud run model training error with matrix factorization","user":{"login":"benlam62","id":12960170,"node_id":"MDQ6VXNlcjEyOTYwMTcw","avatar_url":"https://avatars2.githubusercontent.com/u/12960170?v=4","gravatar_id":"","url":"https://api.github.com/users/benlam62","html_url":"https://github.com/benlam62","followers_url":"https://api.github.com/users/benlam62/followers","following_url":"https://api.github.com/users/benlam62/following{/other_user}","gists_url":"https://api.github.com/users/benlam62/gists{/gist_id}","starred_url":"https://api.github.com/users/benlam62/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benlam62/subscriptions","organizations_url":"https://api.github.com/users/benlam62/orgs","repos_url":"https://api.github.com/users/benlam62/repos","events_url":"https://api.github.com/users/benlam62/events{/privacy}","received_events_url":"https://api.github.com/users/benlam62/received_events","type":"User","site_admin":false},"labels":[{"id":452160105,"node_id":"MDU6TGFiZWw0NTIxNjAxMDU=","url":"https://api.github.com/repos/GoogleCloudPlatform/cloudml-samples/labels/bug","name":"bug","color":"ee0701","default":true}],"state":"open","locked":false,"assignee":{"login":"happyhuman","id":3443249,"node_id":"MDQ6VXNlcjM0NDMyNDk=","avatar_url":"https://avatars1.githubusercontent.com/u/3443249?v=4","gravatar_id":"","url":"https://api.github.com/users/happyhuman","html_url":"https://github.com/happyhuman","followers_url":"https://api.github.com/users/happyhuman/followers","following_url":"https://api.github.com/users/happyhuman/following{/other_user}","gists_url":"https://api.github.com/users/happyhuman/gists{/gist_id}","starred_url":"https://api.github.com/users/happyhuman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/happyhuman/subscriptions","organizations_url":"https://api.github.com/users/happyhuman/orgs","repos_url":"https://api.github.com/users/happyhuman/repos","events_url":"https://api.github.com/users/happyhuman/events{/privacy}","received_events_url":"https://api.github.com/users/happyhuman/received_events","type":"User","site_admin":false},"assignees":[{"login":"happyhuman","id":3443249,"node_id":"MDQ6VXNlcjM0NDMyNDk=","avatar_url":"https://avatars1.githubusercontent.com/u/3443249?v=4","gravatar_id":"","url":"https://api.github.com/users/happyhuman","html_url":"https://github.com/happyhuman","followers_url":"https://api.github.com/users/happyhuman/followers","following_url":"https://api.github.com/users/happyhuman/following{/other_user}","gists_url":"https://api.github.com/users/happyhuman/gists{/gist_id}","starred_url":"https://api.github.com/users/happyhuman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/happyhuman/subscriptions","organizations_url":"https://api.github.com/users/happyhuman/orgs","repos_url":"https://api.github.com/users/happyhuman/repos","events_url":"https://api.github.com/users/happyhuman/events{/privacy}","received_events_url":"https://api.github.com/users/happyhuman/received_events","type":"User","site_admin":false}],"milestone":null,"comments":1,"created_at":"2018-09-27T08:07:22Z","updated_at":"2018-10-13T05:13:38Z","closed_at":null,"author_association":"NONE","body":"When I tried the Movielens sample, the cloud run model training job failed with matrix factorization. The pre-processing step can be run successfully.\r\n\r\nThe log message is captured as below:\r\n\r\n`Hyperparameter Tuning Trial #1 Failed before any other successful trials were completed. The failed trial had parameters: learning_rate=0.20949118997690069, . The trial's error message was: The replica master 0 exited with a non-zero status of 1. Termination reason: Error. Traceback (most recent call last): [...] return self.install_item(spec, dist.location, tmpdir, deps) File \"/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py\", line 666, in install_item dists = self.install_eggs(spec, download, tmpdir) File \"/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py\", line 856, in install_eggs return self.build_and_install(setup_script, setup_base) File \"/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py\", line 1078, in build_and_install self.run_setup(setup_script, setup_base, args) File \"/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py\", line 1063, in run_setup run_setup(setup_script, args) File \"/usr/lib/python2.7/dist-packages/setuptools/sandbox.py\", line 68, in run_setup DirectorySandbox(setup_dir).run(runner) File \"/usr/lib/python2.7/dist-packages/setuptools/sandbox.py\", line 114, in run return func() File \"/usr/lib/python2.7/dist-packages/setuptools/sandbox.py\", line 67, in runner _execfile(setup_script, ns) File \"/usr/lib/python2.7/dist-packages/setuptools/sandbox.py\", line 43, in _execfile exec(code, globals, locals) File \"/tmp/easy_install-kiZ27t/setuptools-40.4.1/setup.py\", line 194, in <module> File \"/usr/lib/python2.7/distutils/core.py\", line 111, in setup _setup_distribution = dist = klass(attrs) File \"/usr/lib/python2.7/dist-packages/setuptools/dist.py\", line 266, in __init__ _Distribution.__init__(self,attrs) File \"/usr/lib/python2.7/distutils/dist.py\", line 287, in __init__ self.finalize_options() File \"/usr/lib/python2.7/dist-packages/setuptools/dist.py\", line 301, in finalize_options ep.load()(self, ep.name, value) File \"/usr/lib/python2.7/dist-packages/pkg_resources.py\", line 2195, in load raise ImportError(\"%r has no %r attribute\" % (entry, attr)) ImportError: <module 'setuptools.dist' from '/usr/lib/python2.7/dist-packages/setuptools/dist.pyc'> has no 'check_specifier' attribute To find out more about why your job exited please check the logs: https://console.cloud.google.com/logs/viewer?project=21100138813&resource=ml_job%2Fjob_id%2Fmovielens_factorization_20180920_162902&advancedFilter=resource.type%3D%22ml_job%22%0Aresource.labels.job_id%3D%22movielens_factorization_20180920_162902%22`\r\n\r\n**To Reproduce**\r\n\r\nThe training input is:\r\n\r\n```\r\n{\r\n  \"scaleTier\": \"CUSTOM\",\r\n  \"masterType\": \"standard_gpu\",\r\n  \"packageUris\": [\r\n    \"gs://ben-lam-project-1-ml/movielens_factorization_20180920_162902/aa659c246aec14c8fdfe4890b48a83f5c5d2fac87f45a129bf20c40e061967b8/movielens-1.0.tar.gz\"\r\n  ],\r\n  \"pythonModule\": \"trainer.task\",\r\n  \"args\": [\r\n    \"--raw_metadata_path\",\r\n    \"gs://ben-lam-project-1-ml/benlam/movielens/movielens_20180920_124737/raw_metadata\",\r\n    \"--transform_savedmodel\",\r\n    \"gs://ben-lam-project-1-ml/benlam/movielens/movielens_20180920_124737/transform_fn\",\r\n    \"--eval_data_paths\",\r\n    \"gs://ben-lam-project-1-ml/benlam/movielens/movielens_20180920_124737/features_eval*.tfrecord.gz\",\r\n    \"--train_data_paths\",\r\n    \"gs://ben-lam-project-1-ml/benlam/movielens/movielens_20180920_124737/features_train*.tfrecord.gz\",\r\n    \"--output_path\",\r\n    \"gs://ben-lam-project-1-ml/benlam/movielens/model/movielens_factorization_20180920_162902\",\r\n    \"--model_type\",\r\n    \"matrix_factorization\",\r\n    \"--eval_type\",\r\n    \"ranking\",\r\n    \"--l2_weight_decay\",\r\n    \"0.01\",\r\n    \"--learning_rate\",\r\n    \"0.05\",\r\n    \"--train_steps\",\r\n    \"500000\",\r\n    \"--eval_steps\",\r\n    \"500\",\r\n    \"--movie_embedding_dim\",\r\n    \"64\",\r\n    \"--top_k_infer\",\r\n    \"100\"\r\n  ],\r\n  \"hyperparameters\": {\r\n    \"goal\": \"MAXIMIZE\",\r\n    \"params\": [\r\n      {\r\n        \"parameterName\": \"learning_rate\",\r\n        \"minValue\": 0.001,\r\n        \"maxValue\": 0.3,\r\n        \"type\": \"DOUBLE\",\r\n        \"scaleType\": \"UNIT_LOG_SCALE\"\r\n      }\r\n    ],\r\n    \"maxTrials\": 10,\r\n    \"maxParallelTrials\": 5,\r\n    \"hyperparameterMetricTag\": \"recall_at_10\"\r\n  },\r\n  \"region\": \"us-central1\"\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/cloudml-samples/issues/comments/429512106","html_url":"https://github.com/GoogleCloudPlatform/cloudml-samples/issues/258#issuecomment-429512106","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/cloudml-samples/issues/258","id":429512106,"node_id":"MDEyOklzc3VlQ29tbWVudDQyOTUxMjEwNg==","user":{"login":"puneith","id":8098778,"node_id":"MDQ6VXNlcjgwOTg3Nzg=","avatar_url":"https://avatars3.githubusercontent.com/u/8098778?v=4","gravatar_id":"","url":"https://api.github.com/users/puneith","html_url":"https://github.com/puneith","followers_url":"https://api.github.com/users/puneith/followers","following_url":"https://api.github.com/users/puneith/following{/other_user}","gists_url":"https://api.github.com/users/puneith/gists{/gist_id}","starred_url":"https://api.github.com/users/puneith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/puneith/subscriptions","organizations_url":"https://api.github.com/users/puneith/orgs","repos_url":"https://api.github.com/users/puneith/repos","events_url":"https://api.github.com/users/puneith/events{/privacy}","received_events_url":"https://api.github.com/users/puneith/received_events","type":"User","site_admin":false},"created_at":"2018-10-13T05:13:38Z","updated_at":"2018-10-13T05:13:38Z","author_association":"CONTRIBUTOR","body":"@benlam62 did you change versions in setup.py?"}}
{"actor":{"display_login":"puneith"}}