title: Re-training inception google cloud stuck at global step 0
body: <p>I am following the flowers tutorials for re-training inception on google cloud ml. I can run the tutorial, train, predict, just fine.</p>
  <p>I then substituted the flowers dataset for a test dataset of my own. Optical character recognition of image digits.</p>
  <p><a href="https://i.stack.imgur.com/f49ku.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/f49ku.jpg" alt="enter image description here" /></a></p>
  <p>My full code is <a href="https://github.com/bw4sz/MeerkatReader/blob/master/cloudML/MeerkatReader.sh" rel="nofollow noreferrer">here</a></p>
  <p>Dict File for <a href="https://github.com/bw4sz/MeerkatReader/blob/master/cloudML/dict.txt" rel="nofollow noreferrer">labels</a></p>
  <p>Eval <a href="https://github.com/bw4sz/MeerkatReader/blob/master/cloudML/testing_data.csv" rel="nofollow noreferrer">set</a></p>
  <p>Training <a href="https://github.com/bw4sz/MeerkatReader/blob/master/cloudML/training_data.csv" rel="nofollow noreferrer">Set</a></p>
  <p>Running from recent docker build provided by google.</p>
  <pre><code>`docker run -it -p &quot;127.0.0.1:8080:8080&quot; --entrypoint=/bin/bash  gcr.io/cloud-datalab/datalab:local-20161227
  </code></pre>
  <p>I can preprocess files, and submit the training job using</p>
  <pre><code>  # Submit training job.
  gcloud beta ml jobs submit training &quot;$JOB_ID&quot; \
    --module-name trainer.task \
    --package-path trainer \
    --staging-bucket &quot;$BUCKET&quot; \
    --region us-central1 \
    -- \
    --output_path &quot;${GCS_PATH}/training&quot; \
    --eval_data_paths &quot;${GCS_PATH}/preproc/eval*&quot; \
    --train_data_paths &quot;${GCS_PATH}/preproc/train*&quot;
  </code></pre>
  <p>but it never makes it past global step 0. The flowers tutorial trained in about ~ 1 hr on the free tier. I have let my training go as long as 11 hrs. No movement.</p>
  <p><a href="https://i.stack.imgur.com/jyjUM.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jyjUM.png" alt="enter image description here" /></a></p>
  <p>Looking over at stackdriver, nothing progresses.</p>
  <p><a href="https://i.stack.imgur.com/r4bTl.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/r4bTl.png" alt="enter image description here" /></a></p>
  <p>I have also tried a tiny toy dataset of 20 training images, and 10 eval images. Same issue.</p>
  <p>The GCS Bucket ends up looking like this
  <a href="https://i.stack.imgur.com/w0SQe.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/w0SQe.png" alt="enter image description here" /></a></p>
  <p>Perhaps unsurprisingly, I can't visualize this log in tensorboard, nothing to show.</p>
  <p>Full training log:</p>
  <pre><code>INFO    2017-01-10 17:22:00 +0000       unknown_task            Validating job requirements...
  INFO    2017-01-10 17:22:01 +0000       unknown_task            Job creation request has been successfully validated.
  INFO    2017-01-10 17:22:01 +0000       unknown_task            Job MeerkatReader_MeerkatReader_20170110_170701 is queued.
  INFO    2017-01-10 17:22:07 +0000       unknown_task            Waiting for job to be provisioned.
  INFO    2017-01-10 17:22:07 +0000       unknown_task            Waiting for TensorFlow to start.
  INFO    2017-01-10 17:22:10 +0000       master-replica-0                Running task with arguments: --cluster={&quot;master&quot;: [&quot;master-d4f6-0:2222&quot;]} --task={&quot;type&quot;: &quot;master&quot;, &quot;index&quot;: 0} --job={
  INFO    2017-01-10 17:22:10 +0000       master-replica-0                  &quot;package_uris&quot;: [&quot;gs://api-project-773889352370-ml/MeerkatReader_MeerkatReader_20170110_170701/f78d90a60f615a2d108d06557818eb4f82ffa94a/trainer-0.1.tar.gz&quot;],
  INFO    2017-01-10 17:22:10 +0000       master-replica-0                  &quot;python_module&quot;: &quot;trainer.task&quot;,
  INFO    2017-01-10 17:22:10 +0000       master-replica-0                  &quot;args&quot;: [&quot;--output_path&quot;, &quot;gs://api-project-773889352370-ml/MeerkatReader/MeerkatReader_MeerkatReader_20170110_170701/training&quot;, &quot;--eval_data_paths&quot;, &quot;gs://api-project-773889352370-ml/MeerkatReader/MeerkatReader_MeerkatReader_20170110_170701/preproc/eval*&quot;, &quot;--train_data_paths&quot;, &quot;gs://api-project-773889352370-ml/MeerkatReader/MeerkatReader_MeerkatReader_20170110_170701/preproc/train*&quot;],
  INFO    2017-01-10 17:22:10 +0000       master-replica-0                  &quot;region&quot;: &quot;us-central1&quot;
  INFO    2017-01-10 17:22:10 +0000       master-replica-0                } --beta
  INFO    2017-01-10 17:22:10 +0000       master-replica-0                Downloading the package: gs://api-project-773889352370-ml/MeerkatReader_MeerkatReader_20170110_170701/f78d90a60f615a2d108d06557818eb4f82ffa94a/trainer-0.1.tar.gz
  INFO    2017-01-10 17:22:10 +0000       master-replica-0                Running command: gsutil -q cp gs://api-project-773889352370-ml/MeerkatReader_MeerkatReader_20170110_170701/f78d90a60f615a2d108d06557818eb4f82ffa94a/trainer-0.1.tar.gz trainer-0.1.tar.gz
  INFO    2017-01-10 17:22:12 +0000       master-replica-0                Building wheels for collected packages: trainer
  INFO    2017-01-10 17:22:12 +0000       master-replica-0                creating '/tmp/tmpSgdSzOpip-wheel-/trainer-0.1-cp27-none-any.whl' and adding '.' to it
  INFO    2017-01-10 17:22:12 +0000       master-replica-0                adding 'trainer/model.py'
  INFO    2017-01-10 17:22:12 +0000       master-replica-0                adding 'trainer/util.py'
  INFO    2017-01-10 17:22:12 +0000       master-replica-0                adding 'trainer/preprocess.py'
  INFO    2017-01-10 17:22:12 +0000       master-replica-0                adding 'trainer/task.py'
  INFO    2017-01-10 17:22:12 +0000       master-replica-0                adding 'trainer-0.1.dist-info/metadata.json'
  INFO    2017-01-10 17:22:12 +0000       master-replica-0                adding 'trainer-0.1.dist-info/WHEEL'
  INFO    2017-01-10 17:22:12 +0000       master-replica-0                adding 'trainer-0.1.dist-info/METADATA'
  INFO    2017-01-10 17:22:12 +0000       master-replica-0                  Running setup.py bdist_wheel for trainer: finished with status 'done'
  INFO    2017-01-10 17:22:12 +0000       master-replica-0                  Stored in directory: /root/.cache/pip/wheels/e8/0c/c7/b77d64796dbbac82503870c4881d606fa27e63942e07c75f0e
  INFO    2017-01-10 17:22:12 +0000       master-replica-0                Successfully built trainer
  INFO    2017-01-10 17:22:13 +0000       master-replica-0                Running command: python -m trainer.task --output_path gs://api-project-773889352370-ml/MeerkatReader/MeerkatReader_MeerkatReader_20170110_170701/training --eval_data_paths gs://api-project-773889352370-ml/MeerkatReader/MeerkatReader_MeerkatReader_20170110_170701/preproc/eval* --train_data_paths gs://api-project-773889352370-ml/MeerkatReader/MeerkatReader_MeerkatReader_20170110_170701/preproc/train*
  INFO    2017-01-10 17:22:14 +0000       master-replica-0                Starting master/0
  INFO    2017-01-10 17:22:14 +0000       master-replica-0                Initialize GrpcChannelCache for job master -&gt; {0 -&gt; localhost:2222}
  INFO    2017-01-10 17:22:14 +0000       master-replica-0                Started server with target: grpc://localhost:2222
  ERROR   2017-01-10 17:22:16 +0000       master-replica-0                device_filters: &quot;/job:ps&quot;
  INFO    2017-01-10 17:22:19 +0000       master-replica-0                global_step/sec: 0
  </code></pre>
  <p>Just repeating the last line until I kill it.</p>
  <p>Is my mental model for this service incorrect? All suggestions welcome.</p>