title: How to access Google Cloud Storage Bucket from AI Platform job
body: <p>My Google AI Platform / ML Engine training job doesn't seem to have access to the training file I put into a Google Cloud Storage bucket.</p>
  
  <p>Google's AI Platform / ML Engine requires you store training data files in one of their Cloud Storage buckets. Accessing locally from CLI works fine. However, when I send a training job (after ensuring the data is in the appropriate location in my Cloud Storage bucket), I get an error seeming to be due to no access to the bucket <code>Link URL</code>.</p>
  
  <p>The error is from trying to read what looks to me like the contents of a web page that Google served up saying "Hey, you don't have access to this." I see this <code>gaia.loginAutoRedirect.start(5000,</code> and a URL with this flag at the end: <code>noautologin=true</code>.</p>
  
  <p>I know permissions between AI Platform and Cloud Storage are a thing, but both are under the same project. The walkthroughs I'm using at very least imply that no further action is required if under the same project.</p>
  
  <p>I am assuming I need to use the <code>Link URL</code> provided in the bucket Overview tab. Tried the <code>Link for gsutil</code> but the python (from Google's <a href="https://github.com/GoogleCloudPlatform/cloudml-samples" rel="nofollow noreferrer">CloudML Samples repo</a>) was upset about using <code>gs://</code>.</p>
  
  <p>I think Google's examples are proving insufficient since their example data is from a public URL rather than a private Cloud Storage bucket.</p>
  
  <p>Ultimately, the error message I get is a Python error. But like I said, this is preceded by a bunch of gross <code>INFO</code> logs of HTML/CSS/JS from Google saying I don't have permission to get the file I'm trying to get. These logs are actually just because I added a print statement to the <code>util.py</code> file as well - right before <code>read_csv()</code> on the train file. (So the Python parse error is due to trying to parse HTML as a CSV).</p>
  
  <pre><code>... 
  INFO    g("gaia.loginAutoRedirect.stop",function(){var b=n;b.b=!0;b.a&amp;&amp;(clearInterval(b.a),b.a=null)});
  INFO    gaia.loginAutoRedirect.start(5000,
  INFO    'https:\x2F\x2Faccounts.google.com\x2FServiceLogin?continue=https%3A%2F%2Fstorage.cloud.google.com%2F&lt;BUCKET_NAME&gt;%2Fdata%2F%2Ftrain.csv\x26followup=https%3A%2F%2Fstorage.cloud.google.com%2F&lt;BUCKET_NAME&gt;%2Fdata%2F%2Ftrain.csv\x26service=cds\x26passive=1209600\x26noautologin=true',
  ERROR   Command '['python', '-m', u'trainer.task', u'--train-files', u'gs://&lt;BUCKET_NAME&gt;/data/train.csv', u'--eval-files', u'gs://&lt;BUCKET_NAME&gt;/data/test.csv', u'--batch-pct', u'0.2', u'--num-epochs', u'1000', u'--verbosity', u'DEBUG', '--job-dir', u'gs://&lt;BUCKET_NAME&gt;/predictor']' returned non-zero exit status 1.
  </code></pre>
  
  <pre><code>Traceback (most recent call last):
    File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
      "__main__", fname, loader, pkg_name)
    File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
      exec code in run_globals
    File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 137, in &lt;module&gt;
      train_and_evaluate(args)
    File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 80, in train_and_evaluate
      train_x, train_y, eval_x, eval_y = util.load_data()
    File "/root/.local/lib/python2.7/site-packages/trainer/util.py", line 168, in load_data
      train_df = pd.read_csv(training_file_path, header=0, names=_CSV_COLUMNS, na_values='?')
    File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 678, in parser_f
      return _read(filepath_or_buffer, kwds)
    File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 446, in _read
      data = parser.read(nrows)
    File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 1036, in read
      ret = self._engine.read(nrows)
    File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 1848, in read
      data = self._reader.read(nrows)
    File "pandas/_libs/parsers.pyx", line 876, in pandas._libs.parsers.TextReader.read
    File "pandas/_libs/parsers.pyx", line 891, in pandas._libs.parsers.TextReader._read_low_memory
    File "pandas/_libs/parsers.pyx", line 945, in pandas._libs.parsers.TextReader._read_rows
    File "pandas/_libs/parsers.pyx", line 932, in pandas._libs.parsers.TextReader._tokenize_rows
    File "pandas/_libs/parsers.pyx", line 2112, in pandas._libs.parsers.raise_parser_error
  ParserError: Error tokenizing data. C error: Expected 5 fields in line 205, saw 961
  </code></pre>
  
  <p>To get the data, I'm more or less trying to mimic this:
  <a href="https://github.com/GoogleCloudPlatform/cloudml-samples/blob/master/census/tf-keras/trainer/util.py" rel="nofollow noreferrer">https://github.com/GoogleCloudPlatform/cloudml-samples/blob/master/census/tf-keras/trainer/util.py</a></p>
  
  <p>Various ways I have tried to address my bucket in my copy of util.py:
  <code>https://console.cloud.google.com/storage/browser/&lt;BUCKET_NAME&gt;/data</code> (think this was the "Link URL" back in May)<br>
  <code>https://storage.cloud.google.com/&lt;BUCKET_NAME&gt;/data</code> (this is the "Link URL" now - in July)<br>
  <code>gs://&lt;BUCKET_NAME&gt;/data</code> (this is the URI - which gives a different error about not liking <code>gs</code> as a url type)</p>