<p>Uploading to GCE from a pod inside GKE takes really long. I hoped the upgrade to kubernetes 1.1 would help, but it didn't. It is faster, but not as fast as it should be. I made some benchmarks, uploading a single file with 100MiB:</p>

<ul>
<li><p>docker 1.7.2 local</p>

<p>took {20m51s240ms}, that's about ~{0.07993605115907274}MB/s</p></li>
<li><p>docker 1.8.3 local</p>

<p>took {3m51s193ms}, that's about ~{0.4329004329004329}MB/s</p></li>
<li><p>docker 1.9.0 local</p>

<p>took {3m51s424ms}, that's about ~{0.4329004329004329}MB/s</p></li>
<li><p>kubernetes 1.0</p>

<p>took {1h10s952ms}, that's about ~{0.027700831024930747}MB/s</p></li>
<li><p>kubernetes 1.1.2 (docker 1.8.3)</p>

<p>took {32m11s359ms}, that's about ~{0.05178663904712584}MB/s</p></li>
</ul>

<p>As you can see the thruput doubles with kubernetes 1.1.2, but is still really slow. If I want to upload 1GB I have to wait for ~5 hours, this can't be the expected behaviour. GKE runs inside the Google infrastructure, so I expect that it should be faster or at least as fast as uploading from local.</p>

<p>I also noted a very high CPU load (70%) while uploading. It was tested with a <code>n1-highmem-4</code> machine-type and a single RC/pod that was doing nothing then the upload.</p>

<p>I'm using the java client with the GAV coordinates <code>com.google.appengine.tools:appengine-gcs-client:0.5</code></p>

<p>The relevant code is as follows:</p>

<pre><code>InputStream inputStream = ...; // 100MB RandomData from RAM
StorageObject so = new StorageObject().setContentType("text/plain").setName(objectName);
AbstractInputStreamContent content = new InputStreamContent("text/plain", inputStream);
Stopwatch watch = Stopwatch.createStarted();
storage.objects().insert(bucket.getName(), so, content).execute();
watch.stop();
</code></pre>

<p>Copying a 100MB file using a manually installed gcloud with <code>gsutil cp</code> took nearly no time (3 seconds). So it might be an issue with the java-library? The question still remains, how to improve the upload time using the java-library?</p>

<p>Solution is to enable "DirectUpload", so instead of writing</p>

<pre><code>storage.objects().insert(bucket.getName(), so, content).execute();
</code></pre>

<p>you have to write:</p>

<pre><code>    Storage.Objects.Insert insert = storage.objects().insert(bucket.getName(), so, content);
    insert.getMediaHttpUploader().setDirectUploadEnabled(true);
    insert.execute();
</code></pre>

<p>Performance I get with this solution:</p>

<ul>
<li>took {13s515ms}, that's about ~{7.6923076923076925}MB/s</li>
</ul>

<p>JavaDoc for the <code>setDirectUploadEnabled</code>:</p>

<blockquote>
  <p>Sets whether direct media upload is enabled or disabled.</p>
  
  <p>If value is set to true then a direct upload will be done where the
  whole media content is uploaded in a single request. If value is set
  to false then the upload uses the resumable media upload protocol to
  upload in data chunks.</p>
  
  <p>Direct upload is recommended if the content size falls below a certain
  minimum limit. This is because there's minimum block write size for
  some Google APIs, so if the resumable request fails in the space of
  that first block, the client will have to restart from the beginning
  anyway.</p>
  
  <p>Defaults to false.</p>
</blockquote>
