title: Highly variable performance on datastore and memcache operations (GAE)
body: <p>I am trying to optimize performance on GAE but once I deploy I get very unstable results. It's really hard to see if each optimization actually works because datastore and memcache operations take a very variable time (it ranges from milliseconds to seconds for the same operations).</p>
  
  <p>For these tests I am the only one making only one request on the application by refreshing the homepage. There is no other people/traffic happening (besides my own browser requesting images/css/js files from the page).</p>
  
  <p><strong>Edit:</strong> To make sure that the drops were not due to concurrent requests from the browser (images/css/js), I've redone the test by requesting ONLY the page with urllib2.urlopen(). Problem persists.</p>
  
  <p>My questions are:</p>
  
  <ul>
  <li><strong>1) Is this something to expect due to the fact that machines/resources are shared?</strong></li>
  <li><strong>2) What are the most common cases where this behavior can happen?</strong></li>
  <li><strong>3) Where can I go from there?</strong></li>
  </ul>
  
  <p>Here is a very slow datastore get (memcache was just flushed):
  <img src="https://i.stack.imgur.com/PXXJy.png" alt="Ultra slow datastore get">
  <a href="https://i.stack.imgur.com/PXXJy.png" rel="nofollow noreferrer">Full size</a></p>
  
  <p>Here is a very slow memcache get (things are cached because of the previous request):
  <img src="https://i.stack.imgur.com/Ef9WH.png" alt="Slow memcache get">
  <a href="https://i.stack.imgur.com/Ef9WH.png" rel="nofollow noreferrer">Full size</a></p>
  
  <p>Here is a slow but faster memcache get (same repro step as the previous one, different calls are slow):
  <img src="https://i.stack.imgur.com/qkKpS.png" alt="enter image description here">
  <a href="https://i.stack.imgur.com/qkKpS.png" rel="nofollow noreferrer">Full size</a></p>