title: PyPI is slow. How do I run my own server?
body: <p>When a new developer joins the team, or Jenkins runs a complete build, I need to create a fresh virtualenv. I often find that setting up a virtualenv with Pip and a large number (more than 10) of requirements takes a very long time to install everything from PyPI.
  Often it fails altogether with:</p>
  
  <pre><code>Downloading/unpacking Django==1.4.5 (from -r requirements.pip (line 1))
  Exception:
  Traceback (most recent call last):
    File "/var/lib/jenkins/jobs/hermes-web/workspace/web/.venv/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg/pip/basecommand.py", line 107, in main
      status = self.run(options, args)
    File "/var/lib/jenkins/jobs/hermes-web/workspace/web/.venv/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg/pip/commands/install.py", line 256, in run
      requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
    File "/var/lib/jenkins/jobs/hermes-web/workspace/web/.venv/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg/pip/req.py", line 1018, in prepare_files
      self.unpack_url(url, location, self.is_download)
    File "/var/lib/jenkins/jobs/hermes-web/workspace/web/.venv/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg/pip/req.py", line 1142, in unpack_url
      retval = unpack_http_url(link, location, self.download_cache, self.download_dir)
    File "/var/lib/jenkins/jobs/hermes-web/workspace/web/.venv/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg/pip/download.py", line 463, in unpack_http_url
      download_hash = _download_url(resp, link, temp_location)
    File "/var/lib/jenkins/jobs/hermes-web/workspace/web/.venv/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg/pip/download.py", line 380, in _download_url
      chunk = resp.read(4096)
    File "/usr/lib64/python2.6/socket.py", line 353, in read
      data = self._sock.recv(left)
    File "/usr/lib64/python2.6/httplib.py", line 538, in read
      s = self.fp.read(amt)
    File "/usr/lib64/python2.6/socket.py", line 353, in read
      data = self._sock.recv(left)
  timeout: timed out
  </code></pre>
  
  <p>I'm aware of Pip's <code>--use-mirrors</code> flag, and sometimes people on my team have worked around by using <code>--index-url http://f.pypi.python.org/simple</code> (or another mirror) until they have a mirror that responds in a timely fashion. We're in the UK, but there's a PyPI mirror in Germany, and we don't have issues downloading data from other sites.</p>
  
  <p>So, I'm looking at ways to mirror PyPI internally for our team.</p>
  
  <p>The options I've looked at are:</p>
  
  <ol>
  <li><p>Running my own PyPI instance. There's the official PyPI implementation: <a href="https://bitbucket.org/loewis/pypi" rel="noreferrer">CheeseShop</a> as well as several third party implementations, such as: <a href="https://github.com/benliles/djangopypi" rel="noreferrer">djangopypi</a> and <a href="https://github.com/schmir/pypiserver" rel="noreferrer">pypiserver</a> (see footnote)</p>
  
  <p>The problem with this approach is that I'm not interested in full PyPI functionality with file upload, I just want to mirror the content it provides.</p></li>
  <li><p>Running a PyPI mirror with <a href="https://bitbucket.org/loewis/pep381client/" rel="noreferrer">pep381client</a> or <a href="https://github.com/mviera/pypi-mirror" rel="noreferrer">pypi-mirror</a>.</p>
  
  <p>This looks like it could work, but it requires my mirror to download everything from PyPI first. I've set up a test instance of pep381client, but my download speed varies between 5 Kb/s and 200 Kb/s (bits, not bytes). Unless there's a copy of the full PyPI archive somewhere, it will take me weeks to have a useful mirror.</p></li>
  <li><p>Using a PyPI round-robin proxy such as <a href="https://code.google.com/p/yopypi/" rel="noreferrer">yopypi</a>. </p>
  
  <p>This is irrelevant now that <a href="http://pypi.python.org" rel="noreferrer">http://pypi.python.org</a> itself consists of <a href="http://www.pypi-mirrors.org/" rel="noreferrer">several geographically distinct servers</a>.</p></li>
  <li><p>Copying around a virtualenv between developers, or hosting a <a href="http://honza.ca/2012/02/how-not-to-depend-on-pypi" rel="noreferrer">folder of the current project's dependencies</a>.</p>
  
  <p>This doesn't scale: we have several different Python projects whose dependencies change (slowly) over time. As soon as the dependencies of any project change, this central folder must be updated to add the new dependencies. Copying the virtualenv is worse than copying the packages though, since any Python packages with C modules need to be compiled for the target system. Our team has both Linux and OS X users.</p>
  
  <p>(This still looks like the best option of a bad bunch.)</p></li>
  <li><p>Using an intelligent PyPI caching proxy: <a href="https://github.com/camptocamp/collective.eggproxy" rel="noreferrer">collective.eggproxy</a></p>
  
  <p>This seems like it would be a very good solution, but <a href="https://pypi.python.org/pypi/collective.eggproxy" rel="noreferrer">the last version on PyPI is dated 2009</a> and discusses mod_python.</p></li>
  </ol>
  
  <p>What do other large Python teams do? What's the best solution to quickly install the same set of python packages?</p>
  
  <p>Footnotes:</p>
  
  <ul>
  <li>I've seen the question <a href="https://stackoverflow.com/questions/1235331/how-to-roll-my-own-pypi/12874682#12874682">How to roll my own PyPI?</a>, but that question relates to hosting private code.</li>
  <li>The Python wiki <a href="http://wiki.python.org/moin/PyPiImplementations" rel="noreferrer">lists alternative PyPI implementations</a></li>
  <li>I've also recently discovered <a href="https://crate.io/" rel="noreferrer">Crate.io</a> but I don't believe that helps me when using Pip.</li>
  <li>There's a website monitoring <a href="http://www.pypi-mirrors.org/" rel="noreferrer">PyPI mirror status</a></li>
  <li><a href="http://reinout.vanrees.org/weblog/2013/02/28/pypi-discussion.html" rel="noreferrer">Some packages on PyPI have their files hosted elsewhere</a> so even a perfect mirror won't help all dependencies</li>
  </ul>