<p>full pdf content is in the amazon cloud.</p>

<p>while there are > 600k papers on arXiv the total size of the pdf is &lt; 1/2 TB</p>

<p><a href="http://arxiv.org/help/bulk_data_s3" rel="nofollow noreferrer">http://arxiv.org/help/bulk_data_s3</a></p>

<p>T.</p>
ArXiv replication brainstorming
<p>The arXiv e-print archive has several terabytes of papers from various fields of science.  Some users would like to maintain a full copy of this data on their own computers, while others just want to download the most recent papers in a particular category.  They are looking to reduce bandwidth load using some kind of distributed download system (e.g. BitTorrent).  I'm looking for ideas for a program or set of programs that would cover all of this.</p>