<p><a href="http://arxiv.org/help/faq/cache" rel="nofollow noreferrer" title="arXiv recommends squid">arXiv recommends squid</a> in httpd accelerator mode for precisely this purpose. Any particular reason why this is not good enough?</p>
ArXiv replication brainstorming
<p>The arXiv e-print archive has several terabytes of papers from various fields of science.  Some users would like to maintain a full copy of this data on their own computers, while others just want to download the most recent papers in a particular category.  They are looking to reduce bandwidth load using some kind of distributed download system (e.g. BitTorrent).  I'm looking for ideas for a program or set of programs that would cover all of this.</p>